feat(xce-test-cli-ux): wrap every blocking RPC with the interrupt adapter
Routes all 26 blocking xcecli_* call sites across ~24 commands through xcetest_run_interruptible(), wires xcetest_set_active_connection() at execute_line()'s dispatch point (active_ctx->ctx, read fresh per command), and fixes main()'s REPL loop to distinguish a real EOF from an EINTR-interrupted readline() so Ctrl-C at an empty prompt discards the line instead of exiting the session. Live-discovered and fixed bug-1794: fill_job_alloc() never initializes xce_job_alloc_t's nres_shared/allocs_shared fields -- harmless as stack garbage before this task, genuine heap garbage once moved into a heap-allocated ctx block (a necessary consequence of the adapter's heap-only precondition). Fixed by switching all ~26 call-context allocations from malloc to calloc uniformly. Also fixed an argv-aliasing hazard: cmd_hello/cmd_jobckpt's -m values aliased optarg, which execute_line() frees immediately after return including on the abandoned path -- now strdup'd into the heap block. Also closed a safety-critical gap this task's own design introduced (bug-1795): before this task the REPL was fully synchronous, so leave_ctx() (called by up/quit) could never run while an RPC was outstanding. An abandoned worker can now persist past command completion, so leave_ctx() tearing down the connection could use-after-free it. Fixed via xcetest_wait_connection_idle(), which blocks on the same per-connection semaphore the adapter already tracks, then releases it, proving no worker is active before teardown. Proven with a deterministic white-box RED->GREEN test, since the race itself is too timing-dependent for a black-box test to reliably force. 1361/1361 across the full controller suite (+2), 0 regressions. Task 3.2 of .kiro/specs/xce-test-cli-ux/tasks.md. Requirements: 1.1, 1.2, 1.3, 1.4, 1.5 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
This commit is contained in:
parent
f152c59e75
commit
45120af2cc
7 changed files with 1887 additions and 70 deletions
|
|
@ -106,7 +106,7 @@
|
|||
- _Boundary: Flag Parsing Layer_
|
||||
- _Depends: 1.1, 1.2_
|
||||
|
||||
- [ ] 3.2 Wrap every blocking request with the interruptible-call adapter
|
||||
- [x] 3.2 Wrap every blocking request with the interruptible-call adapter
|
||||
- Route every command's blocking request through the interruptible-call adapter built in 1.4/1.5, so an interrupt during any of them abandons the wait immediately, reports the command as failed, and leaves the active connection and context intact
|
||||
- Ensure the session never exits as a side effect of a single interrupt keypress, and that an interrupt at an empty prompt simply discards the current input line
|
||||
- Observable completion: interrupting a command that is blocked waiting on a request returns control to the prompt immediately (well before the request's own timeout) and the same connection accepts a subsequent command normally
|
||||
|
|
@ -178,6 +178,7 @@
|
|||
## Implementation Notes
|
||||
- `src/tools/xce-build.sh controller asan xce.build --incremental` writes its build dir at `src/xce.build/` (relative to the script's own location), not a repo-root `xce.build/` — meson logs are at `src/xce.build/meson-logs/testlog.txt`. (task 1.1)
|
||||
- New Unity test files need two wiring points, not just meson.build's source list: also register an `extern void test_<module>(void);` + `TEST(test_<module>);` in `src/controller/testsuite/controller_test.c`, mirroring the existing `test_rcfile` registration, or the suite silently never runs. (task 1.1)
|
||||
- **Safety-critical finding**: wrapping RPC calls in an abandonable worker thread means `leave_ctx()` (called by `up`/`quit`) can now run while a previously-interrupted command's worker is still using the connection — before this task the REPL was fully synchronous so this UAF couldn't exist. Fixed via `xcetest_wait_connection_idle(conn_key)` (interruptcall.c), which blocks on the SAME per-connection semaphore the adapter already uses, then immediately releases it, proving no worker is active before teardown proceeds. Call it before any future connection-teardown path, not just `leave_ctx()`, if one is added later. A timing-dependent black-box test alone could NOT reliably prove this race (ran clean 20/20 even with the fix disabled) — the reliable proof came from a white-box test directly exercising the wait function against a real abandoned worker. When a fix's failure mode is a sub-millisecond race, don't trust a black-box test's clean run as RED-phase evidence; get a deterministic white-box reproduction instead. (task 3.2, bug-1795)
|
||||
- All 17 flag-accepting commands migrated to `flagspec_t`+`getopt_long` (review verified essentially every command line-by-line via git diff context, not just a sample — zero regressions to any existing short flag's letter/value-format/default). `cmd_walcompare`'s migration needs a leading `+` in its optstring (`getopt_long`'s "stop at first non-option") to preserve its pre-migration "flags only before positionals" behavior — without it, argv permuting would change which tokens are treated as flags vs. member paths. Only 7 of the 17 migrated commands got dedicated black-box tests (`qctrl`, `jobinfo`, `jobkill`, `hostctrl`, `setpend`, `resupdate`, `walcompare`); the other 10 (`hello`, `jobrun`, `jobsub`, `resval`, `jobckpt`, `runtasks`, `jobstart`, `jobfinish`, `jobfail`, `join_worker_parse_attrs`) have no valid-flag regression test at all beyond this review's manual line-by-line diff check — worth a follow-up smoke test per command if this area gets touched again. (task 3.1)
|
||||
- `cmd_join_worker`'s sibling fields `maxswap`/`maxtmp` have the exact same stale-MB-scale bug as bug-1557's `maxmem` (1024/2048 bytes instead of KB/MB-appropriate values) but were deliberately left unfixed here: `USAGE_JOIN_WORKER` currently documents these exact stale values accurately, so fixing only the code would create a new code/docs mismatch outside this task's mandate. Worth a coordinated follow-up bug fixing both fields' defaults and their help text together. A new `test_join_worker_sibling_defaults_unchanged` test locks in today's values so that follow-up isn't confused with this task's maxmem-specific assertions. (task 2.6)
|
||||
- When a fix touches a pure function stuck inside `test.c` (no networking/REPL/global-state dependency), extract it into its own module with a standalone meson `test()` target rather than settling for a throwaway/manual verification — this codebase already has two precedents for exactly this (`resval_typeresolve.c`, `misc.c`'s runusage helpers; now also `joblist_display.c` for `worker_status_str`/`job_state_str`/`format_job_time`). `test.c` itself is never linked into `controller_unit_test`, so this extract-to-standalone-module pattern is the only way to get permanent automated coverage for such functions — a scratch/deleted verification harness is not sufficient and will be sent back on review. `cmd_reslist`/`joblist_query`/`cmd_getconf`'s table-printing bodies remain genuinely untestable without a live cluster (fetch+print are inlined together, confirmed on review) and stay uncovered by permanent tests for now. (task 2.5)
|
||||
|
|
|
|||
|
|
@ -21536,6 +21536,61 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-09T16:24:19Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1794",
|
||||
"timestamp": "2026-09-09T19:03:51Z",
|
||||
"error_message": "xce_test cmd_jobrun crashes (ASAN allocation-size-too-big abort in alloc_packet, requested size ~0xffffffffc3c3c4b0) when run live against a real controller, after xce-test-cli-ux task 3.2 wrapped its xcecli_jobdispatch() call through the interruptible-call adapter (heap-allocated ctx block instead of a stack-local xce_job_alloc_t).",
|
||||
"file": "src/controller/testsuite/test.c (cmd_jobrun, fill_job_alloc, jobrun_call_t/jobrun_thunk)",
|
||||
"root_cause": "PRE-EXISTING, latent: fill_job_alloc() (test.c) only ever sets xce_job_alloc_t.nhosts/.allocs -- it never initializes the struct's other two fields, nres_shared and allocs_shared (xce/types.h). Before task 3.2, xce_job_alloc_t jalloc was a plain stack-local in cmd_jobrun, so those two never-set fields held whatever incidental value that stack frame happened to have -- harmless in practice, apparently usually small/zero. Task 3.2 moved this struct into a heap-allocated jobrun_call_t block (required by the interruptible-call adapter's contract: fn must only touch heap-owned memory, never the caller's stack, since an abandoned call's worker thread can outlive the caller's own stack frame) allocated via plain malloc(), which leaves ALL bytes as genuine heap garbage -- confirmed live (real single-node deploy, jobrun -j <jobid> -m worker-1:1): the garbage nres_shared/allocs_shared corrupted xce_jobdispatch_req's wire-size computation in lib/request.c, producing a multi-exabyte allocation request that ASAN correctly aborted on.",
|
||||
"fix": "FIXED (2026-09-09, xce-test-cli-ux task 3.2). Every per-call-site heap block (all ~26 <cmd>_call_t typedefs test.c's interrupt-wrapping introduces) now allocates via calloc(1, sizeof(*call)) instead of malloc(sizeof(*call)), zero-initializing every field uniformly rather than auditing each wrapped request struct for a similarly-never-fully-set field one at a time -- the same defensive default an accidentally-zero stack frame was already silently relied on for. Verified by re-running the exact live repro (jobrun against a real single-node deploy) after the fix: completes normally with no ASAN error.",
|
||||
"tags": [
|
||||
"controller",
|
||||
"testsuite",
|
||||
"xce_test",
|
||||
"cmd_jobrun",
|
||||
"fill_job_alloc",
|
||||
"uninitialized-memory",
|
||||
"heap-allocation",
|
||||
"asan",
|
||||
"interruptcall",
|
||||
"xce-test-cli-ux",
|
||||
"task-3.2"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1791"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-09T19:03:51Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1795",
|
||||
"timestamp": "2026-09-09T20:10:00Z",
|
||||
"error_message": "leave_ctx() (test.c, called by cmd_up/cmd_quit) could free a connection's api_ctx_t (xcecli_stop()+xcecli_ctx_fini(), src/lib/client.c) while an abandoned worker thread from a prior Ctrl-C-interrupted RPC on that SAME connection was still running and still reading/writing it -- a genuine use-after-free, found by kiro-review during xce-test-cli-ux task 3.2's own round-1 review (not reproduced live/under ASAN on that run; identified structurally by tracing the code paths).",
|
||||
"file": "src/controller/testsuite/test.c (leave_ctx), src/controller/testsuite/interruptcall.c/.h (xcetest_wait_connection_idle, new), src/lib/client.c (xcecli_wakeup, xcecli_stop, xcecli_ctx_fini, read-only for this fix)",
|
||||
"root_cause": "Task 3.2's own interruptible-call adapter (tasks 1.4/1.5, interruptcall.c) runs every blocking xce_* RPC on a detached worker thread; when the operator interrupts (Ctrl-C) a blocked call, xcetest_run_interruptible() returns -EINTR to the REPL immediately, but the worker thread it spawned keeps running -- possibly for up to its own ~10s cbuf_timedget deadline -- reading/writing whatever connection state its ctx closes over. Before this task, the REPL was fully synchronous, so leave_ctx() (test.c, shared by cmd_up's '..' and cmd_quit) could never run while an RPC was outstanding on that connection; task 3.2's adapter broke that invariant by letting control return to the prompt WHILE a worker is still abandoned-but-running. leave_ctx() already called xcecli_wakeup(active_ctx->ctx, -EINTR) before tearing the connection down, but that is only a best-effort nudge (it wakes the underlying condvar wait sooner than the worker's own deadline) -- nothing forced leave_ctx() to actually WAIT for that woken worker to finish before proceeding to xcecli_stop() (tears down the live socket + failover lock) and xcecli_ctx_fini() (unconditionally free()s the whole api_ctx_t). If the operator's very next action after abandoning a hung command is 'up' or 'quit' -- an entirely ordinary thing to do -- the still-running worker can be mid-read/write on memory that just got freed out from under it. This directly contradicts the Interruptible Call Adapter's own documented invariant ('at most one worker thread -- running or abandoned-but-still-running -- is ever active per connection at a time', interruptcall.h/design.md) -- that invariant was enforced only BETWEEN xcetest_run_interruptible() calls (via the per-connection semaphore registry, tasks 1.4/1.5), never against connection TEARDOWN, since leave_ctx() never itself goes through xcetest_run_interruptible().",
|
||||
"fix": "FIXED (2026-09-09, xce-test-cli-ux task 3.2 remediation round 1). Added xcetest_wait_connection_idle(void* conn_key) to interruptcall.c/.h, reusing the EXACT SAME per-connection semaphore registry lookup (xcetest_conn_lock_for()) that xcetest_run_interruptible() already builds and maintains -- not a new/parallel mechanism. It does a plain BLOCKING sem_wait() (deliberately not the interruptible sem_trywait()-poll variant used elsewhere -- this is a narrow, safety-critical wait where correctness outranks Ctrl-C responsiveness, and it resolves quickly in practice since xcecli_wakeup() already nudges the worker) then immediately sem_post()s the semaphore back rather than holding it -- the semantic is 'prove no worker is active on this connection right now', not 'hold the connection locked'. leave_ctx() now calls xcetest_wait_connection_idle(ctx->ctx) right after the existing xcecli_wakeup() call and right before xcecli_stop()/xcecli_ctx_fini() -- since both cmd_up and cmd_quit call this SAME shared leave_ctx(), both paths are covered by a single call site. New black-box regression test test_interrupt_then_immediate_teardown_no_uaf (test_test_cli.c, reuses the existing clitest_start_blackhole_listener() harness helper) interrupts a blocked hello RPC against a black-hole TCP peer and sends 'up' IMMEDIATELY afterward -- with no settling wait for the interrupted prompt to redisplay first, unlike the sibling interrupt test -- to race leave_ctx()'s teardown against the abandoned worker on purpose; asserts the session exits only via a normal (non-signal) status and that stderr never contains an 'AddressSanitizer' diagnostic anywhere across the whole session. Verified under the asan build profile.",
|
||||
"tags": [
|
||||
"controller",
|
||||
"testsuite",
|
||||
"xce_test",
|
||||
"leave_ctx",
|
||||
"cmd_up",
|
||||
"cmd_quit",
|
||||
"interruptcall",
|
||||
"use-after-free",
|
||||
"asan",
|
||||
"concurrency",
|
||||
"semaphore",
|
||||
"xce-test-cli-ux",
|
||||
"task-3.2",
|
||||
"kiro-review-finding"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1794"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-09T20:10:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -314,3 +314,41 @@ int xcetest_run_interruptible(xcetest_blocking_fn fn, void* ctx)
|
|||
xcetest_sleep_poll_interval();
|
||||
}
|
||||
}
|
||||
|
||||
/* See interruptcall.h for the full contract. Reuses xcetest_conn_lock_for()
|
||||
* -- the SAME lazy-lookup/create used by xcetest_run_interruptible() above
|
||||
* -- rather than a parallel registry, so this genuinely observes the same
|
||||
* per-connection semaphore any in-flight or abandoned worker on `conn_key`
|
||||
* is holding. */
|
||||
void xcetest_wait_connection_idle(void* conn_key)
|
||||
{
|
||||
sem_t* lock;
|
||||
int rc;
|
||||
|
||||
lock = xcetest_conn_lock_for(conn_key);
|
||||
if (!lock) {
|
||||
/* Allocation failure in the lazy-create path: this key has never
|
||||
* been registered, so by definition no worker could be holding a
|
||||
* lock for it via this registry. Nothing to wait for. */
|
||||
return;
|
||||
}
|
||||
|
||||
/* Plain BLOCKING wait, deliberately not the interruptible
|
||||
* sem_trywait()-poll loop used above -- see interruptcall.h's
|
||||
* "CONTRACT" section on this function. A signal delivered while
|
||||
* blocked here can legitimately interrupt sem_wait() with EINTR even
|
||||
* though nothing else ever posts spuriously; retry rather than treat
|
||||
* that as "acquired" or bail out early, since either would defeat
|
||||
* this function's entire purpose (proving the wait genuinely
|
||||
* completed before the caller proceeds to tear the connection down). */
|
||||
do {
|
||||
rc = sem_wait(lock);
|
||||
} while ((rc != 0) && (errno == EINTR));
|
||||
assert(rc == 0); /* NDEBUG caveat: see xcetest_worker_main()'s comment. */
|
||||
|
||||
/* Immediately release: the semantic is "prove idle right now," not
|
||||
* "hold the connection locked." */
|
||||
rc = sem_post(lock);
|
||||
(void)rc;
|
||||
assert(rc == 0);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -249,6 +249,64 @@ void xcetest_set_active_connection(void* conn_key);
|
|||
*/
|
||||
int xcetest_run_interruptible(xcetest_blocking_fn fn, void* ctx);
|
||||
|
||||
/*
|
||||
* xcetest_wait_connection_idle (xce-test-cli-ux task 3.2 remediation round
|
||||
* 1, bug-1795) -- blocks the calling thread until no worker thread
|
||||
* (running or abandoned-but-still-running) is active on `conn_key`, per
|
||||
* this module's own "at most one worker thread per connection" invariant,
|
||||
* then returns immediately.
|
||||
*
|
||||
* WHY THIS EXISTS: xcetest_run_interruptible()'s per-connection lock
|
||||
* registry above enforces that invariant BETWEEN calls on the same
|
||||
* connection -- a second call on a connection an abandoned worker still
|
||||
* holds waits (interruptibly) for that worker to finish. It does NOT by
|
||||
* itself protect connection TEARDOWN: test.c's leave_ctx() (called by
|
||||
* `up`/`quit`) tears down the connection object out from under an
|
||||
* abandoned-but-still-running worker with no wait/join of its own,
|
||||
* because leave_ctx() never calls xcetest_run_interruptible() -- it is
|
||||
* connection teardown, not a new RPC. xcecli_wakeup() (called by
|
||||
* leave_ctx() just before this function, best-effort) nudges that worker
|
||||
* to unwind faster than its own ~10s deadline, but nothing forced
|
||||
* leave_ctx() to actually wait for it -- a genuine, exploitable
|
||||
* use-after-free window (any abandoned worker still reading/writing the
|
||||
* connection object while leave_ctx()'s xcecli_stop()/xcecli_ctx_fini()
|
||||
* frees it). This function closes that gap by reusing the SAME
|
||||
* per-connection semaphore registry xcetest_run_interruptible() already
|
||||
* builds and maintains, rather than inventing a second mechanism.
|
||||
*
|
||||
* CONTRACT: looks up (or lazily creates, via the exact same internal
|
||||
* lookup xcetest_run_interruptible() itself uses) the sem_t for
|
||||
* `conn_key`, does a PLAIN BLOCKING sem_wait() on it (deliberately NOT
|
||||
* the interruptible sem_trywait()-poll variant used inside
|
||||
* xcetest_run_interruptible() -- this is a narrow, safety-critical wait
|
||||
* where correctness matters more than Ctrl-C responsiveness; it should
|
||||
* resolve quickly in the ordinary case since the caller's own
|
||||
* xcecli_wakeup() call already nudges the worker to finish sooner), then
|
||||
* immediately sem_post()s it back before returning. It does NOT hold the
|
||||
* semaphore locked on return -- the semantic this function provides is
|
||||
* "prove that, at this instant, no worker is active on this connection,"
|
||||
* not "hold this connection locked for some ongoing operation of the
|
||||
* caller's own." A caller that needs the latter must acquire and hold the
|
||||
* lock itself (which only xcetest_run_interruptible() currently does).
|
||||
*
|
||||
* A transient EINTR from sem_wait() itself (a signal delivered while
|
||||
* blocked, distinct from `errno`-based break_out signal state) is
|
||||
* retried internally rather than surfaced -- this function's whole
|
||||
* purpose is to make the wait unconditional, not skippable.
|
||||
*
|
||||
* Safe to call with any `conn_key` previously passed to
|
||||
* xcetest_set_active_connection() (or, for that matter, any key never
|
||||
* seen before -- the lazy-create path simply creates a fresh, immediately
|
||||
* uncontended semaphore for it and returns right away). Must be called
|
||||
* from the same single REPL thread this module's other entry points are
|
||||
* documented to run on (see this header's "CONCURRENCY OF THE REGISTRY
|
||||
* ITSELF" note) -- it is not itself safe to call concurrently from
|
||||
* multiple threads, though the semaphore it waits on is of course safe to
|
||||
* be concurrently sem_post()'d by a worker thread while this function
|
||||
* waits on it.
|
||||
*/
|
||||
void xcetest_wait_connection_idle(void* conn_key);
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/*
|
||||
* Test-only instrumentation, NOT part of the production API and never
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -464,6 +464,112 @@ static void test_interrupted_while_waiting_for_lock_never_spawns_worker(void)
|
|||
wait_for_abandoned_free_count(1);
|
||||
}
|
||||
|
||||
/* -- (iv) xcetest_wait_connection_idle() (task 3.2 remediation round 1,
|
||||
* bug-1795) -- deterministic proof of the exact mechanism leave_ctx() now
|
||||
* relies on to close the connection-teardown use-after-free.
|
||||
*
|
||||
* test_test_cli.c's black-box test_interrupt_then_immediate_teardown_no_uaf
|
||||
* exercises the SAME fix end-to-end through a real xce_test subprocess, but
|
||||
* the actual race window there (between an abandoned worker being woken by
|
||||
* xcecli_wakeup() and leave_ctx()'s teardown proceeding) is sub-millisecond
|
||||
* and timing-dependent -- it did not reliably reproduce as a failing test
|
||||
* even with the fix disabled in this environment (matching the original
|
||||
* round-1 review's own observation that a clean ASAN run is evidence the
|
||||
* race didn't fire on THAT run's timing, not evidence the gap doesn't
|
||||
* exist). This test instead proves xcetest_wait_connection_idle() itself,
|
||||
* in isolation, with a WIDE, deterministic timing margin (a 700ms abandoned
|
||||
* worker, not a sub-millisecond one) -- a direct, reliable RED (this
|
||||
* function returning early would fail it) / GREEN (the shipped
|
||||
* implementation passes it) demonstration of the invariant leave_ctx()
|
||||
* depends on. */
|
||||
|
||||
#define WCI_SLOW_SLEEP_MS 700
|
||||
#define WCI_MIN_WAIT_MS 400 /* comfortably below the ~700ms - \
|
||||
* SIGNAL_DELAY_MS remaining sleep once the \
|
||||
* call is abandoned, comfortably above \
|
||||
* scheduling noise if \
|
||||
* xcetest_wait_connection_idle() did NOT \
|
||||
* actually wait (that failure mode returns \
|
||||
* in ~0ms). */
|
||||
|
||||
static int wci_slow_fn(void* arg)
|
||||
{
|
||||
(void)arg;
|
||||
sleep_ms(WCI_SLOW_SLEEP_MS);
|
||||
return SLOW_FN_RC;
|
||||
}
|
||||
|
||||
static void test_wait_connection_idle_blocks_until_abandoned_worker_finishes(
|
||||
void)
|
||||
{
|
||||
static int conn_e; /* private fake connection key for this test */
|
||||
pthread_t signaler;
|
||||
struct timespec wait_start;
|
||||
long waited_ms;
|
||||
int rc2;
|
||||
|
||||
g_test_break_out = 0;
|
||||
atomic_store(&xcetest_test_abandoned_free_count, 0);
|
||||
xcetest_install_sigint(test_sigint_handler);
|
||||
xcetest_set_active_connection(&conn_e);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(
|
||||
0, pthread_create(&signaler, NULL, send_sigint_after_delay, NULL));
|
||||
|
||||
int rc = xcetest_run_interruptible(wci_slow_fn, NULL);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(-EINTR, rc);
|
||||
TEST_ASSERT_EQUAL_INT(0, pthread_join(signaler, NULL));
|
||||
|
||||
/* At this instant the abandoned worker is still running wci_slow_fn()
|
||||
* -- it has NOT yet freed its call block or released conn_e's lock
|
||||
* (interruptcall.c's abandoned path). This is exactly the production
|
||||
* race leave_ctx() closes: xcetest_wait_connection_idle() below must
|
||||
* genuinely block until that worker finishes, not return immediately. */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, atomic_load(&xcetest_test_abandoned_free_count),
|
||||
"the abandoned worker must still be running at this point for "
|
||||
"this test to actually exercise the wait -- if this fires, the "
|
||||
"worker finished implausibly fast and the timing margin above "
|
||||
"needs to grow, not shrink");
|
||||
|
||||
clock_gettime(CLOCK_MONOTONIC, &wait_start);
|
||||
xcetest_wait_connection_idle(&conn_e);
|
||||
waited_ms = elapsed_ms_since(&wait_start);
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
waited_ms >= WCI_MIN_WAIT_MS,
|
||||
"xcetest_wait_connection_idle() returned too quickly -- it must "
|
||||
"genuinely block until the abandoned worker finishes, not return "
|
||||
"immediately (this is precisely the bug-1795 use-after-free gap: "
|
||||
"a caller proceeding to tear down the connection before this "
|
||||
"point would race the still-running worker)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
1, atomic_load(&xcetest_test_abandoned_free_count),
|
||||
"the abandoned worker's cleanup must have genuinely completed by "
|
||||
"the time xcetest_wait_connection_idle() returns");
|
||||
|
||||
/* The semaphore must be left available (posted back), not held --
|
||||
* proven by a normal subsequent call on the SAME connection key
|
||||
* completing without waiting at all. Reset the fake interrupt flag
|
||||
* first -- xcetest_interrupted() is a pure, non-clearing read (see
|
||||
* xcetest_interrupt.h's CONTRACT), so the flag the earlier SIGINT set
|
||||
* is still "on" here and would otherwise spuriously abandon this call
|
||||
* too, same reasoning as every other multi-call scenario in this
|
||||
* file. */
|
||||
g_test_break_out = 0;
|
||||
clock_gettime(CLOCK_MONOTONIC, &wait_start);
|
||||
rc2 = xcetest_run_interruptible(fast_fn, NULL);
|
||||
waited_ms = elapsed_ms_since(&wait_start);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(NORMAL_FN_RC, rc2);
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
waited_ms < WCI_MIN_WAIT_MS,
|
||||
"xcetest_wait_connection_idle() must release the semaphore back "
|
||||
"immediately after proving idleness, not hold it -- a subsequent "
|
||||
"call on the same connection should not itself have to wait");
|
||||
}
|
||||
|
||||
void test_interruptcall(void)
|
||||
{
|
||||
UnitySetTestFile(__FILE__);
|
||||
|
|
@ -472,4 +578,5 @@ void test_interruptcall(void)
|
|||
RUN_TEST(test_same_connection_key_serializes_overlapping_calls);
|
||||
RUN_TEST(test_different_connection_keys_run_independently);
|
||||
RUN_TEST(test_interrupted_while_waiting_for_lock_never_spawns_worker);
|
||||
RUN_TEST(test_wait_connection_idle_blocks_until_abandoned_worker_finishes);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -151,8 +151,10 @@
|
|||
* CONCERNS in this task's status report if a future task instead needs
|
||||
* cross-file reuse.
|
||||
*/
|
||||
#include <arpa/inet.h>
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
#include <netinet/in.h>
|
||||
#include <poll.h>
|
||||
#include <pty.h>
|
||||
#include <signal.h>
|
||||
|
|
@ -160,8 +162,10 @@
|
|||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/socket.h>
|
||||
#include <sys/wait.h>
|
||||
#include <termios.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <unity/unity.h>
|
||||
|
|
@ -1434,6 +1438,465 @@ static void test_walcompare_unrecognized_flag_rejected_with_usage(void)
|
|||
clitest_buf_free(&err);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Interruptible Call Adapter end-to-end (task 3.2, Requirement 1) */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
/* clitest_start_blackhole_listener - starts a "black hole" TCP peer on
|
||||
* 127.0.0.1:`port`: it accepts every incoming connection but never reads
|
||||
* or writes a single byte on any of them, so a client that connects and
|
||||
* sends a request gets no reply, ever. This is the vehicle for exercising
|
||||
* Requirement 1's "blocked awaiting a cluster RPC reply" scenario
|
||||
* deterministically: xce_test's own client library still completes a
|
||||
* fast, successful TCP connect() (something IS listening and accepting),
|
||||
* so the actual multi-second block this test needs to interrupt is
|
||||
* exactly the one interruptcall.c's own header describes -- the blocking
|
||||
* `xceapi.c` reply wait (`cbuf_timedget`'s fixed 10s deadline), not the
|
||||
* OS-level connect() path. Returns the listener's pid (to be reaped via
|
||||
* clitest_stop_blackhole_listener()) on success, or -1 on failure. */
|
||||
static pid_t clitest_start_blackhole_listener(int port)
|
||||
{
|
||||
int lfd;
|
||||
struct sockaddr_in addr;
|
||||
int opt = 1;
|
||||
pid_t pid;
|
||||
|
||||
lfd = socket(AF_INET, SOCK_STREAM, 0);
|
||||
if (lfd < 0) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
|
||||
|
||||
memset(&addr, 0, sizeof(addr));
|
||||
addr.sin_family = AF_INET;
|
||||
addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
|
||||
addr.sin_port = htons((uint16_t)port);
|
||||
|
||||
if (bind(lfd, (struct sockaddr*)&addr, sizeof(addr)) != 0) {
|
||||
close(lfd);
|
||||
return -1;
|
||||
}
|
||||
if (listen(lfd, 16) != 0) {
|
||||
close(lfd);
|
||||
return -1;
|
||||
}
|
||||
|
||||
pid = fork();
|
||||
if (pid < 0) {
|
||||
close(lfd);
|
||||
return -1;
|
||||
}
|
||||
|
||||
if (pid == 0) {
|
||||
/* Child: accept forever, never read/write any accepted socket --
|
||||
* a deliberate silent peer, so the parent xce_test's blocking RPC
|
||||
* wait never completes on its own. */
|
||||
for (;;) {
|
||||
int cfd = accept(lfd, NULL, NULL);
|
||||
|
||||
if (cfd < 0) {
|
||||
if (errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
_exit(0);
|
||||
}
|
||||
/* Leave `cfd` open and untouched; just keep accepting more. */
|
||||
}
|
||||
}
|
||||
|
||||
close(lfd); /* the parent process only needs the child running */
|
||||
return pid;
|
||||
}
|
||||
|
||||
static void clitest_stop_blackhole_listener(pid_t pid)
|
||||
{
|
||||
if (pid > 0) {
|
||||
kill(pid, SIGKILL);
|
||||
waitpid(pid, NULL, 0);
|
||||
}
|
||||
}
|
||||
|
||||
static long clitest_elapsed_ms(const struct timespec* start)
|
||||
{
|
||||
struct timespec now;
|
||||
|
||||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
return ((long)(now.tv_sec - start->tv_sec) * 1000L)
|
||||
+ ((now.tv_nsec - start->tv_nsec) / 1000000L);
|
||||
}
|
||||
|
||||
/* test_interrupt_abandons_blocked_rpc_well_before_timeout -- Requirement
|
||||
* 1.1/1.3/1.5, this task's Required Test Scenario 1+2: a command blocked
|
||||
* on a real RPC reply (against the black-hole peer above, so the ~10s
|
||||
* cbuf_timedget deadline is genuinely reachable) is interrupted by a real
|
||||
* SIGINT; control must return to the prompt in a small fraction of that
|
||||
* 10s deadline, the command must be reported as failed (Requirement 1.3),
|
||||
* and the SAME connection/context must accept a normal, non-RPC follow-up
|
||||
* command immediately afterward (proving the context/connection were left
|
||||
* intact, Requirement 1.5), all without the session exiting. */
|
||||
static void test_interrupt_abandons_blocked_rpc_well_before_timeout(void)
|
||||
{
|
||||
char xce_test_path[PATH_MAX];
|
||||
char* const argv[] = { "xce_test", "--norc", NULL };
|
||||
clitest_session_t sess;
|
||||
clitest_buf_t out;
|
||||
clitest_buf_t err;
|
||||
int exit_code = -999;
|
||||
int port = 19990 + (int)(getpid() % 1000);
|
||||
char masters_cmd[64];
|
||||
pid_t blackhole;
|
||||
struct timespec interrupt_sent_at;
|
||||
|
||||
blackhole = clitest_start_blackhole_listener(port);
|
||||
TEST_ASSERT_TRUE_MESSAGE(blackhole > 0,
|
||||
"must be able to start the black-hole TCP "
|
||||
"listener this test blocks against");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, resolve_xce_test_path(xce_test_path, sizeof(xce_test_path)),
|
||||
"must be able to resolve the sibling xce_test binary's path");
|
||||
|
||||
clitest_buf_init(&out);
|
||||
clitest_buf_init(&err);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, clitest_spawn(&sess, xce_test_path, argv),
|
||||
"clitest_spawn must launch xce_test behind a real pty");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce>", false, 5000),
|
||||
"expected the initial 'xce>' interactive prompt on stdout");
|
||||
|
||||
snprintf(masters_cmd, sizeof(masters_cmd), "set masters 127.0.0.1:%d",
|
||||
port);
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, masters_cmd));
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "client"));
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce/client>", false, 5000),
|
||||
"entering the client context must succeed even though its "
|
||||
"connection target is a silent black hole (connect setup does not "
|
||||
"itself block)");
|
||||
|
||||
/* `hello` issues exactly one blocking xcecli_hello() RPC (interrupt-
|
||||
* call-adapter-wrapped, task 3.2) -- against a peer that never
|
||||
* replies, this call would otherwise block for ~10s. */
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "hello"));
|
||||
|
||||
/* Give the command time to actually reach the blocking RPC wait
|
||||
* (connect + send are fast; this margin is generous relative to
|
||||
* both) before sending the interrupt, so this test genuinely
|
||||
* exercises the "interrupt a wait already in flight" path rather
|
||||
* than racing dispatch itself. */
|
||||
usleep(300000); /* 300ms */
|
||||
|
||||
/* Reset the capture buffers before sending the interrupt: "xce/client>"
|
||||
* already appears once in `out` from when this session first entered
|
||||
* the client context above, so waiting for that SAME substring against
|
||||
* the accumulated buffer would trivially "succeed" on the very first
|
||||
* poll regardless of whether a NEW prompt is genuinely printed after
|
||||
* the interrupt -- discarding prior output makes the wait below (and
|
||||
* the elapsed-time measurement it gates) observe a real, fresh event. */
|
||||
clitest_buf_free(&out);
|
||||
clitest_buf_free(&err);
|
||||
clitest_buf_init(&out);
|
||||
clitest_buf_init(&err);
|
||||
|
||||
clock_gettime(CLOCK_MONOTONIC, &interrupt_sent_at);
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_signal(&sess, SIGINT));
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce/client>", false, 3000),
|
||||
"the client-context prompt must reappear after the interrupt, well "
|
||||
"before the RPC's own ~10s deadline");
|
||||
|
||||
TEST_ASSERT_LESS_THAN_MESSAGE(
|
||||
2000L, clitest_elapsed_ms(&interrupt_sent_at),
|
||||
"control must return to the prompt in a small fraction of the RPC's "
|
||||
"~10s deadline (observed well under 2s in practice, given the "
|
||||
"adapter's documented 50ms poll interval)");
|
||||
|
||||
/* Requirement 1.5: the SAME connection/context must accept a normal
|
||||
* follow-up command immediately -- `echo` makes no RPC call at all,
|
||||
* so an immediate, correct reply here proves the session (and its
|
||||
* client context) is genuinely alive and responsive, not merely
|
||||
* appearing to be so. */
|
||||
TEST_ASSERT_EQUAL_INT(
|
||||
0, clitest_send_line(&sess, "echo cli_after_interrupt_marker"));
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "cli_after_interrupt_marker",
|
||||
false, 3000),
|
||||
"a follow-up command on the same connection/context must execute "
|
||||
"normally right after the interrupt");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "quit"));
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, clitest_wait_for_exit(&sess, &out, &err, 5000, &exit_code),
|
||||
"xce_test did not exit within the harness's safety timeout after "
|
||||
"'quit'");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
1, exit_code,
|
||||
"the interrupted 'hello' must be recorded as a failed command "
|
||||
"(Requirement 1.3), matching this session's overall exit code");
|
||||
|
||||
clitest_kill_and_reap(&sess);
|
||||
clitest_close(&sess);
|
||||
clitest_stop_blackhole_listener(blackhole);
|
||||
clitest_buf_free(&out);
|
||||
clitest_buf_free(&err);
|
||||
}
|
||||
|
||||
/* test_interrupt_at_empty_prompt_discards_line_without_exiting --
|
||||
* Requirement 1.2/1.4, this task's Required Test Scenario 3: a Ctrl-C with
|
||||
* nothing in flight (a partially-typed, unsubmitted line at the prompt)
|
||||
* must discard that line and redisplay the prompt, never exit the session
|
||||
* as a side effect -- proven by sending a real, valid command right after
|
||||
* and confirming it executes normally. */
|
||||
static void test_interrupt_at_empty_prompt_discards_line_without_exiting(void)
|
||||
{
|
||||
char xce_test_path[PATH_MAX];
|
||||
char* const argv[] = { "xce_test", "--norc", NULL };
|
||||
clitest_session_t sess;
|
||||
clitest_buf_t out;
|
||||
clitest_buf_t err;
|
||||
int exit_code = -999;
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, resolve_xce_test_path(xce_test_path, sizeof(xce_test_path)),
|
||||
"must be able to resolve the sibling xce_test binary's path");
|
||||
|
||||
clitest_buf_init(&out);
|
||||
clitest_buf_init(&err);
|
||||
|
||||
/* disable_echo=false (clitest_spawn_ex, not plain clitest_spawn): this
|
||||
* test needs libedit's REAL interactive line-editing state (in
|
||||
* particular, that a partial, uncommitted line is genuinely discarded
|
||||
* from its edit buffer on the EINTR path) -- clitest_spawn()'s default
|
||||
* echo-off session makes THIS project's linked libedit fall back to a
|
||||
* silent/non-interactive input mode (task 2.3's own documented finding,
|
||||
* see clitest_spawn_ex()'s header comment) that does NOT discard a
|
||||
* pending partial line across an interrupted read(), so a truly
|
||||
* interactive echo-on session is required to observe the correct
|
||||
* (and, empirically confirmed against a genuine terminal-like pty,
|
||||
* actually-happening) discard behavior here. */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, clitest_spawn_ex(&sess, xce_test_path, argv, false),
|
||||
"clitest_spawn_ex must launch xce_test behind a real pty with echo "
|
||||
"on");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce>", false, 5000),
|
||||
"expected the initial 'xce>' interactive prompt on stdout");
|
||||
|
||||
/* A partial, unsubmitted line -- nothing in flight, no command has
|
||||
* been dispatched yet. */
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_partial(&sess, "ech"));
|
||||
usleep(300000); /* let libedit's readline() actually be blocked on it */
|
||||
|
||||
/* Reset the capture buffers before sending the interrupt: "xce>"
|
||||
* already appears once in `out` from the session's initial prompt
|
||||
* above, so waiting for that SAME substring against the accumulated
|
||||
* buffer would trivially "succeed" on the very first poll regardless
|
||||
* of whether a NEW prompt is genuinely printed after the interrupt. */
|
||||
clitest_buf_free(&out);
|
||||
clitest_buf_free(&err);
|
||||
clitest_buf_init(&out);
|
||||
clitest_buf_init(&err);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_signal(&sess, SIGINT));
|
||||
|
||||
/* Wait for the REDISPLAYED prompt (main()'s new break_out-then-printf
|
||||
* path) before sending anything else -- this is what makes the next
|
||||
* step race-free: only once "xce>" has genuinely reappeared do we know
|
||||
* the interrupted readline() call has already returned and a BRAND
|
||||
* NEW readline() call, with a fresh/empty edit buffer, is active.
|
||||
* Sending the next line immediately after clitest_send_signal()
|
||||
* (without waiting for this) would race the child's own signal
|
||||
* delivery/handling latency -- if that next line's bytes arrive before
|
||||
* the child has actually re-entered its blocking read(), they get
|
||||
* queued as more input to the STILL-PENDING original read() instead of
|
||||
* starting a fresh one, undetectably from this harness's side (this
|
||||
* was empirically confirmed while building this test: it manifests as
|
||||
* a corrupted "echecho ..." command whose raw, still-echoed-back typed
|
||||
* text nonetheless contains the search marker, making the naive
|
||||
* text-search assertion below pass despite the underlying command
|
||||
* actually failing -- caught instead by the exit-code assertion at the
|
||||
* end, which is what motivated this explicit wait). */
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce>", false, 3000),
|
||||
"the prompt must be redisplayed after Ctrl-C at an empty/partial "
|
||||
"prompt, before this test sends anything further");
|
||||
|
||||
/* Requirement 1.4: never exit as a side effect of a single Ctrl-C.
|
||||
* Send a real, valid command and confirm it executes normally --
|
||||
* this is the semantic, redraw-independent proof this session is
|
||||
* still alive and its discarded partial line did not leak into the
|
||||
* next command's input (e.g. becoming "echecho ..."). */
|
||||
TEST_ASSERT_EQUAL_INT(
|
||||
0, clitest_send_line(&sess, "echo cli_still_alive_marker"));
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "cli_still_alive_marker",
|
||||
false, 3000),
|
||||
"a real command sent right after Ctrl-C at an empty/partial prompt "
|
||||
"must execute normally, proving the session did not exit");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "quit"));
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, clitest_wait_for_exit(&sess, &out, &err, 5000, &exit_code),
|
||||
"xce_test did not exit within the harness's safety timeout after "
|
||||
"'quit'");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, exit_code,
|
||||
"a session with no genuinely failing command must still exit 0");
|
||||
|
||||
clitest_kill_and_reap(&sess);
|
||||
clitest_close(&sess);
|
||||
clitest_buf_free(&out);
|
||||
clitest_buf_free(&err);
|
||||
}
|
||||
|
||||
/* test_interrupt_then_immediate_teardown_no_uaf -- xce-test-cli-ux task 3.2
|
||||
* remediation round 1 regression guard (bug-1795). Interrupts a blocked RPC
|
||||
* exactly like test_interrupt_abandons_blocked_rpc_well_before_timeout
|
||||
* above, but where that test deliberately WAITS for the interrupted prompt
|
||||
* to redisplay before sending anything else (so ITS OWN elapsed-time
|
||||
* measurement is race-free), THIS test does the opposite on purpose: it
|
||||
* sends ".." (the "go up a context" command, bound to cmd_up) IMMEDIATELY
|
||||
* after the SIGINT, with no settling wait at all -- racing leave_ctx()'s
|
||||
* connection teardown (invoked via cmd_up) against
|
||||
* whatever worker thread was abandoned by the interrupt, which may still
|
||||
* be running for up to its own ~10s deadline (interruptcall.c's
|
||||
* xcetest_run_interruptible()/xcecli_wakeup()). This is exactly the crux
|
||||
* the round-1 review flagged as a genuine use-after-free: before this
|
||||
* remediation, leave_ctx() called xcecli_stop()/xcecli_ctx_fini() (which
|
||||
* unconditionally frees the connection's api_ctx_t, src/lib/client.c)
|
||||
* right after a merely best-effort xcecli_wakeup() nudge, with nothing
|
||||
* forcing it to actually wait for that abandoned worker to finish
|
||||
* touching the connection first. The fix under test is
|
||||
* xcetest_wait_connection_idle() (interruptcall.c/.h), which leave_ctx()
|
||||
* now calls, right before teardown, to block until the SAME per-connection
|
||||
* semaphore registry xcetest_run_interruptible() itself uses proves no
|
||||
* worker is active on this connection.
|
||||
*
|
||||
* The actual proof this test relies on is running under ASAN (this
|
||||
* project's `asan` build profile) with a completely clean stderr -- a
|
||||
* genuine heap-use-after-free is caught by ASAN the instant it happens,
|
||||
* independent of anything this test does afterward, so the exit-code and
|
||||
* "no AddressSanitizer text anywhere in stderr" assertions below are what
|
||||
* actually catch a regression; a merely-non-hanging, prompt-reappeared
|
||||
* session alone would not have caught the original bug on a timing-lucky
|
||||
* run. */
|
||||
static void test_interrupt_then_immediate_teardown_no_uaf(void)
|
||||
{
|
||||
char xce_test_path[PATH_MAX];
|
||||
char* const argv[] = { "xce_test", "--norc", NULL };
|
||||
clitest_session_t sess;
|
||||
clitest_buf_t out;
|
||||
clitest_buf_t err;
|
||||
int exit_code = -999;
|
||||
int port = 19990 + (int)(getpid() % 1000) + 1;
|
||||
char masters_cmd[64];
|
||||
pid_t blackhole;
|
||||
|
||||
blackhole = clitest_start_blackhole_listener(port);
|
||||
TEST_ASSERT_TRUE_MESSAGE(blackhole > 0,
|
||||
"must be able to start the black-hole TCP "
|
||||
"listener this test blocks against");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, resolve_xce_test_path(xce_test_path, sizeof(xce_test_path)),
|
||||
"must be able to resolve the sibling xce_test binary's path");
|
||||
|
||||
clitest_buf_init(&out);
|
||||
clitest_buf_init(&err);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, clitest_spawn(&sess, xce_test_path, argv),
|
||||
"clitest_spawn must launch xce_test behind a real pty");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce>", false, 5000),
|
||||
"expected the initial 'xce>' interactive prompt on stdout");
|
||||
|
||||
snprintf(masters_cmd, sizeof(masters_cmd), "set masters 127.0.0.1:%d",
|
||||
port);
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, masters_cmd));
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "client"));
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce/client>", false, 5000),
|
||||
"entering the client context must succeed even though its "
|
||||
"connection target is a silent black hole");
|
||||
|
||||
/* Same as the sibling test above: issue the blocking `hello` RPC
|
||||
* against the black hole, then give it a moment to actually reach the
|
||||
* blocking wait before interrupting -- this margin is only about
|
||||
* getting the WORKER thread genuinely blocked inside the RPC, not
|
||||
* about settling anything after the interrupt (see below, which is
|
||||
* the entire point of this test). */
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "hello"));
|
||||
usleep(300000); /* 300ms */
|
||||
|
||||
/* Reset capture buffers before the interrupt, same rationale as the
|
||||
* sibling test above ("xce/client>" already appears once from
|
||||
* entering the context). */
|
||||
clitest_buf_free(&out);
|
||||
clitest_buf_free(&err);
|
||||
clitest_buf_init(&out);
|
||||
clitest_buf_init(&err);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_signal(&sess, SIGINT));
|
||||
|
||||
/* THE POINT OF THIS TEST: send the "go up a context" command ("..",
|
||||
* bound to cmd_up -- see test.c's command tables) IMMEDIATELY, with NO
|
||||
* settling wait for the interrupted prompt to reappear first (unlike
|
||||
* the sibling test above). This is deliberately the adversarial timing
|
||||
* for the use-after-free the review found -- leave_ctx() (invoked
|
||||
* here via cmd_up) must genuinely BLOCK, via
|
||||
* xcetest_wait_connection_idle(), until the abandoned worker thread
|
||||
* has actually finished with this connection before
|
||||
* xcecli_stop()/xcecli_ctx_fini() free it, rather than racing it. */
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, ".."));
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "xce>", false, 5000),
|
||||
"the top-level 'xce>' prompt must reappear after '..' tears down "
|
||||
"the interrupted client connection, with no crash in between");
|
||||
|
||||
/* A real follow-up command on the brand-new top-level context proves
|
||||
* the whole process (not merely the pty) is still alive and
|
||||
* undamaged after the teardown race above. */
|
||||
TEST_ASSERT_EQUAL_INT(
|
||||
0, clitest_send_line(&sess, "echo cli_after_teardown_marker"));
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
clitest_wait_for_text(&sess, &out, &err, "cli_after_teardown_marker",
|
||||
false, 3000),
|
||||
"a follow-up command after the interrupt-then-immediate-teardown "
|
||||
"race must execute normally");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, clitest_send_line(&sess, "quit"));
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, clitest_wait_for_exit(&sess, &out, &err, 5000, &exit_code),
|
||||
"xce_test did not exit within the harness's safety timeout after "
|
||||
"'quit'");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
exit_code >= 0,
|
||||
"the session must exit normally, never via a fatal signal (e.g. "
|
||||
"ASAN's SIGABRT/SIGSEGV on a detected heap-use-after-free)");
|
||||
TEST_ASSERT_NULL_MESSAGE(
|
||||
strstr(err.data, "AddressSanitizer"),
|
||||
"no AddressSanitizer diagnostic (e.g. heap-use-after-free) may "
|
||||
"appear anywhere in stderr across this whole session");
|
||||
|
||||
clitest_kill_and_reap(&sess);
|
||||
clitest_close(&sess);
|
||||
clitest_stop_blackhole_listener(blackhole);
|
||||
clitest_buf_free(&out);
|
||||
clitest_buf_free(&err);
|
||||
}
|
||||
|
||||
void test_test_cli(void)
|
||||
{
|
||||
UnitySetTestFile(__FILE__);
|
||||
|
|
@ -1446,4 +1909,7 @@ void test_test_cli(void)
|
|||
RUN_TEST(test_qctrl_flag_forms_short_and_long_both_accepted);
|
||||
RUN_TEST(test_migrated_commands_sample_reject_unrecognized_flag);
|
||||
RUN_TEST(test_walcompare_unrecognized_flag_rejected_with_usage);
|
||||
RUN_TEST(test_interrupt_abandons_blocked_rpc_well_before_timeout);
|
||||
RUN_TEST(test_interrupt_at_empty_prompt_discards_line_without_exiting);
|
||||
RUN_TEST(test_interrupt_then_immediate_teardown_no_uaf);
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue