feat(xce-test-cli-ux): wrap every blocking RPC with the interrupt adapter

Routes all 26 blocking xcecli_* call sites across ~24 commands through
xcetest_run_interruptible(), wires xcetest_set_active_connection() at
execute_line()'s dispatch point (active_ctx->ctx, read fresh per
command), and fixes main()'s REPL loop to distinguish a real EOF from
an EINTR-interrupted readline() so Ctrl-C at an empty prompt discards
the line instead of exiting the session.

Live-discovered and fixed bug-1794: fill_job_alloc() never initializes
xce_job_alloc_t's nres_shared/allocs_shared fields -- harmless as stack
garbage before this task, genuine heap garbage once moved into a
heap-allocated ctx block (a necessary consequence of the adapter's
heap-only precondition). Fixed by switching all ~26 call-context
allocations from malloc to calloc uniformly. Also fixed an argv-aliasing
hazard: cmd_hello/cmd_jobckpt's -m values aliased optarg, which
execute_line() frees immediately after return including on the
abandoned path -- now strdup'd into the heap block.

Also closed a safety-critical gap this task's own design introduced
(bug-1795): before this task the REPL was fully synchronous, so
leave_ctx() (called by up/quit) could never run while an RPC was
outstanding. An abandoned worker can now persist past command
completion, so leave_ctx() tearing down the connection could
use-after-free it. Fixed via xcetest_wait_connection_idle(), which
blocks on the same per-connection semaphore the adapter already
tracks, then releases it, proving no worker is active before teardown.
Proven with a deterministic white-box RED->GREEN test, since the race
itself is too timing-dependent for a black-box test to reliably force.

1361/1361 across the full controller suite (+2), 0 regressions.

Task 3.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.2, 1.3, 1.4, 1.5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
This commit is contained in:
Rongsong Shen 2026-09-09 20:58:18 +00:00
parent f152c59e75
commit 45120af2cc
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
7 changed files with 1887 additions and 70 deletions

View file

@ -21536,6 +21536,61 @@
],
"occurrences": 1,
"last_seen": "2026-09-09T16:24:19Z"
},
{
"id": "bug-1794",
"timestamp": "2026-09-09T19:03:51Z",
"error_message": "xce_test cmd_jobrun crashes (ASAN allocation-size-too-big abort in alloc_packet, requested size ~0xffffffffc3c3c4b0) when run live against a real controller, after xce-test-cli-ux task 3.2 wrapped its xcecli_jobdispatch() call through the interruptible-call adapter (heap-allocated ctx block instead of a stack-local xce_job_alloc_t).",
"file": "src/controller/testsuite/test.c (cmd_jobrun, fill_job_alloc, jobrun_call_t/jobrun_thunk)",
"root_cause": "PRE-EXISTING, latent: fill_job_alloc() (test.c) only ever sets xce_job_alloc_t.nhosts/.allocs -- it never initializes the struct's other two fields, nres_shared and allocs_shared (xce/types.h). Before task 3.2, xce_job_alloc_t jalloc was a plain stack-local in cmd_jobrun, so those two never-set fields held whatever incidental value that stack frame happened to have -- harmless in practice, apparently usually small/zero. Task 3.2 moved this struct into a heap-allocated jobrun_call_t block (required by the interruptible-call adapter's contract: fn must only touch heap-owned memory, never the caller's stack, since an abandoned call's worker thread can outlive the caller's own stack frame) allocated via plain malloc(), which leaves ALL bytes as genuine heap garbage -- confirmed live (real single-node deploy, jobrun -j <jobid> -m worker-1:1): the garbage nres_shared/allocs_shared corrupted xce_jobdispatch_req's wire-size computation in lib/request.c, producing a multi-exabyte allocation request that ASAN correctly aborted on.",
"fix": "FIXED (2026-09-09, xce-test-cli-ux task 3.2). Every per-call-site heap block (all ~26 <cmd>_call_t typedefs test.c's interrupt-wrapping introduces) now allocates via calloc(1, sizeof(*call)) instead of malloc(sizeof(*call)), zero-initializing every field uniformly rather than auditing each wrapped request struct for a similarly-never-fully-set field one at a time -- the same defensive default an accidentally-zero stack frame was already silently relied on for. Verified by re-running the exact live repro (jobrun against a real single-node deploy) after the fix: completes normally with no ASAN error.",
"tags": [
"controller",
"testsuite",
"xce_test",
"cmd_jobrun",
"fill_job_alloc",
"uninitialized-memory",
"heap-allocation",
"asan",
"interruptcall",
"xce-test-cli-ux",
"task-3.2"
],
"related_bugs": [
"bug-1791"
],
"occurrences": 1,
"last_seen": "2026-09-09T19:03:51Z"
},
{
"id": "bug-1795",
"timestamp": "2026-09-09T20:10:00Z",
"error_message": "leave_ctx() (test.c, called by cmd_up/cmd_quit) could free a connection's api_ctx_t (xcecli_stop()+xcecli_ctx_fini(), src/lib/client.c) while an abandoned worker thread from a prior Ctrl-C-interrupted RPC on that SAME connection was still running and still reading/writing it -- a genuine use-after-free, found by kiro-review during xce-test-cli-ux task 3.2's own round-1 review (not reproduced live/under ASAN on that run; identified structurally by tracing the code paths).",
"file": "src/controller/testsuite/test.c (leave_ctx), src/controller/testsuite/interruptcall.c/.h (xcetest_wait_connection_idle, new), src/lib/client.c (xcecli_wakeup, xcecli_stop, xcecli_ctx_fini, read-only for this fix)",
"root_cause": "Task 3.2's own interruptible-call adapter (tasks 1.4/1.5, interruptcall.c) runs every blocking xce_* RPC on a detached worker thread; when the operator interrupts (Ctrl-C) a blocked call, xcetest_run_interruptible() returns -EINTR to the REPL immediately, but the worker thread it spawned keeps running -- possibly for up to its own ~10s cbuf_timedget deadline -- reading/writing whatever connection state its ctx closes over. Before this task, the REPL was fully synchronous, so leave_ctx() (test.c, shared by cmd_up's '..' and cmd_quit) could never run while an RPC was outstanding on that connection; task 3.2's adapter broke that invariant by letting control return to the prompt WHILE a worker is still abandoned-but-running. leave_ctx() already called xcecli_wakeup(active_ctx->ctx, -EINTR) before tearing the connection down, but that is only a best-effort nudge (it wakes the underlying condvar wait sooner than the worker's own deadline) -- nothing forced leave_ctx() to actually WAIT for that woken worker to finish before proceeding to xcecli_stop() (tears down the live socket + failover lock) and xcecli_ctx_fini() (unconditionally free()s the whole api_ctx_t). If the operator's very next action after abandoning a hung command is 'up' or 'quit' -- an entirely ordinary thing to do -- the still-running worker can be mid-read/write on memory that just got freed out from under it. This directly contradicts the Interruptible Call Adapter's own documented invariant ('at most one worker thread -- running or abandoned-but-still-running -- is ever active per connection at a time', interruptcall.h/design.md) -- that invariant was enforced only BETWEEN xcetest_run_interruptible() calls (via the per-connection semaphore registry, tasks 1.4/1.5), never against connection TEARDOWN, since leave_ctx() never itself goes through xcetest_run_interruptible().",
"fix": "FIXED (2026-09-09, xce-test-cli-ux task 3.2 remediation round 1). Added xcetest_wait_connection_idle(void* conn_key) to interruptcall.c/.h, reusing the EXACT SAME per-connection semaphore registry lookup (xcetest_conn_lock_for()) that xcetest_run_interruptible() already builds and maintains -- not a new/parallel mechanism. It does a plain BLOCKING sem_wait() (deliberately not the interruptible sem_trywait()-poll variant used elsewhere -- this is a narrow, safety-critical wait where correctness outranks Ctrl-C responsiveness, and it resolves quickly in practice since xcecli_wakeup() already nudges the worker) then immediately sem_post()s the semaphore back rather than holding it -- the semantic is 'prove no worker is active on this connection right now', not 'hold the connection locked'. leave_ctx() now calls xcetest_wait_connection_idle(ctx->ctx) right after the existing xcecli_wakeup() call and right before xcecli_stop()/xcecli_ctx_fini() -- since both cmd_up and cmd_quit call this SAME shared leave_ctx(), both paths are covered by a single call site. New black-box regression test test_interrupt_then_immediate_teardown_no_uaf (test_test_cli.c, reuses the existing clitest_start_blackhole_listener() harness helper) interrupts a blocked hello RPC against a black-hole TCP peer and sends 'up' IMMEDIATELY afterward -- with no settling wait for the interrupted prompt to redisplay first, unlike the sibling interrupt test -- to race leave_ctx()'s teardown against the abandoned worker on purpose; asserts the session exits only via a normal (non-signal) status and that stderr never contains an 'AddressSanitizer' diagnostic anywhere across the whole session. Verified under the asan build profile.",
"tags": [
"controller",
"testsuite",
"xce_test",
"leave_ctx",
"cmd_up",
"cmd_quit",
"interruptcall",
"use-after-free",
"asan",
"concurrency",
"semaphore",
"xce-test-cli-ux",
"task-3.2",
"kiro-review-finding"
],
"related_bugs": [
"bug-1794"
],
"occurrences": 1,
"last_seen": "2026-09-09T20:10:00Z"
}
]
}