myxce/.kiro/specs/xce-test-cli-ux/design.md

Design Document: xce-test-cli-ux

Overview

Purpose: This feature makes xce_test's command-line surface predictable and fail-loud for the engineers who drive controller/worker/scheduler/HA testing through it — consistent flag validation across every command, a Ctrl-C that actually cancels a stuck RPC, structured output for scripted callers, complete per-command help, a safety gate on destructive actions, and in-session script composition via source. It also fixes two known display/default bugs (bug-1711, bug-1557) found during the CLI-UX audit that motivated this spec.

Users: SDETs and engineers running interactive soak/E2E sessions against live or degraded HA clusters, and script authors piping xce_test non-interactively in automated test harnesses.

Impact: Adds a small set of cross-cutting internal modules (flag parsing/validation, an interruptible-RPC wrapper, a confirmation gate, a JSON list formatter) to the existing single-file xce_test REPL client (src/controller/testsuite/test.c, ~4550 lines), migrates ~17 commands' flag parsing to a shared pattern, and corrects two existing defects — without changing wire protocol, RPC semantics, or any existing short flag's meaning.

Goals

• Every flag-accepting command uniformly rejects an unrecognized/malformed flag and offers a long-form equivalent, named consistently for the same concept across commands.
• Ctrl-C reliably cancels a blocked RPC wait or the current input line — never the whole session, never silently doing nothing.
• The five destructive commands (jobkill, shutdown, qctrl -i/-c, hostctrl -c, upgradeconfig) require confirmation, safely bypassable per-command or per-session for scripts.
• help <cmd> is complete (description, full flag docs, example, context/related-command notes) and consistent for every command.
• workerlist/joblist/reslist support a JSON output mode with the same fields as their tables, unabbreviated.
• source <file> composes reusable command sequences, safely (cycle-detected).
• bug-1711's display defects and bug-1557's stale join default are corrected.

Non-Goals

• No change to controller/worker/scheduler wire protocol or RPC semantics.
• No xce_test → xce_harness rename.
• No full curses/TUI mode, no color/theming.
• No new .xce_testrc auto-executed startup-file feature (already shipped by the separate xce-test-rc-file spec).
• No renaming or reassignment of any existing short flag's meaning.
• No change to the auto-variable/exit-code-verdict/shell-escape mechanisms already shipped by xce-test-e2e-primitives.

Boundary Commitments

This Spec Owns

• xce_test's own CLI surface: flag parsing, validation, and typo-suggestion for every command that accepts flags (Flag Parsing Layer).
• Client-side Ctrl-C cancellation of a blocked RPC wait (Interruptible Call Adapter) — cancellation is local only; it does not cancel the in-flight request server-side. Owns serializing any two calls that share a connection (including a new call issued right after a prior one was abandoned) so a connection's single-outstanding-reply state is never touched concurrently.
• The destructive-command confirmation gate, its five-command enumeration, its bypass flag, and its session-scoped toggle (Confirmation Gate).
• JSON output for workerlist/joblist/reslist (JSON List Formatter).
• Command-name tab completion scoped to the active context (Tab Completion).
• help <cmd> content and generation, including the active-context-only not-found behavior (Help Renderer).
• The new source <file> command, its cycle detection, and its explicit missing-file error (Source Command).
• The specific display/default corrections enumerated in Requirements 11 and 12 (bug-1711, bug-1557).
• Process-level -h/--help/--version (Process Entry).

Out of Boundary

• libxce's blocking RPC primitives (cbuf_timedget, xceapi.c's xce_* functions) — not modified. Requirement 1 is achieved entirely client-side, inside test.c/testsuite.
• Controller/worker/scheduler wire protocol and RPC semantics — unaffected by any part of this design.
• rcfile.c's own path-resolution and startup-time missing-file behavior — untouched; source reuses rcfile_load as-is and layers its own missing-file/cycle behavior on top, in the new command, not in rcfile.c.
• Any existing short flag's accepted values, default, or meaning — never reassigned (only a long-form alias is added alongside).
• The exit-code/auto-variable verdict machinery and shell-escape mechanism from xce-test-e2e-primitives — read-only dependency, not modified.
• Authentication/connection setup to the controller/worker/scheduler — unchanged.
• The controller daemon's HA/persistence/consensus invariants (src/controller/CLAUDE.md's TLA+ mapping, election safety, commit durability, etc.) — checked and found not applicable: xce_test is a client binary linked against libxce, not the controller daemon process, and no part of this design touches daemon code, WAL, or replication state.

Allowed Dependencies

• cJSON (via the already-linked struct2json_dep) — for the JSON List Formatter. No new external dependency.
• libedit's completion API (already linked) — for Tab Completion.
• POSIX threads (already transitively linked via libxce) — for the Interruptible Call Adapter.
• rcfile_load() (existing, unmodified) — called directly by the Source Command.
• cmd_set's existing settings mechanism — extended (not replaced) by the Confirmation Gate's toggle.

Revalidation Triggers

• Any change to libxce's blocking-call deadline (cbuf_deadline_10s) changes the worst-case lifetime of an abandoned worker thread this design assumes.
• Any change to rcfile_load's signature or missing-file semantics invalidates the Source Command's reuse contract.
• A new cmd_* command added after this spec ships must adopt the flagspec_t pattern to stay covered by Requirements 2/6/8/10 — otherwise it silently falls outside these guarantees.
• Any future decision to reassign an existing short flag's meaning (explicitly deferred, not this spec) would invalidate the flagspec_t tables' short/long/concept mappings built here.

Architecture

Existing Architecture Analysis

• Flag parsing is three uncoordinated patterns today: a plain-getopt() loop (~16 commands, each with its own hand-written switch/default), positional-only argument checking (offline/diagnostic commands), and one fully hand-rolled long-flag string-compare loop (cmd_walcompare). No shared helper exists for parsing, rejection, or usage rendering — this disconnect between parsing code and the hand-authored USAGE_* string table is the direct root cause of bug-1711's drift.
• Interrupt handling: SIGINT → sigbreak() sets a static break_out flag, read in exactly one place (cmd_hello's repeat loop). Every blocking xce_* RPC call waits on pthread_cond_timedwait with a fixed 10-second deadline (cbuf_deadline_10s/cbuf_timedget, in libxce) — a wait a signal-flag cannot interrupt.
• REPL input: libedit (readline()/add_history() only), no completion or key-binding customization registered.
• Session settings: a single global settings[] array shared (not actually per-context) by all four command_ctx_t instances; cmd_set's -e/+e/continue-on-error toggle is the existing pattern for a session-scoped boolean.
• Serialization: cJSON is already linked (via struct2json_dep) and used narrowly for job exdata blobs; Cap'n Proto is also linked but exclusively for wire-protocol/checkpoint-blob decoding, not general CLI output.
• Script loading: rcfile_load(path, runner) is a generic, already-reusable "read a file line by line, execute each via a callback" primitive with no cycle protection and a silent-success policy on a missing file (correct for its one existing caller, the startup rc file).

Architecture Pattern & Boundary Map

graph TB
    User[Operator or Script]
    Readline[Readline Loop]
    Dispatch[Command Dispatcher]
    FlagSpec[Flag Parsing Layer]
    Confirm[Confirmation Gate]
    CmdFunc[Command Handler]
    Interrupt[Interruptible Call Adapter]
    Json[JSON List Formatter]
    Source[Source Command]
    RcFile[Rcfile Loader]
    LibXce[Libxce Rpc Client]
    Help[Help Renderer]

    User --> Readline
    Readline --> Dispatch
    Dispatch --> FlagSpec
    FlagSpec --> Confirm
    Confirm --> CmdFunc
    CmdFunc --> Interrupt
    Interrupt --> LibXce
    CmdFunc --> Json
    Dispatch --> Source
    Source --> RcFile
    RcFile --> Dispatch
    FlagSpec --> Help
    Dispatch --> Help

Architecture Integration:

• Selected pattern: a layered CLI pipeline — Readline → Dispatcher → Flag Parsing → (Confirmation Gate, destructive commands only) → Command Handler → Interruptible Call Adapter → libxce.
• Domain/feature boundaries: Flag Parsing is one cohesive layer shared by every flag-accepting command; the Interruptible Call Adapter is a second cross-cutting layer used only where a blocking RPC occurs; the Confirmation Gate is a third, used only by the five destructive commands.
• Existing patterns preserved: hand-authored USAGE_* prose for description/example/related-command text; the single-global session-settings pattern (cmd_set); rcfile.c's file-loading primitive, reused unmodified.
• New components rationale: each addresses one specific root cause found in discovery — flag drift, a non-interruptible wait, no cycle guard, no typo-suggestion utility.
• Steering compliance: TDD (a test_<module>.c per new module, red-green-refactor), gnu17/Meson conventions, negative-errno-style (errnox.h) error propagation throughout.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
CLI / testsuite	C (gnu17), Meson	Language/build for all new and modified files	Matches existing test.c
CLI parsing	getopt_long (glibc)	Long-form flag parsing, replacing raw getopt()	Migrates ~17 commands
Structured output	cJSON (already linked via struct2json_dep)	Requirement 5 JSON mode	Adopted, no new dependency
Line editing	libedit (already linked)	Tab completion (Requirement 7)	rl_attempted_completion_function-style hook; exact API to confirm at implementation start
Concurrency	POSIX threads (already transitively linked via libxce)	Interruptible RPC wrapper (Requirement 1)	Detached worker + CAS ownership handoff
Test framework	Unity (project-wide)	New test_*.c files	#ifdef XCE_TESTING for internal-only symbols

File Structure Plan

src/controller/testsuite/ is a flat directory; new modules follow its existing one-helper-module-per-file convention (rcfile.c/h, waldump.c/h, etc.).

New Files

• flagspec.c / flagspec.h — flag-descriptor table format, getopt_long option-table builder, shared rejection + typo-suggestion helper, help-flag-block renderer. (Requirements 2, 6, 8.2, 10)
• interruptcall.c / interruptcall.h — xcetest_run_interruptible(): detached-worker-thread wrapper with CAS-based heap-ownership handoff, plus a per-connection lock registry that serializes overlapping calls (including a call issued right after a prior one was abandoned) on the same connection. (Requirement 1)
• confirm.c / confirm.h — destructive-command confirmation gate, session-scoped toggle, stderr-routed prompt, EINTR-as-decline. (Requirement 9)
• listfmt_json.c / listfmt_json.h — shared cJSON array/object helpers for the three listing commands' JSON mode. (Requirement 5)
• test_flagspec.c, test_interruptcall.c, test_confirm.c, test_listfmt_json.c — Unity unit tests, one per new module, following the existing test_rcfile.c-style convention.
• test_test_cli.c — new black-box/subprocess (pty-driven) test suite for xce_test's own CLI surface (-h/--version, help <cmd> content, tab completion, command-not-found scoping, source cycle/missing-file cases) — this surface has zero prior coverage.

Modified Files

• src/controller/testsuite/test.c — exposes a small xcetest_interrupted() accessor over the existing break_out flag for interruptcall.c to poll; replaces the existing signal(SIGINT, sigbreak) installation in main() with an explicit sigaction() call with SA_RESTART cleared, so confirm.c's confirmation-prompt read reliably observes EINTR; every flag-parsing cmd_* migrated from raw getopt()/hand-rolled parsing to flagspec_t + getopt_long; the ~19 blocking xce_* call sites wrapped via xcetest_run_interruptible(); cmd_set extended with the confirmation-toggle branch (mirroring its existing -e/+e style); cmd_source added and registered in all four command_t tables; xcetest_complete/xcetest_complete_generator registered against libedit in main(); cmd_help extended to call flagspec_render_help(); USAGE_* constants split into a generated flag-list portion and retained hand-authored prose; worker_status_str, cmd_help's not-found message, cmd_runtasks, cmd_reslist, cmd_joblist, cmd_getconf corrected per Requirement 11; cmd_join_worker's default maxmem corrected per Requirement 12; the argp doc string in main() extended to describe the in-session help command per Requirement 4.3 (4.1/4.2's -h/--help/--version exit-without-REPL behavior is already argp's default and needs no new code, only verification via test_test_cli.c).
• src/controller/meson.build — adds the four new .c files to xce_test_src, and their test_*.c counterparts plus test_test_cli.c to the appropriate test target(s).

System Flows

Interruptible RPC Call and Ctrl-C Abandonment

sequenceDiagram
    participant Op as Operator
    participant Cmd as CommandHandler
    participant Adapter as InterruptibleAdapter
    participant Lock as ConnectionLock
    participant Worker as WorkerThread
    participant Rpc as LibxceRpc

    Op->>Cmd: invoke a command with a blocking RPC
    Cmd->>Adapter: run_interruptible call
    Adapter->>Lock: acquire lock for the active connection, interruptibly
    Lock-->>Adapter: acquired
    Adapter->>Worker: spawn detached thread
    Worker->>Rpc: blocking xce call, up to ten second deadline
    Adapter->>Adapter: poll break_out every fifty milliseconds
    Op-->>Adapter: Ctrl-C sets break_out
    Adapter->>Adapter: CAS running to abandoned succeeds
    Adapter-->>Cmd: return EINTR immediately, lock stays held
    Cmd-->>Op: report command as failed, prompt returns
    Rpc-->>Worker: reply arrives later
    Worker->>Worker: CAS running to done fails, worker frees the block
    Worker->>Lock: release lock

A second command issued against the same connection after an abandonment blocks on this same lock-acquisition step — itself interruptibly — until the abandoned worker finishes and releases it, bounding the unsafe-overlap window to zero rather than letting two calls share one connection's single-outstanding-reply state concurrently. A command against a different connection is unaffected, since each connection gets its own lock.

Heap-Block Ownership Handoff

stateDiagram-v2
    [*] --> Running
    Running --> DoneUnclaimed: worker finishes, CAS succeeds
    Running --> Abandoned: caller interrupted, CAS succeeds
    DoneUnclaimed --> [*]: caller reads rc and frees the block
    Abandoned --> [*]: worker frees the block after finishing

Exactly one atomic compare-and-swap decides who frees the block, regardless of which side "wins" the race — this is what prevents a dangling-stack or double-free bug in the abandonment path (see research.md, "Interruptible RPC via detached worker thread" decision). This block-ownership handoff is orthogonal to, and independent of, the per-connection lock shown in the sequence diagram above: the lock protects the connection's exclusive use across the call's full lifetime; the CAS protects the heap block's lifetime. Both are released/freed by whichever side is left holding them once the worker finishes.

Destructive Command Confirmation Gate

flowchart TD
    Start[Destructive command invoked]
    CheckBypass{Bypass flag given}
    CheckToggle{Session toggle set to skip}
    CheckTty{Input from interactive TTY}
    Prompt[Prompt operator on stderr]
    Answer{Operator answers}
    Run[Execute command]
    NotRun[Report not run]

    Start --> CheckBypass
    CheckBypass -- yes --> Run
    CheckBypass -- no --> CheckToggle
    CheckToggle -- yes --> Run
    CheckToggle -- no --> CheckTty
    CheckTty -- no --> Run
    CheckTty -- yes --> Prompt
    Prompt --> Answer
    Answer -- yes --> Run
    Answer -- no or ctrl c --> NotRun

The TTY check evaluates the top-level session's own stdin (g_interactive), not whether the current line originates from a sourced file (Requirement 13.7) — a sourced destructive command in an otherwise-interactive session still prompts unless the toggle is set.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3, 1.4, 1.5	Ctrl-C cancels a blocked RPC or the input line, never the session	Interruptible Call Adapter, Command Dispatcher	xcetest_run_interruptible	Interrupt sequence + ownership state diagram
2.1, 2.2, 2.3, 2.4	Uniform rejection of unrecognized/malformed flags	Flag Parsing Layer	flagspec_reject	-
3.1, 3.2, 3.3, 3.4	stderr/stdout discipline	Command Dispatcher, all cmd_* handlers	-	-
4.1, 4.2, 4.3	Process-level -h/--help/--version	Process Entry	argp options	-
5.1, 5.2, 5.3, 5.4	JSON output for listing commands	JSON List Formatter	listfmt_json_*	-
6.1, 6.2, 6.3, 6.4	Long-form, concept-consistent flags	Flag Parsing Layer	flagspec_t table	-
7.1, 7.2, 7.3, 7.4	Command-name tab completion	Tab Completion	xcetest_complete*	-
8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7	Detailed, context-scoped help content	Help Renderer, Flag Parsing Layer	flagspec_render_help, cmd_help	-
9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9	Confirmation gate for destructive commands	Confirmation Gate	confirm_destructive, cmd_set toggle	Confirmation flowchart
10.1, 10.2, 10.3	Actionable flag-typo suggestions	Flag Parsing Layer	flagspec_reject (suggestion path)	-
11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8	bug-1711 display-defect fixes	Help Renderer, and worker_status_str/cmd_runtasks/cmd_reslist/cmd_joblist/cmd_getconf	-	-
12.1, 12.2	bug-1557 join maxmem default fix	cmd_join_worker, Help Renderer	-	-
13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7	source command	Source Command, rcfile_load (reused)	cmd_source	Confirmation flowchart (13.7)

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
Flag Parsing Layer	CLI Core	getopt_long parsing, rejection, suggestion, help-flag rendering from one descriptor table	2, 6, 8.2, 10	getopt_long (P0)	Service
Interruptible Call Adapter	CLI Core	Client-side cancellable wrapper around blocking xce_* calls	1	pthread (P0), libxce xceapi (P0)	Service
Confirmation Gate	CLI Core	Gate + session toggle for destructive commands	9	cmd_set (P1), Flag Parsing Layer (P1)	Service
JSON List Formatter	Output	cJSON-based structured output for the three listing commands	5	cJSON (P0)	Service
Tab Completion	REPL	libedit completion hook scoped to the active context	7	libedit (P0)	Service
Help Renderer	REPL	Assembles help <cmd> from generated flags + hand-authored prose; scopes not-found to the active context	8, 11.2, 12.2	Flag Parsing Layer (P0)	Service
Source Command	REPL	source <file>, cycle detection, reuse of rcfile_load	13	rcfile_load (P0), Confirmation Gate (P1)	Service
Process Entry	Process	argp-based -h/--help/--version	4	argp (glibc) (P0)	Service
Bug-1711 Fixes	Output	Text/formatting corrections in existing cmd_* bodies	11	—	—
Bug-1557 Fix	Data	join's maxmem default correction	12	—	—

Only components introducing a new boundary get a full detail block below; Process Entry and the two bug-fix rows are simple, localized edits to existing functions and need no further elaboration beyond the summary and File Structure Plan.

CLI Core

Flag Parsing Layer

Field	Detail
Intent	One small library, used by every flag-accepting command, that parses via getopt_long from a per-command descriptor table and uniformly rejects/suggests/documents against that same table
Requirements	2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4, 8.2, 10.1, 10.2, 10.3

Responsibilities & Constraints

• Owns the flagspec_t descriptor format, the shared rejection message format, and the shared help-flag-block format.
• Does not own each command's business-logic switch on parsed values — only the parse/reject/describe surface.
• Invariant: a flagspec_t[] table's long-form name for a repeated concept (host, jobid, command line, resource name) is identical across every command that has it — enforced by table review, not at runtime.

Dependencies

• Outbound: getopt_long (glibc) — P0
• Outbound: a small edit-distance utility introduced by this component itself (not an external dependency) — P0

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

typedef struct {
    char short_opt;           /* 0 if long-only */
    const char* long_opt;     /* stable, cross-command-consistent name */
    const char* concept;      /* e.g. "host", "jobid", "command-line" */
    int has_arg;               /* no_argument | required_argument */
    int required;               /* 1 if the command fails without it */
    const char* default_desc;  /* human text, NULL if none */
} flagspec_t;

int   flagspec_build_opts(const flagspec_t* spec, size_t n,
                           struct option* out_opts, char* out_optstring,
                           size_t out_optstring_cap);

int   flagspec_reject(const char* cmd_name, const flagspec_t* spec, size_t n,
                       const char* bad_flag, const char* usage_text);

char* flagspec_render_help(const flagspec_t* spec, size_t n);

• Preconditions: spec is a stable static array for the process lifetime; the caller sets opterr = 0 before its getopt_long loop so libc's own diagnostic never doubles up with flagspec_reject's.
• Postconditions: flagspec_reject always returns a negative value and never partially executes the command.
• Invariants: the built optstring never begins with : — missing-argument detection is flagspec_reject's job, not silent getopt behavior.

Implementation Notes

• Integration: every getopt()-using command and the one hand-rolled long-flag command (cmd_walcompare) migrate to getopt_long + a static flagspec_t[] + a case '?': return flagspec_reject(...) arm.
• Validation: test_flagspec.c covers the suggestion threshold (exactly-one-close-match / no-match / ambiguous), the optstring builder, and the help-render format, independent of any real command.
• Risks: existing short-flag collisions (-m five ways, -c six ways) mean the same short letter maps to a different concept in different commands' tables — expected, not a defect; Requirement 6.2 only requires the long-form name to be consistent per concept, not that short letters be globally unique.
• Note (bug-1704): cmd_qctrl's existing action variable has no initializer and no post-loop check that one of -a/-i/-o/-c was actually given, unlike its sibling cmd_hostctrl, which already initializes to a sentinel and rejects if unset. This migration must apply the same sentinel-and-reject pattern to cmd_qctrl — not just for correctness, but because the Confirmation Gate (Requirement 9.5) depends on reliably knowing whether -i/-c was given, which is unsafe to check against an uninitialized value.

Interruptible Call Adapter

Field	Detail
Intent	Let a command make a blocking xce_* RPC call cancellably from the operator's perspective, without modifying libxce
Requirements	1.1, 1.2, 1.3, 1.4, 1.5

Responsibilities & Constraints

• Owns the heap-block ownership protocol (the CAS state machine) and the poll interval.
• Owns a per-connection lock registry that serializes any two calls sharing the same connection — including a new call issued after a prior one on that connection was abandoned — so a connection's single-outstanding-reply state is never touched by two threads at once.
• Never touches libxce/xceapi.c internals — the wrapped call is opaque.
• Invariant: a heap-allocated call block's lifetime is resolved by exactly one atomic CAS; no code path frees it from both sides.
• Invariant: at most one worker thread — running or abandoned-but-still-running — is ever active per connection at a time.

Dependencies

• Outbound: pthread create/detach, POSIX semaphore (sem_t, one per connection, in an internal registry) — P0 (a mutex was considered and rejected during implementation review: POSIX classifies a non-owning-thread pthread_mutex_unlock() as undefined behavior for every mutex type, including the default, and this design's cross-thread release — the worker thread releasing a lock the caller thread acquired — needs a primitive with no ownership concept; see bug-1791)
• Outbound: the wrapped xce_* call, treated as opaque — P0
• Inbound: every command performing a blocking RPC (~19 call sites) — P0

Contracts: Service [x]

Service Interface

typedef int (*xcetest_blocking_fn)(void* ctx);

typedef enum { XCT_RUNNING, XCT_DONE_UNCLAIMED, XCT_ABANDONED } xcetest_call_state_t;

/* Serializes against any other in-flight or abandoned-but-still-running
 * call that shares the same active connection (keyed internally by the
 * current command_ctx_t's connection pointer, looked up in a small,
 * lazily-populated, process-lifetime registry - at most one entry per
 * context, so a linear scan suffices). Acquiring that per-connection lock
 * is itself interruptible: if break_out fires while waiting for a PRIOR
 * call's abandoned worker to release it, this returns -EINTR before ever
 * spawning fn. */
int xcetest_run_interruptible(xcetest_blocking_fn fn, void* ctx);

• Preconditions: fn touches only what ctx itself heap-owns — never caller-stack memory.
• Postconditions: on -EINTR, the caller must not touch ctx again — ownership has passed to the still-running worker thread. The per-connection lock is released by whichever side (caller or worker) is the last to finish with the call, never left held once both sides are done.
• Invariants: exactly one of {caller, worker} frees the call block, decided by which side wins the RUNNING -> {DONE_UNCLAIMED, ABANDONED} CAS.

Implementation Notes

• Integration: each of the ~19 blocking-RPC call sites gets a small per-site static thunk matching xcetest_blocking_fn; none of them need to pass a connection identity explicitly — the per-connection lock key is read internally from the currently active command_ctx_t's connection pointer (exact global/field name to confirm against test.c at implementation start, alongside the libedit completion API verification already noted in Tab Completion's Risks). A call site that ever operates on a connection other than the currently active one would break this assumption — none is known to today, but this is worth a one-line check per call site during migration.
• Integration: the lock registry's lookup/insert step needs no synchronization of its own — a new registry entry is only ever created from the single REPL thread that issues commands (detached worker threads never initiate a new top-level call; they only run the fn they were given and release the lock/free the block they were handed), so there is no concurrent-insert case to guard against.
• Validation: test_interruptcall.c uses a fake slow fn to exercise both the normal-completion and forced-abandon paths, verifying no double-free and no leak; a second test spawns two overlapping calls against the same fake connection key (one abandoned, one issued immediately after) and asserts they never run concurrently — this is the regression test for the connection-race this component's lock exists to prevent.
• Risks: none material remaining — the connection-race risk this component previously left as an unverified follow-up is now closed by the per-connection lock described above.

Confirmation Gate

Field	Detail
Intent	Gate the five destructive commands behind interactive confirmation, a per-command bypass flag, or a session-scoped toggle
Requirements	9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9

Responsibilities & Constraints

• Owns the destructive-command enumeration (9.5), the session toggle state, the stderr-routed prompt, and Ctrl-C-as-decline.
• Does not own the destructive action itself — only whether it proceeds.

Dependencies

• Outbound: cmd_set (extended with the toggle branch) — P1
• Outbound: Flag Parsing Layer (the per-command --yes bypass is itself a flagspec_t entry) — P1
• Outbound: the process's SIGINT installation (main()) — must be installed without SA_RESTART for the confirmation read to reliably observe EINTR — P0
• Inbound: cmd_jobkill, cmd_shutdown, cmd_qctrl, cmd_hostctrl, cmd_upgradeconfig — P0

Contracts: Service [x]

Service Interface

int  confirm_destructive(const char* action_description);
void confirm_set_skip(int skip);
int  confirm_is_skip_set(void);

• Preconditions: called only at the call sites enumerated by Requirement 9.5 (the enumeration's implementation is authoritative — no other command calls this). The process's SIGINT handler must already be installed without SA_RESTART (see Implementation Notes) — this is a one-time startup precondition, not something each call re-checks.
• Postconditions: a 0 return means the caller reports the command as "not run," never "failed" (9.2, 9.9).
• Invariants: the prompt always goes to stderr (9.8); the TTY check is the top-level session's g_interactive, not "is this line from a sourced file" (13.7).

Implementation Notes

• Integration: each of the five call sites checks confirm_is_skip_set() || !g_interactive || <its own --yes flag> before calling confirm_destructive.
• Integration: the existing signal(SIGINT, sigbreak) installation in main() (test.c:4457) is replaced with an explicit sigaction() call with SA_RESTART cleared. This is verified, not assumed: glibc's signal() uses BSD semantics, which historically restart an interrupted slow syscall by default — if left as-is, the confirmation prompt's blocking read could silently retry instead of returning EINTR, and Requirement 9.9 would never fire. This is a one-time, global change made once at startup; it does not affect the Interruptible Call Adapter, which never relies on EINTR (it polls break_out on a timer instead).
• Validation: test_confirm.c simulates non-TTY stdin for auto-bypass, and a piped decline/EOF/simulated-EINTR answer for the decline paths; test_test_cli.c adds a real pty-driven case that sends an actual SIGINT to the process while a confirmation prompt is displayed, proving the corrected sigaction() installation actually delivers EINTR end-to-end (not just in a simulated unit test).
• Risks: none material remaining — the EINTR-reliance risk this component previously left unverified is now closed by the explicit sigaction()/SA_RESTART fix and its end-to-end test.

Output

JSON List Formatter

Field	Detail
Intent	Shared cJSON array/object plumbing so the three listing commands can emit their existing per-row data as JSON
Requirements	5.1, 5.2, 5.3, 5.4

Responsibilities & Constraints

• Owns only the array/object lifecycle helpers — not each command's field mapping. No generic "record" abstraction is introduced (per the Simplification lens — each command's row struct already exists and is stable).

Dependencies

• Outbound: cJSON (already linked) — P0
• Inbound: cmd_workerlist, cmd_joblist, cmd_reslist — P0

Contracts: Service [x]

Service Interface

cJSON* listfmt_json_begin(void);
void   listfmt_json_add_string(cJSON* row, const char* key, const char* val);
void   listfmt_json_add_number(cJSON* row, const char* key, double val);
cJSON* listfmt_json_new_row(cJSON* array);
void   listfmt_json_finish(cJSON* array);

• Preconditions: called only when the command's --json flag (a boolean flagspec_t entry, the same long-form name across all three commands per Requirement 6.2) is set.
• Postconditions: every field present in the human-readable table row is also present in the JSON object (5.1); no truncation regardless of value length (5.3).

Implementation Notes

• Integration: each command's existing print loop gains an if (json_mode) { ... } else { existing table row } branch.
• Validation: test_listfmt_json.c covers array/object lifecycle and long-value non-truncation; per-command field mapping is covered by that command's own test or the black-box CLI suite.
• Risks: none material.

REPL

Tab Completion

Field	Detail
Intent	Register a libedit completion function scoped to the active context's command table
Requirements	7.1, 7.2, 7.3, 7.4

Responsibilities & Constraints

• Owns the completion-candidate lookup against active_ctx->commands[] only.

Dependencies

• Outbound: libedit completion API — P0 (exact signature to confirm at implementation start, see research.md Risks)

Contracts: Service [x]

Service Interface

char** xcetest_complete(const char* text, int start, int end);
char*  xcetest_complete_generator(const char* text, int state);

• Preconditions: registered only when g_interactive is true.
• Postconditions: never offers a command invalid in the active context (7.4); an unambiguous match completes in full (7.2); multiple matches list candidates without altering the input line (7.3).

Implementation Notes

• Integration: registered once in main() alongside the existing readline/add_history setup. No new header — small enough to live in test.c directly.
• Validation: covered by test_test_cli.c (pty-driven — libedit completion isn't practically unit-testable in isolation).
• Risks: libedit API confirmation (research.md Risks).

Help Renderer

Field	Detail
Intent	Compose help <cmd> from the generated flag block plus each command's hand-authored prose; scope "not found" strictly to the active context
Requirements	8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 11.2, 12.2

Responsibilities & Constraints

• Owns the assembly order (description -> flags -> example -> related-command/context notes) and the not-found message text.
• Does not own the prose content — still hand-authored per command in the USAGE_* constants, now split into a prose part and a flagspec-rendered part.

Dependencies

• Outbound: Flag Parsing Layer's flagspec_render_help — P0

Contracts: Service [x]

Service Interface

Extends the existing cmd_help in test.c — no new header. command_t gains a flagspec_t*/count pair alongside its existing usage pointer; cmd_help's per-command branch calls flagspec_render_help(cmd->flagspec, cmd->flagspec_count) and concatenates it with the command's (now flag-list-free) prose constant.

Implementation Notes

• Integration: every USAGE_* constant is edited to drop its hand-written flag-list lines (now generated) while keeping description/example/related-command prose.
• Validation: test_test_cli.c checks a representative sample of help <cmd> outputs for both a generated flag line and the retained hand-authored example, plus the corrected grammar (11.2) and join's corrected default (12.2).
• Risks: touches all 40 USAGE_* constants — mechanical but broad; sequence as its own task after the Flag Parsing Layer lands.

Source Command

Field	Detail
Intent	source <file>, reusing rcfile_load, with cycle detection and an explicit missing-file error
Requirements	13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7

Responsibilities & Constraints

• Owns the in-flight source-path stack (cycle detection) and the missing-file check.
• Does not own rcfile.c's file-reading loop or its startup-time missing-file policy — unchanged.

Dependencies

• Outbound: rcfile_load (unmodified) — P0
• Outbound: Confirmation Gate (13.7's TTY-condition clarification) — P1

Contracts: Service [x]

Service Interface

int cmd_source(int argc, char* argv[]);          /* registered command_t entry */
int  source_push_path(const char* canonical_path); /* 0 ok, -ELOOP on cycle */
void source_pop_path(void);

• Preconditions: argv[1] is a path; access(path, R_OK) must succeed or cmd_source reports the error itself (bypassing rcfile_load's silent-missing behavior).
• Postconditions: the path stack is popped exactly once per successful push, on every exit path including an aborted line under -e semantics.
• Invariants: realpath() is applied before pushing, so a cycle through symlinks or a differently-spelled relative path is still caught.

Implementation Notes

• Integration: registered as a new entry in top_commands, worker_commands, client_commands, and scheduler_commands.
• Validation: test_test_cli.c covers a two-file mutual-cycle case and a non-existent-file case as black-box scenarios (these need real files on disk).
• Risks: the path stack must be popped on every exit path, not just the success path — call out explicitly in the implementing task.

Data Models

Minimal — xce_test has no persistent storage. The only new data shapes are the C structs already shown above (flagspec_t, the interruptible-call state enum, the per-connection lock registry entry — a connection pointer paired with a sem_t, at most one per live context — and the source-path stack). No database, no schema, no migration.

Error Handling

Error Strategy

Every new or modified path returns a negative errno-style code per this project's existing C convention (errnox.h), and every error/diagnostic message is written to stderr per Requirement 3 — this design extends the existing error-reporting philosophy uniformly rather than introducing a new one.

Error Categories and Responses

• Malformed/unrecognized flag -> flagspec_reject (stderr + usage + optional suggestion, nonzero return) — Requirements 2, 10.
• Blocked RPC interrupted -> -EINTR from xcetest_run_interruptible, reported as "failed" (1.3) except at a confirmation prompt, reported as "not run" (9.9).
• Destructive command declined -> 0 return, explicit "not run" reporting (9.2).
• source missing file / cycle -> stderr error, nonzero return, no partial execution (13.2, 13.5).

Monitoring

Not applicable — xce_test is an interactive/scripted client, not a long-running service. No new logging/metrics infrastructure beyond the existing stderr diagnostics this design extends.

Testing Strategy

Unit Tests

• test_flagspec.c — optstring/option-table builder correctness; suggestion logic (exactly-one-match / no-match / ambiguous-match-suppressed); help-render format; cmd_qctrl's corrected sentinel-and-reject validation (bug-1704).
• test_interruptcall.c — normal completion path and forced-abandon path via a fake slow fn, verifying no double-free under either; two overlapping calls against the same fake connection key (one abandoned, one issued immediately after) never run concurrently, proving the per-connection lock.
• test_confirm.c — non-TTY auto-bypass; decline path; simulated-EINTR-as-decline path; toggle on/off.
• test_listfmt_json.c — array/object lifecycle; no truncation of a long field value.

Integration Tests

• cmd_source two-file mutual-cycle detection; missing-file error; nested (non-cyclic) sourcing composing correctly.
• Flag rejection end-to-end through a real migrated command (e.g. jobkill -z), confirming stderr routing + usage + nonzero exit, not just the shared helper in isolation.
• Confirmation gate against a real destructive command (qctrl -c) with the session toggle flipped, confirming no prompt appears and the command executes.

E2E / Black-box Tests (test_test_cli.c, subprocess/pty-driven — the only viable vehicle for these)

• -h/--help/--version print and exit without entering the REPL (Requirement 4).
• Tab completion at the prompt: partial command name + Tab yields the expected single completion or candidate list, scoped to the active context (Requirement 7).
• help <cmd> output contains both a generated flag line and the hand-authored example for a representative command in each context; help <unknown> reports not-found without a cross-context hint (Requirement 8, 11.2).
• A destructive command run non-interactively (piped stdin) executes without hanging on a prompt (Requirement 9.4).
• A real SIGINT delivered to the process while a confirmation prompt is displayed is observed as EINTR and reported as "not run" (Requirement 9.9), proving the sigaction()/SA_RESTART fix end-to-end rather than only in a simulated unit test.

No dedicated Performance/Load section is included — requirements.md states no performance target, and the Interruptible Call Adapter's 50ms poll interval is the only latency-relevant parameter, already justified in its component detail above.

Security Considerations

xce_test remains an internal test-only client; no new authentication/authorization surface is introduced, and connection/credential handling to the controller/worker/scheduler is unchanged and out of this spec's boundary. The Confirmation Gate (Requirement 9) is this design's one safety-relevant feature — it exists specifically to reduce the blast radius of an operator's typo or stale jobid against a live or degraded HA cluster during soak testing, not to guard against any external threat actor.