myxce/.kiro/specs/xce-test-cli-ux/design.md

# Design Document: xce-test-cli-ux

## Overview

**Purpose**: This feature makes `xce_test`'s command-line surface predictable and fail-loud for the engineers who drive controller/worker/scheduler/HA testing through it — consistent flag validation across every command, a Ctrl-C that actually cancels a stuck RPC, structured output for scripted callers, complete per-command help, a safety gate on destructive actions, and in-session script composition via `source`. It also fixes two known display/default bugs (bug-1711, bug-1557) found during the CLI-UX audit that motivated this spec.

**Users**: SDETs and engineers running interactive soak/E2E sessions against live or degraded HA clusters, and script authors piping `xce_test` non-interactively in automated test harnesses.

**Impact**: Adds a small set of cross-cutting internal modules (flag parsing/validation, an interruptible-RPC wrapper, a confirmation gate, a JSON list formatter) to the existing single-file `xce_test` REPL client (`src/controller/testsuite/test.c`, ~4550 lines), migrates ~17 commands' flag parsing to a shared pattern, and corrects two existing defects — without changing wire protocol, RPC semantics, or any existing short flag's meaning.

### Goals
- Every flag-accepting command uniformly rejects an unrecognized/malformed flag and offers a long-form equivalent, named consistently for the same concept across commands.
- Ctrl-C reliably cancels a blocked RPC wait or the current input line — never the whole session, never silently doing nothing.
- The five destructive commands (`jobkill`, `shutdown`, `qctrl -i/-c`, `hostctrl -c`, `upgradeconfig`) require confirmation, safely bypassable per-command or per-session for scripts.
- `help <cmd>` is complete (description, full flag docs, example, context/related-command notes) and consistent for every command.
- `workerlist`/`joblist`/`reslist` support a JSON output mode with the same fields as their tables, unabbreviated.
- `source <file>` composes reusable command sequences, safely (cycle-detected).
- bug-1711's display defects and bug-1557's stale `join` default are corrected.

### Non-Goals
- No change to controller/worker/scheduler wire protocol or RPC semantics.
- No `xce_test` → `xce_harness` rename.
- No full curses/TUI mode, no color/theming.
- No new `.xce_testrc` auto-executed startup-file feature (already shipped by the separate `xce-test-rc-file` spec).
- No renaming or reassignment of any existing short flag's meaning.
- No change to the auto-variable/exit-code-verdict/shell-escape mechanisms already shipped by `xce-test-e2e-primitives`.

## Boundary Commitments

### This Spec Owns
- `xce_test`'s own CLI surface: flag parsing, validation, and typo-suggestion for every command that accepts flags (Flag Parsing Layer).
- Client-side Ctrl-C cancellation of a blocked RPC wait (Interruptible Call Adapter) — cancellation is local only; it does not cancel the in-flight request server-side. Owns serializing any two calls that share a connection (including a new call issued right after a prior one was abandoned) so a connection's single-outstanding-reply state is never touched concurrently.
- The destructive-command confirmation gate, its five-command enumeration, its bypass flag, and its session-scoped toggle (Confirmation Gate).
- JSON output for `workerlist`/`joblist`/`reslist` (JSON List Formatter).
- Command-name tab completion scoped to the active context (Tab Completion).
- `help <cmd>` content and generation, including the active-context-only not-found behavior (Help Renderer).
- The new `source <file>` command, its cycle detection, and its explicit missing-file error (Source Command).
- The specific display/default corrections enumerated in Requirements 11 and 12 (bug-1711, bug-1557).
- Process-level `-h`/`--help`/`--version` (Process Entry).

### Out of Boundary
- `libxce`'s blocking RPC primitives (`cbuf_timedget`, `xceapi.c`'s `xce_*` functions) — not modified. Requirement 1 is achieved entirely client-side, inside `test.c`/testsuite.
- Controller/worker/scheduler wire protocol and RPC semantics — unaffected by any part of this design.
- `rcfile.c`'s own path-resolution and startup-time missing-file behavior — untouched; `source` reuses `rcfile_load` as-is and layers its own missing-file/cycle behavior on top, in the new command, not in `rcfile.c`.
- Any existing short flag's accepted values, default, or meaning — never reassigned (only a long-form alias is added alongside).
- The exit-code/auto-variable verdict machinery and shell-escape mechanism from `xce-test-e2e-primitives` — read-only dependency, not modified.
- Authentication/connection setup to the controller/worker/scheduler — unchanged.
- The controller daemon's HA/persistence/consensus invariants (`src/controller/CLAUDE.md`'s TLA+ mapping, election safety, commit durability, etc.) — checked and found not applicable: `xce_test` is a client binary linked against `libxce`, not the controller daemon process, and no part of this design touches daemon code, WAL, or replication state.

### Allowed Dependencies
- cJSON (via the already-linked `struct2json_dep`) — for the JSON List Formatter. No new external dependency.
- libedit's completion API (already linked) — for Tab Completion.
- POSIX threads (already transitively linked via `libxce`) — for the Interruptible Call Adapter.
- `rcfile_load()` (existing, unmodified) — called directly by the Source Command.
- `cmd_set`'s existing settings mechanism — extended (not replaced) by the Confirmation Gate's toggle.

### Revalidation Triggers
- Any change to `libxce`'s blocking-call deadline (`cbuf_deadline_10s`) changes the worst-case lifetime of an abandoned worker thread this design assumes.
- Any change to `rcfile_load`'s signature or missing-file semantics invalidates the Source Command's reuse contract.
- A new `cmd_*` command added after this spec ships must adopt the `flagspec_t` pattern to stay covered by Requirements 2/6/8/10 — otherwise it silently falls outside these guarantees.
- Any future decision to reassign an existing short flag's meaning (explicitly deferred, not this spec) would invalidate the `flagspec_t` tables' short/long/concept mappings built here.

## Architecture

### Existing Architecture Analysis

- **Flag parsing** is three uncoordinated patterns today: a plain-`getopt()` loop (~16 commands, each with its own hand-written `switch`/`default`), positional-only argument checking (offline/diagnostic commands), and one fully hand-rolled long-flag string-compare loop (`cmd_walcompare`). No shared helper exists for parsing, rejection, or usage rendering — this disconnect between parsing code and the hand-authored `USAGE_*` string table is the direct root cause of bug-1711's drift.
- **Interrupt handling**: `SIGINT` → `sigbreak()` sets a static `break_out` flag, read in exactly one place (`cmd_hello`'s repeat loop). Every blocking `xce_*` RPC call waits on `pthread_cond_timedwait` with a fixed 10-second deadline (`cbuf_deadline_10s`/`cbuf_timedget`, in `libxce`) — a wait a signal-flag cannot interrupt.
- **REPL input**: libedit (`readline()`/`add_history()` only), no completion or key-binding customization registered.
- **Session settings**: a single global `settings[]` array shared (not actually per-context) by all four `command_ctx_t` instances; `cmd_set`'s `-e`/`+e`/`continue-on-error` toggle is the existing pattern for a session-scoped boolean.
- **Serialization**: cJSON is already linked (via `struct2json_dep`) and used narrowly for job exdata blobs; Cap'n Proto is also linked but exclusively for wire-protocol/checkpoint-blob decoding, not general CLI output.
- **Script loading**: `rcfile_load(path, runner)` is a generic, already-reusable "read a file line by line, execute each via a callback" primitive with no cycle protection and a silent-success policy on a missing file (correct for its one existing caller, the startup rc file).

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    User[Operator or Script]
    Readline[Readline Loop]
    Dispatch[Command Dispatcher]
    FlagSpec[Flag Parsing Layer]
    Confirm[Confirmation Gate]
    CmdFunc[Command Handler]
    Interrupt[Interruptible Call Adapter]
    Json[JSON List Formatter]
    Source[Source Command]
    RcFile[Rcfile Loader]
    LibXce[Libxce Rpc Client]
    Help[Help Renderer]

    User --> Readline
    Readline --> Dispatch
    Dispatch --> FlagSpec
    FlagSpec --> Confirm
    Confirm --> CmdFunc
    CmdFunc --> Interrupt
    Interrupt --> LibXce
    CmdFunc --> Json
    Dispatch --> Source
    Source --> RcFile
    RcFile --> Dispatch
    FlagSpec --> Help
    Dispatch --> Help
```

**Architecture Integration**:
- **Selected pattern**: a layered CLI pipeline — Readline → Dispatcher → Flag Parsing → (Confirmation Gate, destructive commands only) → Command Handler → Interruptible Call Adapter → `libxce`.
- **Domain/feature boundaries**: Flag Parsing is one cohesive layer shared by every flag-accepting command; the Interruptible Call Adapter is a second cross-cutting layer used only where a blocking RPC occurs; the Confirmation Gate is a third, used only by the five destructive commands.
- **Existing patterns preserved**: hand-authored `USAGE_*` prose for description/example/related-command text; the single-global session-settings pattern (`cmd_set`); `rcfile.c`'s file-loading primitive, reused unmodified.
- **New components rationale**: each addresses one specific root cause found in discovery — flag drift, a non-interruptible wait, no cycle guard, no typo-suggestion utility.
- **Steering compliance**: TDD (a `test_<module>.c` per new module, red-green-refactor), `gnu17`/Meson conventions, negative-errno-style (`errnox.h`) error propagation throughout.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| CLI / testsuite | C (gnu17), Meson | Language/build for all new and modified files | Matches existing `test.c` |
| CLI parsing | `getopt_long` (glibc) | Long-form flag parsing, replacing raw `getopt()` | Migrates ~17 commands |
| Structured output | cJSON (already linked via `struct2json_dep`) | Requirement 5 JSON mode | Adopted, no new dependency |
| Line editing | libedit (already linked) | Tab completion (Requirement 7) | `rl_attempted_completion_function`-style hook; exact API to confirm at implementation start |
| Concurrency | POSIX threads (already transitively linked via `libxce`) | Interruptible RPC wrapper (Requirement 1) | Detached worker + CAS ownership handoff |
| Test framework | Unity (project-wide) | New `test_*.c` files | `#ifdef XCE_TESTING` for internal-only symbols |

## File Structure Plan

`src/controller/testsuite/` is a flat directory; new modules follow its existing one-helper-module-per-file convention (`rcfile.c/h`, `waldump.c/h`, etc.).

### New Files
- `flagspec.c` / `flagspec.h` — flag-descriptor table format, `getopt_long` option-table builder, shared rejection + typo-suggestion helper, help-flag-block renderer. (Requirements 2, 6, 8.2, 10)
- `interruptcall.c` / `interruptcall.h` — `xcetest_run_interruptible()`: detached-worker-thread wrapper with CAS-based heap-ownership handoff, plus a per-connection lock registry that serializes overlapping calls (including a call issued right after a prior one was abandoned) on the same connection. (Requirement 1)
- `confirm.c` / `confirm.h` — destructive-command confirmation gate, session-scoped toggle, stderr-routed prompt, EINTR-as-decline. (Requirement 9)
- `listfmt_json.c` / `listfmt_json.h` — shared cJSON array/object helpers for the three listing commands' JSON mode. (Requirement 5)
- `test_flagspec.c`, `test_interruptcall.c`, `test_confirm.c`, `test_listfmt_json.c` — Unity unit tests, one per new module, following the existing `test_rcfile.c`-style convention.
- `test_test_cli.c` — new black-box/subprocess (pty-driven) test suite for `xce_test`'s own CLI surface (`-h`/`--version`, `help <cmd>` content, tab completion, command-not-found scoping, `source` cycle/missing-file cases) — this surface has zero prior coverage.

### Modified Files
- `src/controller/testsuite/test.c` — exposes a small `xcetest_interrupted()` accessor over the existing `break_out` flag for `interruptcall.c` to poll; replaces the existing `signal(SIGINT, sigbreak)` installation in `main()` with an explicit `sigaction()` call with `SA_RESTART` cleared, so `confirm.c`'s confirmation-prompt read reliably observes `EINTR`; every flag-parsing `cmd_*` migrated from raw `getopt()`/hand-rolled parsing to `flagspec_t` + `getopt_long`; the ~19 blocking `xce_*` call sites wrapped via `xcetest_run_interruptible()`; `cmd_set` extended with the confirmation-toggle branch (mirroring its existing `-e`/`+e` style); `cmd_source` added and registered in all four `command_t` tables; `xcetest_complete`/`xcetest_complete_generator` registered against libedit in `main()`; `cmd_help` extended to call `flagspec_render_help()`; `USAGE_*` constants split into a generated flag-list portion and retained hand-authored prose; `worker_status_str`, `cmd_help`'s not-found message, `cmd_runtasks`, `cmd_reslist`, `cmd_joblist`, `cmd_getconf` corrected per Requirement 11; `cmd_join_worker`'s default `maxmem` corrected per Requirement 12; the `argp` `doc` string in `main()` extended to describe the in-session `help` command per Requirement 4.3 (4.1/4.2's `-h`/`--help`/`--version` exit-without-REPL behavior is already `argp`'s default and needs no new code, only verification via `test_test_cli.c`).
- `src/controller/meson.build` — adds the four new `.c` files to `xce_test_src`, and their `test_*.c` counterparts plus `test_test_cli.c` to the appropriate test target(s).

## System Flows

### Interruptible RPC Call and Ctrl-C Abandonment

```mermaid
sequenceDiagram
    participant Op as Operator
    participant Cmd as CommandHandler
    participant Adapter as InterruptibleAdapter
    participant Lock as ConnectionLock
    participant Worker as WorkerThread
    participant Rpc as LibxceRpc

    Op->>Cmd: invoke a command with a blocking RPC
    Cmd->>Adapter: run_interruptible call
    Adapter->>Lock: acquire lock for the active connection, interruptibly
    Lock-->>Adapter: acquired
    Adapter->>Worker: spawn detached thread
    Worker->>Rpc: blocking xce call, up to ten second deadline
    Adapter->>Adapter: poll break_out every fifty milliseconds
    Op-->>Adapter: Ctrl-C sets break_out
    Adapter->>Adapter: CAS running to abandoned succeeds
    Adapter-->>Cmd: return EINTR immediately, lock stays held
    Cmd-->>Op: report command as failed, prompt returns
    Rpc-->>Worker: reply arrives later
    Worker->>Worker: CAS running to done fails, worker frees the block
    Worker->>Lock: release lock
```

A second command issued against the *same* connection after an abandonment blocks on this same lock-acquisition step — itself interruptibly — until the abandoned worker finishes and releases it, bounding the unsafe-overlap window to zero rather than letting two calls share one connection's single-outstanding-reply state concurrently. A command against a *different* connection is unaffected, since each connection gets its own lock.

### Heap-Block Ownership Handoff

```mermaid
stateDiagram-v2
    [*] --> Running
    Running --> DoneUnclaimed: worker finishes, CAS succeeds
    Running --> Abandoned: caller interrupted, CAS succeeds
    DoneUnclaimed --> [*]: caller reads rc and frees the block
    Abandoned --> [*]: worker frees the block after finishing
```

Exactly one atomic compare-and-swap decides who frees the block, regardless of which side "wins" the race — this is what prevents a dangling-stack or double-free bug in the abandonment path (see `research.md`, "Interruptible RPC via detached worker thread" decision). This block-ownership handoff is orthogonal to, and independent of, the per-connection lock shown in the sequence diagram above: the lock protects the *connection*'s exclusive use across the call's full lifetime; the CAS protects the *heap block*'s lifetime. Both are released/freed by whichever side is left holding them once the worker finishes.

### Destructive Command Confirmation Gate

```mermaid
flowchart TD
    Start[Destructive command invoked]
    CheckBypass{Bypass flag given}
    CheckToggle{Session toggle set to skip}
    CheckTty{Input from interactive TTY}
    Prompt[Prompt operator on stderr]
    Answer{Operator answers}
    Run[Execute command]
    NotRun[Report not run]

    Start --> CheckBypass
    CheckBypass -- yes --> Run
    CheckBypass -- no --> CheckToggle
    CheckToggle -- yes --> Run
    CheckToggle -- no --> CheckTty
    CheckTty -- no --> Run
    CheckTty -- yes --> Prompt
    Prompt --> Answer
    Answer -- yes --> Run
    Answer -- no or ctrl c --> NotRun
```

The TTY check evaluates the top-level session's own stdin (`g_interactive`), not whether the current line originates from a `source`d file (Requirement 13.7) — a sourced destructive command in an otherwise-interactive session still prompts unless the toggle is set.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1, 1.2, 1.3, 1.4, 1.5 | Ctrl-C cancels a blocked RPC or the input line, never the session | Interruptible Call Adapter, Command Dispatcher | `xcetest_run_interruptible` | Interrupt sequence + ownership state diagram |
| 2.1, 2.2, 2.3, 2.4 | Uniform rejection of unrecognized/malformed flags | Flag Parsing Layer | `flagspec_reject` | - |
| 3.1, 3.2, 3.3, 3.4 | stderr/stdout discipline | Command Dispatcher, all `cmd_*` handlers | - | - |
| 4.1, 4.2, 4.3 | Process-level `-h`/`--help`/`--version` | Process Entry | `argp` options | - |
| 5.1, 5.2, 5.3, 5.4 | JSON output for listing commands | JSON List Formatter | `listfmt_json_*` | - |
| 6.1, 6.2, 6.3, 6.4 | Long-form, concept-consistent flags | Flag Parsing Layer | `flagspec_t` table | - |
| 7.1, 7.2, 7.3, 7.4 | Command-name tab completion | Tab Completion | `xcetest_complete*` | - |
| 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7 | Detailed, context-scoped help content | Help Renderer, Flag Parsing Layer | `flagspec_render_help`, `cmd_help` | - |
| 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9 | Confirmation gate for destructive commands | Confirmation Gate | `confirm_destructive`, `cmd_set` toggle | Confirmation flowchart |
| 10.1, 10.2, 10.3 | Actionable flag-typo suggestions | Flag Parsing Layer | `flagspec_reject` (suggestion path) | - |
| 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8 | bug-1711 display-defect fixes | Help Renderer, and `worker_status_str`/`cmd_runtasks`/`cmd_reslist`/`cmd_joblist`/`cmd_getconf` | - | - |
| 12.1, 12.2 | bug-1557 `join` maxmem default fix | `cmd_join_worker`, Help Renderer | - | - |
| 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7 | `source` command | Source Command, `rcfile_load` (reused) | `cmd_source` | Confirmation flowchart (13.7) |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|--------------------------|-----------|
| Flag Parsing Layer | CLI Core | `getopt_long` parsing, rejection, suggestion, help-flag rendering from one descriptor table | 2, 6, 8.2, 10 | `getopt_long` (P0) | Service |
| Interruptible Call Adapter | CLI Core | Client-side cancellable wrapper around blocking `xce_*` calls | 1 | pthread (P0), `libxce` `xceapi` (P0) | Service |
| Confirmation Gate | CLI Core | Gate + session toggle for destructive commands | 9 | `cmd_set` (P1), Flag Parsing Layer (P1) | Service |
| JSON List Formatter | Output | cJSON-based structured output for the three listing commands | 5 | cJSON (P0) | Service |
| Tab Completion | REPL | libedit completion hook scoped to the active context | 7 | libedit (P0) | Service |
| Help Renderer | REPL | Assembles `help <cmd>` from generated flags + hand-authored prose; scopes not-found to the active context | 8, 11.2, 12.2 | Flag Parsing Layer (P0) | Service |
| Source Command | REPL | `source <file>`, cycle detection, reuse of `rcfile_load` | 13 | `rcfile_load` (P0), Confirmation Gate (P1) | Service |
| Process Entry | Process | `argp`-based `-h`/`--help`/`--version` | 4 | `argp` (glibc) (P0) | Service |
| Bug-1711 Fixes | Output | Text/formatting corrections in existing `cmd_*` bodies | 11 | — | — |
| Bug-1557 Fix | Data | `join`'s `maxmem` default correction | 12 | — | — |

Only components introducing a new boundary get a full detail block below; Process Entry and the two bug-fix rows are simple, localized edits to existing functions and need no further elaboration beyond the summary and File Structure Plan.

### CLI Core

#### Flag Parsing Layer

| Field | Detail |
|-------|--------|
| Intent | One small library, used by every flag-accepting command, that parses via `getopt_long` from a per-command descriptor table and uniformly rejects/suggests/documents against that same table |
| Requirements | 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4, 8.2, 10.1, 10.2, 10.3 |

**Responsibilities & Constraints**
- Owns the `flagspec_t` descriptor format, the shared rejection message format, and the shared help-flag-block format.
- Does not own each command's business-logic switch on parsed values — only the parse/reject/describe surface.
- Invariant: a `flagspec_t[]` table's long-form name for a repeated concept (host, jobid, command line, resource name) is identical across every command that has it — enforced by table review, not at runtime.

**Dependencies**
- Outbound: `getopt_long` (glibc) — P0
- Outbound: a small edit-distance utility introduced by this component itself (not an external dependency) — P0

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
typedef struct {
    char short_opt;           /* 0 if long-only */
    const char* long_opt;     /* stable, cross-command-consistent name */
    const char* concept;      /* e.g. "host", "jobid", "command-line" */
    int has_arg;               /* no_argument | required_argument */
    int required;               /* 1 if the command fails without it */
    const char* default_desc;  /* human text, NULL if none */
} flagspec_t;

int   flagspec_build_opts(const flagspec_t* spec, size_t n,
                           struct option* out_opts, char* out_optstring,
                           size_t out_optstring_cap);

int   flagspec_reject(const char* cmd_name, const flagspec_t* spec, size_t n,
                       const char* bad_flag, const char* usage_text);

char* flagspec_render_help(const flagspec_t* spec, size_t n);
```
- Preconditions: `spec` is a stable static array for the process lifetime; the caller sets `opterr = 0` before its `getopt_long` loop so libc's own diagnostic never doubles up with `flagspec_reject`'s.
- Postconditions: `flagspec_reject` always returns a negative value and never partially executes the command.
- Invariants: the built optstring never begins with `:` — missing-argument detection is `flagspec_reject`'s job, not silent `getopt` behavior.

**Implementation Notes**
- Integration: every `getopt()`-using command and the one hand-rolled long-flag command (`cmd_walcompare`) migrate to `getopt_long` + a static `flagspec_t[]` + a `case '?': return flagspec_reject(...)` arm.
- Validation: `test_flagspec.c` covers the suggestion threshold (exactly-one-close-match / no-match / ambiguous), the optstring builder, and the help-render format, independent of any real command.
- Risks: existing short-flag collisions (`-m` five ways, `-c` six ways) mean the same short letter maps to a *different* `concept` in different commands' tables — expected, not a defect; Requirement 6.2 only requires the long-form name to be consistent per concept, not that short letters be globally unique.
- Note (bug-1704): `cmd_qctrl`'s existing `action` variable has no initializer and no post-loop check that one of `-a`/`-i`/`-o`/`-c` was actually given, unlike its sibling `cmd_hostctrl`, which already initializes to a sentinel and rejects if unset. This migration must apply the same sentinel-and-reject pattern to `cmd_qctrl` — not just for correctness, but because the Confirmation Gate (Requirement 9.5) depends on reliably knowing whether `-i`/`-c` was given, which is unsafe to check against an uninitialized value.

#### Interruptible Call Adapter

| Field | Detail |
|-------|--------|
| Intent | Let a command make a blocking `xce_*` RPC call cancellably from the operator's perspective, without modifying `libxce` |
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5 |

**Responsibilities & Constraints**
- Owns the heap-block ownership protocol (the CAS state machine) and the poll interval.
- Owns a per-connection lock registry that serializes any two calls sharing the same connection — including a new call issued after a prior one on that connection was abandoned — so a connection's single-outstanding-reply state is never touched by two threads at once.
- Never touches `libxce`/`xceapi.c` internals — the wrapped call is opaque.
- Invariant: a heap-allocated call block's lifetime is resolved by exactly one atomic CAS; no code path frees it from both sides.
- Invariant: at most one worker thread — running or abandoned-but-still-running — is ever active per connection at a time.

**Dependencies**
- Outbound: pthread create/detach, POSIX semaphore (`sem_t`, one per connection, in an internal registry) — P0 (a mutex was considered and rejected during implementation review: POSIX classifies a non-owning-thread `pthread_mutex_unlock()` as undefined behavior for every mutex type, including the default, and this design's cross-thread release — the worker thread releasing a lock the caller thread acquired — needs a primitive with no ownership concept; see bug-1791)
- Outbound: the wrapped `xce_*` call, treated as opaque — P0
- Inbound: every command performing a blocking RPC (~19 call sites) — P0

**Contracts**: Service [x]

##### Service Interface
```c
typedef int (*xcetest_blocking_fn)(void* ctx);

typedef enum { XCT_RUNNING, XCT_DONE_UNCLAIMED, XCT_ABANDONED } xcetest_call_state_t;

/* Serializes against any other in-flight or abandoned-but-still-running
 * call that shares the same active connection (keyed internally by the
 * current command_ctx_t's connection pointer, looked up in a small,
 * lazily-populated, process-lifetime registry - at most one entry per
 * context, so a linear scan suffices). Acquiring that per-connection lock
 * is itself interruptible: if break_out fires while waiting for a PRIOR
 * call's abandoned worker to release it, this returns -EINTR before ever
 * spawning fn. */
int xcetest_run_interruptible(xcetest_blocking_fn fn, void* ctx);
```
- Preconditions: `fn` touches only what `ctx` itself heap-owns — never caller-stack memory.
- Postconditions: on `-EINTR`, the caller must not touch `ctx` again — ownership has passed to the still-running worker thread. The per-connection lock is released by whichever side (caller or worker) is the last to finish with the call, never left held once both sides are done.
- Invariants: exactly one of {caller, worker} frees the call block, decided by which side wins the `RUNNING -> {DONE_UNCLAIMED, ABANDONED}` CAS.

**Implementation Notes**
- Integration: each of the ~19 blocking-RPC call sites gets a small per-site static thunk matching `xcetest_blocking_fn`; none of them need to pass a connection identity explicitly — the per-connection lock key is read internally from the currently active `command_ctx_t`'s connection pointer (exact global/field name to confirm against `test.c` at implementation start, alongside the libedit completion API verification already noted in Tab Completion's Risks). A call site that ever operates on a connection *other than* the currently active one would break this assumption — none is known to today, but this is worth a one-line check per call site during migration.
- Integration: the lock registry's lookup/insert step needs no synchronization of its own — a *new* registry entry is only ever created from the single REPL thread that issues commands (detached worker threads never initiate a new top-level call; they only run the `fn` they were given and release the lock/free the block they were handed), so there is no concurrent-insert case to guard against.
- Validation: `test_interruptcall.c` uses a fake slow `fn` to exercise both the normal-completion and forced-abandon paths, verifying no double-free and no leak; a second test spawns two overlapping calls against the same fake connection key (one abandoned, one issued immediately after) and asserts they never run concurrently — this is the regression test for the connection-race this component's lock exists to prevent.
- Risks: none material remaining — the connection-race risk this component previously left as an unverified follow-up is now closed by the per-connection lock described above.

#### Confirmation Gate

| Field | Detail |
|-------|--------|
| Intent | Gate the five destructive commands behind interactive confirmation, a per-command bypass flag, or a session-scoped toggle |
| Requirements | 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9 |

**Responsibilities & Constraints**
- Owns the destructive-command enumeration (9.5), the session toggle state, the stderr-routed prompt, and Ctrl-C-as-decline.
- Does not own the destructive action itself — only whether it proceeds.

**Dependencies**
- Outbound: `cmd_set` (extended with the toggle branch) — P1
- Outbound: Flag Parsing Layer (the per-command `--yes` bypass is itself a `flagspec_t` entry) — P1
- Outbound: the process's SIGINT installation (`main()`) — must be installed without `SA_RESTART` for the confirmation read to reliably observe `EINTR` — P0
- Inbound: `cmd_jobkill`, `cmd_shutdown`, `cmd_qctrl`, `cmd_hostctrl`, `cmd_upgradeconfig` — P0

**Contracts**: Service [x]

##### Service Interface
```c
int  confirm_destructive(const char* action_description);
void confirm_set_skip(int skip);
int  confirm_is_skip_set(void);
```
- Preconditions: called only at the call sites enumerated by Requirement 9.5 (the enumeration's implementation is authoritative — no other command calls this). The process's SIGINT handler must already be installed without `SA_RESTART` (see Implementation Notes) — this is a one-time startup precondition, not something each call re-checks.
- Postconditions: a 0 return means the caller reports the command as "not run," never "failed" (9.2, 9.9).
- Invariants: the prompt always goes to stderr (9.8); the TTY check is the top-level session's `g_interactive`, not "is this line from a sourced file" (13.7).

**Implementation Notes**
- Integration: each of the five call sites checks `confirm_is_skip_set() || !g_interactive || <its own --yes flag>` before calling `confirm_destructive`.
- Integration: the existing `signal(SIGINT, sigbreak)` installation in `main()` (`test.c:4457`) is replaced with an explicit `sigaction()` call with `SA_RESTART` cleared. This is verified, not assumed: glibc's `signal()` uses BSD semantics, which historically restart an interrupted slow syscall by default — if left as-is, the confirmation prompt's blocking read could silently retry instead of returning `EINTR`, and Requirement 9.9 would never fire. This is a one-time, global change made once at startup; it does not affect the Interruptible Call Adapter, which never relies on `EINTR` (it polls `break_out` on a timer instead).
- Validation: `test_confirm.c` simulates non-TTY stdin for auto-bypass, and a piped decline/EOF/simulated-EINTR answer for the decline paths; `test_test_cli.c` adds a real pty-driven case that sends an actual SIGINT to the process while a confirmation prompt is displayed, proving the corrected `sigaction()` installation actually delivers `EINTR` end-to-end (not just in a simulated unit test).
- Risks: none material remaining — the EINTR-reliance risk this component previously left unverified is now closed by the explicit `sigaction()`/`SA_RESTART` fix and its end-to-end test.

### Output

#### JSON List Formatter

| Field | Detail |
|-------|--------|
| Intent | Shared cJSON array/object plumbing so the three listing commands can emit their existing per-row data as JSON |
| Requirements | 5.1, 5.2, 5.3, 5.4 |

**Responsibilities & Constraints**
- Owns only the array/object lifecycle helpers — not each command's field mapping. No generic "record" abstraction is introduced (per the Simplification lens — each command's row struct already exists and is stable).

**Dependencies**
- Outbound: cJSON (already linked) — P0
- Inbound: `cmd_workerlist`, `cmd_joblist`, `cmd_reslist` — P0

**Contracts**: Service [x]

##### Service Interface
```c
cJSON* listfmt_json_begin(void);
void   listfmt_json_add_string(cJSON* row, const char* key, const char* val);
void   listfmt_json_add_number(cJSON* row, const char* key, double val);
cJSON* listfmt_json_new_row(cJSON* array);
void   listfmt_json_finish(cJSON* array);
```
- Preconditions: called only when the command's `--json` flag (a boolean `flagspec_t` entry, the same long-form name across all three commands per Requirement 6.2) is set.
- Postconditions: every field present in the human-readable table row is also present in the JSON object (5.1); no truncation regardless of value length (5.3).

**Implementation Notes**
- Integration: each command's existing print loop gains an `if (json_mode) { ... } else { existing table row } ` branch.
- Validation: `test_listfmt_json.c` covers array/object lifecycle and long-value non-truncation; per-command field mapping is covered by that command's own test or the black-box CLI suite.
- Risks: none material.

### REPL

#### Tab Completion

| Field | Detail |
|-------|--------|
| Intent | Register a libedit completion function scoped to the active context's command table |
| Requirements | 7.1, 7.2, 7.3, 7.4 |

**Responsibilities & Constraints**
- Owns the completion-candidate lookup against `active_ctx->commands[]` only.

**Dependencies**
- Outbound: libedit completion API — P0 (exact signature to confirm at implementation start, see `research.md` Risks)

**Contracts**: Service [x]

##### Service Interface
```c
char** xcetest_complete(const char* text, int start, int end);
char*  xcetest_complete_generator(const char* text, int state);
```
- Preconditions: registered only when `g_interactive` is true.
- Postconditions: never offers a command invalid in the active context (7.4); an unambiguous match completes in full (7.2); multiple matches list candidates without altering the input line (7.3).

**Implementation Notes**
- Integration: registered once in `main()` alongside the existing `readline`/`add_history` setup. No new header — small enough to live in `test.c` directly.
- Validation: covered by `test_test_cli.c` (pty-driven — libedit completion isn't practically unit-testable in isolation).
- Risks: libedit API confirmation (`research.md` Risks).

#### Help Renderer

| Field | Detail |
|-------|--------|
| Intent | Compose `help <cmd>` from the generated flag block plus each command's hand-authored prose; scope "not found" strictly to the active context |
| Requirements | 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 11.2, 12.2 |

**Responsibilities & Constraints**
- Owns the assembly order (description -> flags -> example -> related-command/context notes) and the not-found message text.
- Does not own the prose content — still hand-authored per command in the `USAGE_*` constants, now split into a prose part and a flagspec-rendered part.

**Dependencies**
- Outbound: Flag Parsing Layer's `flagspec_render_help` — P0

**Contracts**: Service [x]

##### Service Interface
Extends the existing `cmd_help` in `test.c` — no new header. `command_t` gains a `flagspec_t*`/count pair alongside its existing `usage` pointer; `cmd_help`'s per-command branch calls `flagspec_render_help(cmd->flagspec, cmd->flagspec_count)` and concatenates it with the command's (now flag-list-free) prose constant.

**Implementation Notes**
- Integration: every `USAGE_*` constant is edited to drop its hand-written flag-list lines (now generated) while keeping description/example/related-command prose.
- Validation: `test_test_cli.c` checks a representative sample of `help <cmd>` outputs for both a generated flag line and the retained hand-authored example, plus the corrected grammar (11.2) and `join`'s corrected default (12.2).
- Risks: touches all 40 `USAGE_*` constants — mechanical but broad; sequence as its own task after the Flag Parsing Layer lands.

#### Source Command

| Field | Detail |
|-------|--------|
| Intent | `source <file>`, reusing `rcfile_load`, with cycle detection and an explicit missing-file error |
| Requirements | 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7 |

**Responsibilities & Constraints**
- Owns the in-flight source-path stack (cycle detection) and the missing-file check.
- Does not own `rcfile.c`'s file-reading loop or its startup-time missing-file policy — unchanged.

**Dependencies**
- Outbound: `rcfile_load` (unmodified) — P0
- Outbound: Confirmation Gate (13.7's TTY-condition clarification) — P1

**Contracts**: Service [x]

##### Service Interface
```c
int cmd_source(int argc, char* argv[]);          /* registered command_t entry */
int  source_push_path(const char* canonical_path); /* 0 ok, -ELOOP on cycle */
void source_pop_path(void);
```
- Preconditions: `argv[1]` is a path; `access(path, R_OK)` must succeed or `cmd_source` reports the error itself (bypassing `rcfile_load`'s silent-missing behavior).
- Postconditions: the path stack is popped exactly once per successful push, on every exit path including an aborted line under `-e` semantics.
- Invariants: `realpath()` is applied before pushing, so a cycle through symlinks or a differently-spelled relative path is still caught.

**Implementation Notes**
- Integration: registered as a new entry in `top_commands`, `worker_commands`, `client_commands`, and `scheduler_commands`.
- Validation: `test_test_cli.c` covers a two-file mutual-cycle case and a non-existent-file case as black-box scenarios (these need real files on disk).
- Risks: the path stack must be popped on every exit path, not just the success path — call out explicitly in the implementing task.

## Data Models

Minimal — `xce_test` has no persistent storage. The only new data shapes are the C structs already shown above (`flagspec_t`, the interruptible-call state enum, the per-connection lock registry entry — a connection pointer paired with a `sem_t`, at most one per live context — and the source-path stack). No database, no schema, no migration.

## Error Handling

### Error Strategy
Every new or modified path returns a negative errno-style code per this project's existing C convention (`errnox.h`), and every error/diagnostic message is written to stderr per Requirement 3 — this design extends the existing error-reporting philosophy uniformly rather than introducing a new one.

### Error Categories and Responses
- **Malformed/unrecognized flag** -> `flagspec_reject` (stderr + usage + optional suggestion, nonzero return) — Requirements 2, 10.
- **Blocked RPC interrupted** -> `-EINTR` from `xcetest_run_interruptible`, reported as "failed" (1.3) except at a confirmation prompt, reported as "not run" (9.9).
- **Destructive command declined** -> 0 return, explicit "not run" reporting (9.2).
- **`source` missing file / cycle** -> stderr error, nonzero return, no partial execution (13.2, 13.5).

### Monitoring
Not applicable — `xce_test` is an interactive/scripted client, not a long-running service. No new logging/metrics infrastructure beyond the existing stderr diagnostics this design extends.

## Testing Strategy

**Unit Tests**
- `test_flagspec.c` — optstring/option-table builder correctness; suggestion logic (exactly-one-match / no-match / ambiguous-match-suppressed); help-render format; `cmd_qctrl`'s corrected sentinel-and-reject validation (bug-1704).
- `test_interruptcall.c` — normal completion path and forced-abandon path via a fake slow `fn`, verifying no double-free under either; two overlapping calls against the same fake connection key (one abandoned, one issued immediately after) never run concurrently, proving the per-connection lock.
- `test_confirm.c` — non-TTY auto-bypass; decline path; simulated-EINTR-as-decline path; toggle on/off.
- `test_listfmt_json.c` — array/object lifecycle; no truncation of a long field value.

**Integration Tests**
- `cmd_source` two-file mutual-cycle detection; missing-file error; nested (non-cyclic) sourcing composing correctly.
- Flag rejection end-to-end through a real migrated command (e.g. `jobkill -z`), confirming stderr routing + usage + nonzero exit, not just the shared helper in isolation.
- Confirmation gate against a real destructive command (`qctrl -c`) with the session toggle flipped, confirming no prompt appears and the command executes.

**E2E / Black-box Tests** (`test_test_cli.c`, subprocess/pty-driven — the only viable vehicle for these)
- `-h`/`--help`/`--version` print and exit without entering the REPL (Requirement 4).
- Tab completion at the prompt: partial command name + Tab yields the expected single completion or candidate list, scoped to the active context (Requirement 7).
- `help <cmd>` output contains both a generated flag line and the hand-authored example for a representative command in each context; `help <unknown>` reports not-found without a cross-context hint (Requirement 8, 11.2).
- A destructive command run non-interactively (piped stdin) executes without hanging on a prompt (Requirement 9.4).
- A real SIGINT delivered to the process while a confirmation prompt is displayed is observed as `EINTR` and reported as "not run" (Requirement 9.9), proving the `sigaction()`/`SA_RESTART` fix end-to-end rather than only in a simulated unit test.

No dedicated Performance/Load section is included — requirements.md states no performance target, and the Interruptible Call Adapter's 50ms poll interval is the only latency-relevant parameter, already justified in its component detail above.

## Security Considerations

`xce_test` remains an internal test-only client; no new authentication/authorization surface is introduced, and connection/credential handling to the controller/worker/scheduler is unchanged and out of this spec's boundary. The Confirmation Gate (Requirement 9) is this design's one safety-relevant feature — it exists specifically to reduce the blast radius of an operator's typo or stale jobid against a live or degraded HA cluster during soak testing, not to guard against any external threat actor.