myxce/.kiro/specs/xce-test-e2e-primitives/design.md

# Design Document — xce-test-e2e-primitives

## Overview

**Purpose:** Make the `xce_test` client usable as a script-driven E2E driver: a piped command sequence whose **process exit code is the pass/fail verdict**, able to drive node lifecycle via a shell-escape, wait for failover deterministically, and assert outcomes in-tool.

**Users:** Test/CI authors and maintainers writing failover/durability/cold-restart E2E scenarios (e.g. the `test-commit-durability.sh` soak).

**Impact:** Today `xce_test` is an interactive client: `execute_line()` parses a line into argv, dispatches to a `command_t.func` (returning `int`, `<0` = error), and the main loop logs failures but **continues and exits 0**. Success is only inferable by grepping prose, and `jobsub` can print a garbage JobId (bug-786). This feature adds: session failure tracking + a real exit code; a `!` shell-escape; a fixed auto-variable substitution pass (`$?`, `$jobid`, `$jobcount`); `wait-primary`, `expect-jobs`, `expect-job`, `echo` commands; `#`/blank-line tolerance; and a `jobsub` reply-status fix. Everything lives in the single test-client TU; the shell remains the orchestrator.

### Goals
- The process exit code reflects whether any command failed (Req 1).
- A `!cmd` shell-escape lets one script drive node lifecycle; its status feeds `$?` (Req 2).
- A fixed, tool-maintained auto-variable set (`$?`, `$jobid`, `$jobcount`) — no user variables/expressions (Req 3).
- `wait-primary` blocks on the existing client failover until an op succeeds (Req 4).
- `expect-jobs`/`expect-job` assertions fail the session on mismatch (Req 5).
- `jobsub` reports success only on a clean ack with a valid JobId (Req 7).

### Non-Goals
- Structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test`; user-assignable variables, expressions, conditionals, or loops (the shell owns these).

## Boundary Commitments

### This Spec Owns
- The `xce_test` command loop's **failure tracking + process exit code**, an **abort-vs-continue** mode flag, and `#`/blank-line handling.
- A **line pre-processing step** in the client that (a) recognizes a leading `!` as a shell-escape and (b) substitutes the fixed auto-variables before tokenizing.
- The fixed **auto-variable state** (`$?`, `$jobid`, `$jobcount`) and the points that set it.
- New client commands: `wait-primary`, `expect-jobs`, `expect-job`, `echo`; the `set continue-on-error` toggle.
- The **`cmd_jobsub` reply-status correctness** fix (bug-786).
- The conversion of `src/tools/test-commit-durability.sh` to use these primitives.

### Out of Boundary
- Controller/replication behavior and opcodes (consumed unchanged).
- libxce client failover/master-rotation internals (consumed by `wait-primary`/`jobsub`/`joblist`).
- Node lifecycle (deploy/kill/restart) — performed by the shell, reachable via `!cmd`.
- Any general scripting language (variables/expressions/control flow).

### Allowed Dependencies
- The existing `command_t`/`cmd_func_t` dispatch, `execute_line`, `parse_argv`, `find_command`, and the main read loop in `test.c`.
- The existing client context (`active_ctx`), `cmd_jobsub`/`cmd_joblist` paths, and the libxce request/failover APIs they already call.
- Standard C `system()` for the shell-escape; a monotonic clock helper for `wait-primary` timeout.

### Revalidation Triggers
- Any change to the `cmd_func_t` contract (return-code semantics) → re-check all existing command handlers.
- A change to how `jobsub`/`joblist` report acks or list jobs → re-check `$jobid`/`$jobcount` capture and the assertions.

## Architecture

### Existing Architecture Analysis
- `execute_line(char* line)`: `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; returns the handler's `int` rc (`<0` error).
- Main loop: reads a line, `rc = execute_line(s)`; on `rc < 0` sets errno via `SET_ERROR` and **continues**; `main` returns 0. So failures never affect the exit code.
- Commands return `int` (`0` ok, `<0` errno-style). `cmd_jobsub`/`cmd_joblist` already drive the client + libxce failover.

### Pattern
Extend-in-place. Introduce a thin **pre-processing + verdict** wrapper around the existing dispatch; the new commands are ordinary `command_t` entries. No new module, no new files.

```mermaid
graph TD
    Loop[main read loop] --> Pre[preprocess line]
    Pre -->|leading '#'/blank| Skip[ignore]
    Pre -->|leading '!'| Shell[run via system; set $?]
    Pre -->|else| Subst[substitute $? / $jobid / $jobcount]
    Subst --> Exec[execute_line -> find_command -> func]
    Exec --> Track[update session-failed + $?]
    Shell --> Track
    Track -->|abort mode + failed| Stop[stop loop]
    Track --> Loop
    Loop --> Verdict[main returns 1 if any failure else 0]
```

## File Structure Plan

### Modified Files
- `src/controller/testsuite/test.c` — all client changes:
  - **Session/verdict state** (file-scope): `g_session_failed` (int), `g_continue_on_error` (bool), and auto-vars `g_last_status` (int), `g_last_jobid[XCE_MAXNAME_LEN]` (string), `g_last_jobcount` (int).
  - **`preprocess_and_run(char* line)`** (new): handle blank/`#`; leading `!` → `system(rest)`, set `g_last_status` from `WEXITSTATUS`; else substitute auto-vars then call `execute_line`. Update `g_session_failed`/`g_last_status` from the result; honor abort-vs-continue. Called from the main loop in place of the direct `execute_line`.
  - **`substitute_autovars(line) → expanded`** (new): replace whole-token `$?`, `$jobid`, `$jobcount` with current values; a reference with no current value yields a command failure (Req 3.6).
  - **New command handlers**: `cmd_wait_primary`, `cmd_expect_jobs`, `cmd_expect_job`, `cmd_echo`, registered in the command table(s).
  - **`cmd_set`**: add `continue-on-error` toggle.
  - **`cmd_jobsub`**: gate the "ok" report on a successful reply status + valid JobId; on success set `g_last_jobid`; otherwise print a typed error and return `<0` (bug-786).
  - **`cmd_joblist`**: set `g_last_jobcount`; expose a reusable internal count/contains helper for the assertions.
  - **`main`**: return `g_session_failed ? 1 : 0`.
- `src/tools/test-commit-durability.sh` — convert to drive `xce_test` via piped scripts using `wait-primary`/`expect-jobs`/`$jobid` and the exit-code verdict instead of prose-grepping + sanitizer filtering.

### New Files
- None.

> Note: the process exit code is only a trustworthy verdict if a clean (leak-free) exit is possible. Reuse the existing test run convention (`ASAN_OPTIONS=detect_leaks=0`) for the soak; leak cleanup of the client is out of scope here and tracked separately.

## Components and Interfaces

| Component | Intent | Req Coverage |
|-----------|--------|--------------|
| Verdict & loop wrapper (`preprocess_and_run`, `g_session_failed`, exit code) | Turn command outcomes into a process verdict; abort/continue | 1.1–1.6, 6.1, 6.2 |
| Shell-escape (`!cmd` via `system`) | Drive node lifecycle / setup from a script | 2.1–2.4 |
| Auto-vars (`substitute_autovars`, `g_last_*`) | Reference last status / JobId / job count | 3.1–3.6 |
| `cmd_wait_primary` | Block until a client op succeeds (failover-aware) | 4.1–4.4 |
| `cmd_expect_jobs` / `cmd_expect_job` | Assert job count / membership; fail session on mismatch | 5.1–5.4 |
| `cmd_echo` | Progress/section messages | 6.3 |
| `cmd_jobsub` fix | Report success only on clean ack + valid JobId; set `$jobid` | 7.1–7.3, 3.2 |
| `test-commit-durability.sh` driver | Reference E2E using the primitives | 8.1–8.3 |

### Verdict & loop wrapper
- **State (file-scope):** `g_session_failed`, `g_continue_on_error`, `g_last_status`, `g_last_jobid[]`, `g_last_jobcount`.
- **Contract:** every executed command updates `g_last_status` and, on failure, sets `g_session_failed`. In default mode the loop stops on first failure; under `continue-on-error` it keeps going. `main` returns non-zero iff `g_session_failed`.

### Shell-escape
- A line whose first non-whitespace char is `!` runs `system(rest_of_line)`. `g_last_status = WEXITSTATUS(rc)` (or non-zero on `system` failure). A non-zero status counts as a command failure.

### Auto-variables
- Whole-token substitution of `$?` (→ `g_last_status`), `$jobid` (→ `g_last_jobid`), `$jobcount` (→ `g_last_jobcount`) before tokenizing. Unset `$jobid`/`$jobcount` → command failure (no garbage). No other variable syntax is recognized.

### cmd_wait_primary
- `wait-primary [timeout_ms]`: loop a cheap client op (e.g. the `joblist`/probe path through `active_ctx`, which already does master rotation) until success or the timeout (monotonic clock); return 0 on success, `<0` on timeout. Default timeout when arg omitted.

### cmd_expect_jobs / cmd_expect_job
- `expect-jobs N`: obtain the current job count (shared helper with `cmd_joblist`); equal → 0, else print expected/actual and return `<0`.
- `expect-job <jobid>`: list jobs; present → 0, else print the missing id and return `<0`.

### cmd_jobsub (bug-786 fix)
- Inspect the reply: only when the controller returns a success status with a well-formed JobId does it print `jobsubmit ok: JobId: <id>` and set `g_last_jobid`. Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id.

## Error Handling
- **Command failure** → `g_last_status != 0`, `g_session_failed = 1`; abort (default) or continue. Non-zero process exit at end.
- **Shell-escape failure** → treated as a command failure via `$?`.
- **Unset auto-var reference** → command failure with a clear message (never substitutes garbage).
- **`wait-primary` timeout** and **assertion mismatch** → command failure with expected/actual reported.

## Testing Strategy

### Unit / behavioral (driven through `xce_test` itself; no controller needed for most)
- Exit code: a script with all-passing commands exits 0; a script with a failing `expect-jobs` exits non-zero (Req 1.1–1.3).
- Abort vs continue: a failing command mid-script stops execution by default; under `set continue-on-error` later commands still run and the final exit is non-zero (Req 1.4, 1.5).
- Shell-escape: `!true` → `$?`=0, success; `!false` → `$?`≠0, session fails (Req 2.1–2.4).
- Auto-vars: `$?` reflects the prior command; an unset `$jobid` reference fails rather than substituting garbage (Req 3.1, 3.6).
- Ergonomics: `#` comment and blank lines are ignored; `echo` succeeds (Req 6).
- `jobsub` reporting: a non-acked submit returns failure and does not set `$jobid`; never prints a malformed id (Req 7) — verifiable against a live single node or by simulating a non-ack.

### Integration / E2E (live `/xce-deploy-ha` cluster)
- Reference driver: converted `test-commit-durability.sh` runs a scenario — `wait-primary`, `jobsub` (capture `$jobid`), `expect-jobs`, `!kill` the primary, `wait-primary`, `expect-job $jobid` — and gates on the `xce_test` exit code (Req 4, 5, 8). Adding a `!`-driven cold-restart-all + `expect-jobs` step exercises the durability path (would catch a bug-778-class regression).

## Requirements Traceability

| Requirement | Component | Test |
|-------------|-----------|------|
| 1.1–1.6 | Verdict & loop wrapper | exit-code + abort/continue tests |
| 2.1–2.4 | Shell-escape | `!true`/`!false` tests |
| 3.1–3.6 | Auto-vars | `$?`/`$jobid` substitution + unset-fails tests |
| 4.1–4.4 | cmd_wait_primary | live failover wait |
| 5.1–5.4 | cmd_expect_jobs / cmd_expect_job | assertion pass/fail tests |
| 6.1–6.3 | loop wrapper + cmd_echo | comment/blank/echo tests |
| 7.1–7.3 | cmd_jobsub fix | non-ack returns failure; no garbage id |
| 8.1–8.3 | test-commit-durability.sh | converted soak runs, exit-code-gated |
```