myxce/.kiro/specs/xce-test-e2e-primitives/design.md

Design Document — xce-test-e2e-primitives

Overview

Purpose: Make the xce_test client usable as a script-driven E2E driver: a piped command sequence whose process exit code is the pass/fail verdict, able to drive node lifecycle via a shell-escape, wait for failover deterministically, and assert outcomes in-tool.

Users: Test/CI authors and maintainers writing failover/durability/cold-restart E2E scenarios (e.g. the test-commit-durability.sh soak).

Impact: Today xce_test is an interactive client: execute_line() parses a line into argv, dispatches to a command_t.func (returning int, <0 = error), and the main loop logs failures but continues and exits 0. Success is only inferable by grepping prose, and jobsub can print a garbage JobId (bug-786). This feature adds: session failure tracking + a real exit code; a ! shell-escape; a fixed auto-variable substitution pass ($?, $jobid, $jobcount); wait-primary, expect-jobs, expect-job, echo commands; #/blank-line tolerance; and a jobsub reply-status fix. Everything lives in the single test-client TU; the shell remains the orchestrator.

Goals

• The process exit code reflects whether any command failed (Req 1).
• A !cmd shell-escape lets one script drive node lifecycle; its status feeds $? (Req 2).
• A fixed, tool-maintained auto-variable set ($?, $jobid, $jobcount) — no user variables/expressions (Req 3).
• wait-primary blocks on the existing client failover until an op succeeds (Req 4).
• expect-jobs/expect-job assertions fail the session on mismatch (Req 5).
• jobsub reports success only on a clean ack with a valid JobId (Req 7).

Non-Goals

• Structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside xce_test; user-assignable variables, expressions, conditionals, or loops (the shell owns these).

Boundary Commitments

This Spec Owns

• The xce_test command loop's failure tracking + process exit code, an abort-vs-continue mode flag, and #/blank-line handling.
• A line pre-processing step in the client that (a) recognizes a leading ! as a shell-escape and (b) substitutes the fixed auto-variables before tokenizing.
• The fixed auto-variable state ($?, $jobid, $jobcount) and the points that set it.
• New client commands: wait-primary, expect-jobs, expect-job, echo; the set continue-on-error toggle.
• The cmd_jobsub reply-status correctness fix (bug-786).
• The conversion of src/tools/test-commit-durability.sh to use these primitives.

Out of Boundary

• Controller/replication behavior and opcodes (consumed unchanged).
• libxce client failover/master-rotation internals (consumed by wait-primary/jobsub/joblist).
• Node lifecycle (deploy/kill/restart) — performed by the shell, reachable via !cmd.
• Any general scripting language (variables/expressions/control flow).

Allowed Dependencies

• The existing command_t/cmd_func_t dispatch, execute_line, parse_argv, find_command, and the main read loop in test.c.
• The existing client context (active_ctx), cmd_jobsub/cmd_joblist paths, and the libxce request/failover APIs they already call.
• Standard C system() for the shell-escape; a monotonic clock helper for wait-primary timeout.

Revalidation Triggers

• Any change to the cmd_func_t contract (return-code semantics) → re-check all existing command handlers.
• A change to how jobsub/joblist report acks or list jobs → re-check $jobid/$jobcount capture and the assertions.

Architecture

Existing Architecture Analysis

• execute_line(char* line): parse_argv → find_command(argv[0]) → command->func(argc, argv); returns the handler's int rc (<0 error).
• Main loop: reads a line, rc = execute_line(s); on rc < 0 sets errno via SET_ERROR and continues; main returns 0. So failures never affect the exit code.
• Commands return int (0 ok, <0 errno-style). cmd_jobsub/cmd_joblist already drive the client + libxce failover.

Pattern

Extend-in-place. Introduce a thin pre-processing + verdict wrapper around the existing dispatch; the new commands are ordinary command_t entries. No new module, no new files.

graph TD
    Loop[main read loop] --> Pre[preprocess line]
    Pre -->|leading '#'/blank| Skip[ignore]
    Pre -->|leading '!'| Shell[run via system; set $?]
    Pre -->|else| Subst[substitute $? / $jobid / $jobcount]
    Subst --> Exec[execute_line -> find_command -> func]
    Exec --> Track[update session-failed + $?]
    Shell --> Track
    Track -->|abort mode + failed| Stop[stop loop]
    Track --> Loop
    Loop --> Verdict[main returns 1 if any failure else 0]

File Structure Plan

Modified Files

• src/controller/testsuite/test.c — all client changes:
  • Session/verdict state (file-scope): g_session_failed (int), g_continue_on_error (bool), and auto-vars g_last_status (int), g_last_jobid[XCE_MAXNAME_LEN] (string), g_last_jobcount (int).
  • preprocess_and_run(char* line) (new): handle blank/#; leading ! → system(rest), set g_last_status from WEXITSTATUS; else substitute auto-vars then call execute_line. Update g_session_failed/g_last_status from the result; honor abort-vs-continue. Called from the main loop in place of the direct execute_line.
  • substitute_autovars(line) → expanded (new): replace whole-token $?, $jobid, $jobcount with current values; a reference with no current value yields a command failure (Req 3.6).

    Superseded by job-array-one-based-index (2026-10-01): whole-token-only substitution is relaxed: a token that is $jobid followed immediately by [ (e.g. $jobid[3]) now has $jobid replaced and the rest of the token copied unchanged, and fails like an unset $jobid when it is unset (job-array-one-based-index Requirement 4.10, 4.11). Kept for history.

  • New command handlers: cmd_wait_primary, cmd_expect_jobs, cmd_expect_job, cmd_echo, registered in the command table(s).
  • cmd_set: add continue-on-error toggle.
  • cmd_jobsub: gate the "ok" report on a successful reply status + valid JobId; on success set g_last_jobid; otherwise print a typed error and return <0 (bug-786).
  • cmd_joblist: set g_last_jobcount; expose a reusable internal count/contains helper for the assertions.
  • main: return g_session_failed ? 1 : 0.
• src/tools/test-commit-durability.sh — convert to drive xce_test via piped scripts using wait-primary/expect-jobs/$jobid and the exit-code verdict instead of prose-grepping + sanitizer filtering.

New Files

• None.

Note: the process exit code is only a trustworthy verdict if a clean (leak-free) exit is possible. Reuse the existing test run convention (ASAN_OPTIONS=detect_leaks=0) for the soak; leak cleanup of the client is out of scope here and tracked separately.

Components and Interfaces

Component	Intent	Req Coverage
Verdict & loop wrapper (preprocess_and_run, g_session_failed, exit code)	Turn command outcomes into a process verdict; abort/continue	1.1–1.6, 6.1, 6.2
Shell-escape (!cmd via system)	Drive node lifecycle / setup from a script	2.1–2.4
Auto-vars (substitute_autovars, g_last_*)	Reference last status / JobId / job count	3.1–3.6
cmd_wait_primary	Block until a client op succeeds (failover-aware)	4.1–4.4
cmd_expect_jobs / cmd_expect_job	Assert job count / membership; fail session on mismatch	5.1–5.4
cmd_echo	Progress/section messages	6.3
cmd_jobsub fix	Report success only on clean ack + valid JobId; set $jobid	7.1–7.3, 3.2
test-commit-durability.sh driver	Reference E2E using the primitives	8.1–8.3

Verdict & loop wrapper

• State (file-scope): g_session_failed, g_continue_on_error, g_last_status, g_last_jobid[], g_last_jobcount.
• Contract: every executed command updates g_last_status and, on failure, sets g_session_failed. In default mode the loop stops on first failure; under continue-on-error it keeps going. main returns non-zero iff g_session_failed.

Shell-escape

• A line whose first non-whitespace char is ! runs system(rest_of_line). g_last_status = WEXITSTATUS(rc) (or non-zero on system failure). A non-zero status counts as a command failure.

Auto-variables

• Whole-token substitution of $? (→ g_last_status), $jobid (→ g_last_jobid), $jobcount (→ g_last_jobcount) before tokenizing. Unset $jobid/$jobcount → command failure (no garbage). No other variable syntax is recognized.

cmd_wait_primary

• wait-primary [timeout_ms]: loop a cheap client op (e.g. the joblist/probe path through active_ctx, which already does master rotation) until success or the timeout (monotonic clock); return 0 on success, <0 on timeout. Default timeout when arg omitted.

cmd_expect_jobs / cmd_expect_job

• expect-jobs N: obtain the current job count (shared helper with cmd_joblist); equal → 0, else print expected/actual and return <0.
• expect-job <jobid>: list jobs; present → 0, else print the missing id and return <0.

cmd_jobsub (bug-786 fix)

• Inspect the reply: only when the controller returns a success status with a well-formed JobId does it print jobsubmit ok: JobId: <id> and set g_last_jobid. Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return <0. Never print "ok" with an uninitialized/garbage id.

Error Handling

• Command failure → g_last_status != 0, g_session_failed = 1; abort (default) or continue. Non-zero process exit at end.
• Shell-escape failure → treated as a command failure via $?.
• Unset auto-var reference → command failure with a clear message (never substitutes garbage).
• wait-primary timeout and assertion mismatch → command failure with expected/actual reported.

Testing Strategy

Unit / behavioral (driven through xce_test itself; no controller needed for most)

• Exit code: a script with all-passing commands exits 0; a script with a failing expect-jobs exits non-zero (Req 1.1–1.3).
• Abort vs continue: a failing command mid-script stops execution by default; under set continue-on-error later commands still run and the final exit is non-zero (Req 1.4, 1.5).
• Shell-escape: !true → $?=0, success; !false → $?≠0, session fails (Req 2.1–2.4).
• Auto-vars: $? reflects the prior command; an unset $jobid reference fails rather than substituting garbage (Req 3.1, 3.6).
• Ergonomics: # comment and blank lines are ignored; echo succeeds (Req 6).
• jobsub reporting: a non-acked submit returns failure and does not set $jobid; never prints a malformed id (Req 7) — verifiable against a live single node or by simulating a non-ack.

Integration / E2E (live /xce-deploy-ha cluster)

• Reference driver: converted test-commit-durability.sh runs a scenario — wait-primary, jobsub (capture $jobid), expect-jobs, !kill the primary, wait-primary, expect-job $jobid — and gates on the xce_test exit code (Req 4, 5, 8). Adding a !-driven cold-restart-all + expect-jobs step exercises the durability path (would catch a bug-778-class regression).

Requirements Traceability

Requirement	Component	Test
1.1–1.6	Verdict & loop wrapper	exit-code + abort/continue tests
2.1–2.4	Shell-escape	!true/!false tests
3.1–3.6	Auto-vars	$?/$jobid substitution + unset-fails tests
4.1–4.4	cmd_wait_primary	live failover wait
5.1–5.4	cmd_expect_jobs / cmd_expect_job	assertion pass/fail tests
6.1–6.3	loop wrapper + cmd_echo	comment/blank/echo tests
7.1–7.3	cmd_jobsub fix	non-ack returns failure; no garbage id
8.1–8.3	test-commit-durability.sh	converted soak runs, exit-code-gated