myxce/.kiro/specs/xce-test-e2e-primitives/requirements.md

Requirements Document

Introduction

This feature adds a small set of scripting primitives to the xce_test client so that a complete end-to-end scenario — submit work, kill the primary, wait for failover, restart/cold-restart nodes, and assert that committed work survived — can be driven from a single piped or scripted command sequence whose process exit code is the pass/fail verdict. Today xce_test is an interactive client whose success can only be inferred by grepping human-readable output (and which can emit a malformed JobId), so E2E scripts must filter prose, suppress sanitizer noise, and poll controller logs for cluster state. These primitives let the shell remain the orchestrator (node lifecycle, loops) while xce_test reliably reports outcomes.

Boundary Context

• In scope: command-failure tracking and a process exit code that reflects it; a !-prefixed shell-escape command; a fixed set of tool-maintained auto-variables ($?, $jobid, optional $jobcount); a wait-primary blocking command; expect-jobs / expect-job assertion commands; script ergonomics (# comments, blank lines, echo); correct jobsub success/failure reporting (bug-786); converting src/tools/test-commit-durability.sh to use these primitives.
• Out of scope: structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside xce_test (kill/restart/deploy stay in the shell, reachable via !cmd); user-assignable variables, expression evaluation, conditionals, or loops (the shell owns these); any change to controller/replication behavior.
• Adjacent expectations: wait-primary and jobsub/joblist rely on the existing libxce client-side master-rotation/failover already built into xce_test; this feature consumes that behavior and does not change it.

Requirements

Requirement 1: Exit code reflects command outcomes

Objective: As an E2E script author, I want the xce_test process to exit non-zero when any command in the session failed, so that a shell can use the exit code alone as the pass/fail verdict.

Acceptance Criteria

1. When a command in the session fails, the xce_test client shall record the session as failed.
2. When the session ends and at least one command failed, the xce_test client shall exit with a non-zero status.
3. When the session ends and every command succeeded, the xce_test client shall exit with status zero.
4. While default (abort-on-failure) mode is active, when a command fails, the xce_test client shall stop executing subsequent commands and exit non-zero.
5. Where set continue-on-error has been issued, when a command fails, the xce_test client shall continue executing subsequent commands and still exit non-zero at session end.
6. The xce_test client shall send all human-readable log and diagnostic output in a way that does not corrupt the determination of the exit-code verdict.

Requirement 2: Shell-escape command

Objective: As an E2E script author, I want to run an arbitrary shell command from within an xce_test script, so that a single script can drive node lifecycle (kill, restart, deploy) and environment setup without a separate orchestrator.

Acceptance Criteria

1. When a script line begins with !, the xce_test client shall execute the remainder of the line as a shell command.
2. When the shell command exits zero, the xce_test client shall treat the ! command as successful.
3. If the shell command exits non-zero, then the xce_test client shall treat the ! command as failed (subject to Requirement 1's exit-code and abort/continue behavior).
4. When a ! command completes, the xce_test client shall set $? to the shell command's exit status.

Requirement 3: Tool-maintained auto-variables

Objective: As an E2E script author, I want a fixed set of auto-variables that later commands can reference, so that I can assert on outcomes without parsing client output in the shell.

Acceptance Criteria

1. When any command completes, the xce_test client shall set $? to that command's status (zero on success, non-zero on failure).
2. When a jobsub command receives a successful acknowledgment, the xce_test client shall set $jobid to the acknowledged JobId.
3. When a subsequent command references $jobid (e.g. expect-job $jobid), the xce_test client shall substitute the most recently acknowledged JobId.

Superseded by job-array-one-based-index (2026-10-01): 3.2/3.3 no longer hold for array submissions. After jobsub -a N (N >= 2) the acknowledged identity is the array identity (no element number), $jobid is set to it, and elements are named $jobid[k], k = 1..N; $jobid followed immediately by [ is now substituted too (see job-array-one-based-index Requirement 4.5, 4.10, 4.11). Ordinary jobsub is unchanged. Kept for history.

4. Where the $jobcount auto-variable is provided, when a joblist command completes, the xce_test client shall set $jobcount to the number of jobs returned.
5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables.
6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value.

Requirement 4: Wait for an elected primary

Objective: As an E2E script author, I want a command that blocks until the cluster has a serving primary, so that scripts can proceed deterministically after a kill or restart without fixed sleeps or log polling.

Acceptance Criteria

1. When wait-primary is issued, the xce_test client shall repeatedly attempt a client operation against the configured masters until one succeeds.
2. When a client operation succeeds within the timeout, the xce_test client shall complete wait-primary successfully.
3. If no client operation succeeds within the timeout, then the xce_test client shall fail wait-primary.
4. Where a timeout argument is supplied (wait-primary <timeout_ms>), the xce_test client shall use it as the maximum wait; where it is omitted, the client shall use a default timeout.

Requirement 5: Outcome assertions

Objective: As an E2E script author, I want assertion commands that fail the session on mismatch, so that durability/availability expectations are checked in-tool rather than by parsing output.

Acceptance Criteria

1. When expect-jobs N is issued and the current job count equals N, the xce_test client shall treat the assertion as passed.
2. If expect-jobs N is issued and the current job count does not equal N, then the xce_test client shall fail the assertion (non-zero status) and report the expected and actual counts.
3. When expect-job <jobid> is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed.
4. If expect-job <jobid> is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId.

Superseded by job-array-one-based-index (2026-10-01): for arrays, expect-job <jobid> no longer matches text: it compares job numbers, an identity without [k] that no job has fails with a hint that elements must be named <array-id>[k], <array-id>[k] checks element k, and a range <array-id>[a-b] is refused (job-array-one-based-index Requirement 4.7-4.9, 5.13, 5.15). Kept for history.

Requirement 6: Script ergonomics

Objective: As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting.

Acceptance Criteria

1. When a script line's first non-whitespace character is #, the xce_test client shall treat the line as a comment and ignore it.
2. When a script line is empty or whitespace-only, the xce_test client shall ignore it without error.
3. When echo <message> is issued, the xce_test client shall emit the message to output and treat the command as successful.

Requirement 7: Correct job-submission reporting

Objective: As an E2E script author, I want jobsub to report success only when a job was genuinely accepted, so that scripts never chase a job the cluster never committed (bug-786).

Acceptance Criteria

1. When jobsub receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set $jobid to that JobId.

Superseded by job-array-one-based-index (2026-10-01): for jobsub -a N (N >= 2) the reported identity and $jobid are the array identity, not an element JobId; elements are $jobid[1]..$jobid[N] (job-array-one-based-index Requirement 4.1, 4.5). Kept for history.

2. If jobsub does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status.
3. The xce_test client shall never report jobsub success with a malformed or uninitialized JobId.

Requirement 8: Reference E2E driver

Objective: As a maintainer, I want the existing durability soak to use these primitives, so that there is a working reference for script-driven E2E and the primitives are exercised.

Acceptance Criteria

1. The test-commit-durability.sh soak shall be converted to drive xce_test using the exit-code verdict and the new primitives instead of grepping prose and filtering sanitizer output.
2. When the converted soak runs against a healthy cluster, it shall report pass/fail using xce_test exit codes and the assertion commands.
3. The converted soak shall preserve its existing coverage (rolling primary kill + restart, zero acknowledged-job loss across failover).