# Requirements Document ## Introduction This feature adds a small set of scripting primitives to the `xce_test` client so that a complete end-to-end scenario — submit work, kill the primary, wait for failover, restart/cold-restart nodes, and assert that committed work survived — can be driven from a single piped or scripted command sequence whose **process exit code is the pass/fail verdict**. Today `xce_test` is an interactive client whose success can only be inferred by grepping human-readable output (and which can emit a malformed JobId), so E2E scripts must filter prose, suppress sanitizer noise, and poll controller logs for cluster state. These primitives let the shell remain the orchestrator (node lifecycle, loops) while `xce_test` reliably reports outcomes. ## Boundary Context - **In scope**: command-failure tracking and a process exit code that reflects it; a `!`-prefixed shell-escape command; a fixed set of tool-maintained auto-variables (`$?`, `$jobid`, optional `$jobcount`); a `wait-primary` blocking command; `expect-jobs` / `expect-job` assertion commands; script ergonomics (`#` comments, blank lines, `echo`); correct `jobsub` success/failure reporting (bug-786); converting `src/tools/test-commit-durability.sh` to use these primitives. - **Out of scope**: structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test` (kill/restart/deploy stay in the shell, reachable via `!cmd`); user-assignable variables, expression evaluation, conditionals, or loops (the shell owns these); any change to controller/replication behavior. - **Adjacent expectations**: `wait-primary` and `jobsub`/`joblist` rely on the existing libxce client-side master-rotation/failover already built into `xce_test`; this feature consumes that behavior and does not change it. ## Requirements ### Requirement 1: Exit code reflects command outcomes **Objective:** As an E2E script author, I want the `xce_test` process to exit non-zero when any command in the session failed, so that a shell can use the exit code alone as the pass/fail verdict. #### Acceptance Criteria 1. When a command in the session fails, the xce_test client shall record the session as failed. 2. When the session ends and at least one command failed, the xce_test client shall exit with a non-zero status. 3. When the session ends and every command succeeded, the xce_test client shall exit with status zero. 4. While default (abort-on-failure) mode is active, when a command fails, the xce_test client shall stop executing subsequent commands and exit non-zero. 5. Where `set continue-on-error` has been issued, when a command fails, the xce_test client shall continue executing subsequent commands and still exit non-zero at session end. 6. The xce_test client shall send all human-readable log and diagnostic output in a way that does not corrupt the determination of the exit-code verdict. ### Requirement 2: Shell-escape command **Objective:** As an E2E script author, I want to run an arbitrary shell command from within an `xce_test` script, so that a single script can drive node lifecycle (kill, restart, deploy) and environment setup without a separate orchestrator. #### Acceptance Criteria 1. When a script line begins with `!`, the xce_test client shall execute the remainder of the line as a shell command. 2. When the shell command exits zero, the xce_test client shall treat the `!` command as successful. 3. If the shell command exits non-zero, then the xce_test client shall treat the `!` command as failed (subject to Requirement 1's exit-code and abort/continue behavior). 4. When a `!` command completes, the xce_test client shall set `$?` to the shell command's exit status. ### Requirement 3: Tool-maintained auto-variables **Objective:** As an E2E script author, I want a fixed set of auto-variables that later commands can reference, so that I can assert on outcomes without parsing client output in the shell. #### Acceptance Criteria 1. When any command completes, the xce_test client shall set `$?` to that command's status (zero on success, non-zero on failure). 2. When a `jobsub` command receives a successful acknowledgment, the xce_test client shall set `$jobid` to the acknowledged JobId. 3. When a subsequent command references `$jobid` (e.g. `expect-job $jobid`), the xce_test client shall substitute the most recently acknowledged JobId. 4. Where the `$jobcount` auto-variable is provided, when a `joblist` command completes, the xce_test client shall set `$jobcount` to the number of jobs returned. 5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables. 6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value. ### Requirement 4: Wait for an elected primary **Objective:** As an E2E script author, I want a command that blocks until the cluster has a serving primary, so that scripts can proceed deterministically after a kill or restart without fixed sleeps or log polling. #### Acceptance Criteria 1. When `wait-primary` is issued, the xce_test client shall repeatedly attempt a client operation against the configured masters until one succeeds. 2. When a client operation succeeds within the timeout, the xce_test client shall complete `wait-primary` successfully. 3. If no client operation succeeds within the timeout, then the xce_test client shall fail `wait-primary`. 4. Where a timeout argument is supplied (`wait-primary `), the xce_test client shall use it as the maximum wait; where it is omitted, the client shall use a default timeout. ### Requirement 5: Outcome assertions **Objective:** As an E2E script author, I want assertion commands that fail the session on mismatch, so that durability/availability expectations are checked in-tool rather than by parsing output. #### Acceptance Criteria 1. When `expect-jobs N` is issued and the current job count equals `N`, the xce_test client shall treat the assertion as passed. 2. If `expect-jobs N` is issued and the current job count does not equal `N`, then the xce_test client shall fail the assertion (non-zero status) and report the expected and actual counts. 3. When `expect-job ` is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed. 4. If `expect-job ` is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId. ### Requirement 6: Script ergonomics **Objective:** As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting. #### Acceptance Criteria 1. When a script line's first non-whitespace character is `#`, the xce_test client shall treat the line as a comment and ignore it. 2. When a script line is empty or whitespace-only, the xce_test client shall ignore it without error. 3. When `echo ` is issued, the xce_test client shall emit the message to output and treat the command as successful. ### Requirement 7: Correct job-submission reporting **Objective:** As an E2E script author, I want `jobsub` to report success only when a job was genuinely accepted, so that scripts never chase a job the cluster never committed (bug-786). #### Acceptance Criteria 1. When `jobsub` receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set `$jobid` to that JobId. 2. If `jobsub` does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status. 3. The xce_test client shall never report `jobsub` success with a malformed or uninitialized JobId. ### Requirement 8: Reference E2E driver **Objective:** As a maintainer, I want the existing durability soak to use these primitives, so that there is a working reference for script-driven E2E and the primitives are exercised. #### Acceptance Criteria 1. The `test-commit-durability.sh` soak shall be converted to drive `xce_test` using the exit-code verdict and the new primitives instead of grepping prose and filtering sanitizer output. 2. When the converted soak runs against a healthy cluster, it shall report pass/fail using `xce_test` exit codes and the assertion commands. 3. The converted soak shall preserve its existing coverage (rolling primary kill + restart, zero acknowledged-job loss across failover).