Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
94 lines
8.5 KiB
Markdown
94 lines
8.5 KiB
Markdown
# Requirements Document
|
|
|
|
## Introduction
|
|
|
|
This feature adds a small set of scripting primitives to the `xce_test` client so that a complete end-to-end scenario — submit work, kill the primary, wait for failover, restart/cold-restart nodes, and assert that committed work survived — can be driven from a single piped or scripted command sequence whose **process exit code is the pass/fail verdict**. Today `xce_test` is an interactive client whose success can only be inferred by grepping human-readable output (and which can emit a malformed JobId), so E2E scripts must filter prose, suppress sanitizer noise, and poll controller logs for cluster state. These primitives let the shell remain the orchestrator (node lifecycle, loops) while `xce_test` reliably reports outcomes.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: command-failure tracking and a process exit code that reflects it; a `!`-prefixed shell-escape command; a fixed set of tool-maintained auto-variables (`$?`, `$jobid`, optional `$jobcount`); a `wait-primary` blocking command; `expect-jobs` / `expect-job` assertion commands; script ergonomics (`#` comments, blank lines, `echo`); correct `jobsub` success/failure reporting (bug-786); converting `src/tools/test-commit-durability.sh` to use these primitives.
|
|
- **Out of scope**: structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test` (kill/restart/deploy stay in the shell, reachable via `!cmd`); user-assignable variables, expression evaluation, conditionals, or loops (the shell owns these); any change to controller/replication behavior.
|
|
- **Adjacent expectations**: `wait-primary` and `jobsub`/`joblist` rely on the existing libxce client-side master-rotation/failover already built into `xce_test`; this feature consumes that behavior and does not change it.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Exit code reflects command outcomes
|
|
|
|
**Objective:** As an E2E script author, I want the `xce_test` process to exit non-zero when any command in the session failed, so that a shell can use the exit code alone as the pass/fail verdict.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a command in the session fails, the xce_test client shall record the session as failed.
|
|
2. When the session ends and at least one command failed, the xce_test client shall exit with a non-zero status.
|
|
3. When the session ends and every command succeeded, the xce_test client shall exit with status zero.
|
|
4. While default (abort-on-failure) mode is active, when a command fails, the xce_test client shall stop executing subsequent commands and exit non-zero.
|
|
5. Where `set continue-on-error` has been issued, when a command fails, the xce_test client shall continue executing subsequent commands and still exit non-zero at session end.
|
|
6. The xce_test client shall send all human-readable log and diagnostic output in a way that does not corrupt the determination of the exit-code verdict.
|
|
|
|
### Requirement 2: Shell-escape command
|
|
|
|
**Objective:** As an E2E script author, I want to run an arbitrary shell command from within an `xce_test` script, so that a single script can drive node lifecycle (kill, restart, deploy) and environment setup without a separate orchestrator.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a script line begins with `!`, the xce_test client shall execute the remainder of the line as a shell command.
|
|
2. When the shell command exits zero, the xce_test client shall treat the `!` command as successful.
|
|
3. If the shell command exits non-zero, then the xce_test client shall treat the `!` command as failed (subject to Requirement 1's exit-code and abort/continue behavior).
|
|
4. When a `!` command completes, the xce_test client shall set `$?` to the shell command's exit status.
|
|
|
|
### Requirement 3: Tool-maintained auto-variables
|
|
|
|
**Objective:** As an E2E script author, I want a fixed set of auto-variables that later commands can reference, so that I can assert on outcomes without parsing client output in the shell.
|
|
|
|
#### Acceptance Criteria
|
|
1. When any command completes, the xce_test client shall set `$?` to that command's status (zero on success, non-zero on failure).
|
|
2. When a `jobsub` command receives a successful acknowledgment, the xce_test client shall set `$jobid` to the acknowledged JobId.
|
|
3. When a subsequent command references `$jobid` (e.g. `expect-job $jobid`), the xce_test client shall substitute the most recently acknowledged JobId.
|
|
4. Where the `$jobcount` auto-variable is provided, when a `joblist` command completes, the xce_test client shall set `$jobcount` to the number of jobs returned.
|
|
5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables.
|
|
6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value.
|
|
|
|
### Requirement 4: Wait for an elected primary
|
|
|
|
**Objective:** As an E2E script author, I want a command that blocks until the cluster has a serving primary, so that scripts can proceed deterministically after a kill or restart without fixed sleeps or log polling.
|
|
|
|
#### Acceptance Criteria
|
|
1. When `wait-primary` is issued, the xce_test client shall repeatedly attempt a client operation against the configured masters until one succeeds.
|
|
2. When a client operation succeeds within the timeout, the xce_test client shall complete `wait-primary` successfully.
|
|
3. If no client operation succeeds within the timeout, then the xce_test client shall fail `wait-primary`.
|
|
4. Where a timeout argument is supplied (`wait-primary <timeout_ms>`), the xce_test client shall use it as the maximum wait; where it is omitted, the client shall use a default timeout.
|
|
|
|
### Requirement 5: Outcome assertions
|
|
|
|
**Objective:** As an E2E script author, I want assertion commands that fail the session on mismatch, so that durability/availability expectations are checked in-tool rather than by parsing output.
|
|
|
|
#### Acceptance Criteria
|
|
1. When `expect-jobs N` is issued and the current job count equals `N`, the xce_test client shall treat the assertion as passed.
|
|
2. If `expect-jobs N` is issued and the current job count does not equal `N`, then the xce_test client shall fail the assertion (non-zero status) and report the expected and actual counts.
|
|
3. When `expect-job <jobid>` is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed.
|
|
4. If `expect-job <jobid>` is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId.
|
|
|
|
### Requirement 6: Script ergonomics
|
|
|
|
**Objective:** As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a script line's first non-whitespace character is `#`, the xce_test client shall treat the line as a comment and ignore it.
|
|
2. When a script line is empty or whitespace-only, the xce_test client shall ignore it without error.
|
|
3. When `echo <message>` is issued, the xce_test client shall emit the message to output and treat the command as successful.
|
|
|
|
### Requirement 7: Correct job-submission reporting
|
|
|
|
**Objective:** As an E2E script author, I want `jobsub` to report success only when a job was genuinely accepted, so that scripts never chase a job the cluster never committed (bug-786).
|
|
|
|
#### Acceptance Criteria
|
|
1. When `jobsub` receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set `$jobid` to that JobId.
|
|
2. If `jobsub` does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status.
|
|
3. The xce_test client shall never report `jobsub` success with a malformed or uninitialized JobId.
|
|
|
|
### Requirement 8: Reference E2E driver
|
|
|
|
**Objective:** As a maintainer, I want the existing durability soak to use these primitives, so that there is a working reference for script-driven E2E and the primitives are exercised.
|
|
|
|
#### Acceptance Criteria
|
|
1. The `test-commit-durability.sh` soak shall be converted to drive `xce_test` using the exit-code verdict and the new primitives instead of grepping prose and filtering sanitizer output.
|
|
2. When the converted soak runs against a healthy cluster, it shall report pass/fail using `xce_test` exit codes and the assertion commands.
|
|
3. The converted soak shall preserve its existing coverage (rolling primary kill + restart, zero acknowledged-job loss across failover).
|