checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
154
.kiro/specs/xce-test-e2e-primitives/design.md
Normal file
154
.kiro/specs/xce-test-e2e-primitives/design.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
# Design Document — xce-test-e2e-primitives
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose:** Make the `xce_test` client usable as a script-driven E2E driver: a piped command sequence whose **process exit code is the pass/fail verdict**, able to drive node lifecycle via a shell-escape, wait for failover deterministically, and assert outcomes in-tool.
|
||||
|
||||
**Users:** Test/CI authors and maintainers writing failover/durability/cold-restart E2E scenarios (e.g. the `test-commit-durability.sh` soak).
|
||||
|
||||
**Impact:** Today `xce_test` is an interactive client: `execute_line()` parses a line into argv, dispatches to a `command_t.func` (returning `int`, `<0` = error), and the main loop logs failures but **continues and exits 0**. Success is only inferable by grepping prose, and `jobsub` can print a garbage JobId (bug-786). This feature adds: session failure tracking + a real exit code; a `!` shell-escape; a fixed auto-variable substitution pass (`$?`, `$jobid`, `$jobcount`); `wait-primary`, `expect-jobs`, `expect-job`, `echo` commands; `#`/blank-line tolerance; and a `jobsub` reply-status fix. Everything lives in the single test-client TU; the shell remains the orchestrator.
|
||||
|
||||
### Goals
|
||||
- The process exit code reflects whether any command failed (Req 1).
|
||||
- A `!cmd` shell-escape lets one script drive node lifecycle; its status feeds `$?` (Req 2).
|
||||
- A fixed, tool-maintained auto-variable set (`$?`, `$jobid`, `$jobcount`) — no user variables/expressions (Req 3).
|
||||
- `wait-primary` blocks on the existing client failover until an op succeeds (Req 4).
|
||||
- `expect-jobs`/`expect-job` assertions fail the session on mismatch (Req 5).
|
||||
- `jobsub` reports success only on a clean ack with a valid JobId (Req 7).
|
||||
|
||||
### Non-Goals
|
||||
- Structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test`; user-assignable variables, expressions, conditionals, or loops (the shell owns these).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The `xce_test` command loop's **failure tracking + process exit code**, an **abort-vs-continue** mode flag, and `#`/blank-line handling.
|
||||
- A **line pre-processing step** in the client that (a) recognizes a leading `!` as a shell-escape and (b) substitutes the fixed auto-variables before tokenizing.
|
||||
- The fixed **auto-variable state** (`$?`, `$jobid`, `$jobcount`) and the points that set it.
|
||||
- New client commands: `wait-primary`, `expect-jobs`, `expect-job`, `echo`; the `set continue-on-error` toggle.
|
||||
- The **`cmd_jobsub` reply-status correctness** fix (bug-786).
|
||||
- The conversion of `src/tools/test-commit-durability.sh` to use these primitives.
|
||||
|
||||
### Out of Boundary
|
||||
- Controller/replication behavior and opcodes (consumed unchanged).
|
||||
- libxce client failover/master-rotation internals (consumed by `wait-primary`/`jobsub`/`joblist`).
|
||||
- Node lifecycle (deploy/kill/restart) — performed by the shell, reachable via `!cmd`.
|
||||
- Any general scripting language (variables/expressions/control flow).
|
||||
|
||||
### Allowed Dependencies
|
||||
- The existing `command_t`/`cmd_func_t` dispatch, `execute_line`, `parse_argv`, `find_command`, and the main read loop in `test.c`.
|
||||
- The existing client context (`active_ctx`), `cmd_jobsub`/`cmd_joblist` paths, and the libxce request/failover APIs they already call.
|
||||
- Standard C `system()` for the shell-escape; a monotonic clock helper for `wait-primary` timeout.
|
||||
|
||||
### Revalidation Triggers
|
||||
- Any change to the `cmd_func_t` contract (return-code semantics) → re-check all existing command handlers.
|
||||
- A change to how `jobsub`/`joblist` report acks or list jobs → re-check `$jobid`/`$jobcount` capture and the assertions.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
- `execute_line(char* line)`: `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; returns the handler's `int` rc (`<0` error).
|
||||
- Main loop: reads a line, `rc = execute_line(s)`; on `rc < 0` sets errno via `SET_ERROR` and **continues**; `main` returns 0. So failures never affect the exit code.
|
||||
- Commands return `int` (`0` ok, `<0` errno-style). `cmd_jobsub`/`cmd_joblist` already drive the client + libxce failover.
|
||||
|
||||
### Pattern
|
||||
Extend-in-place. Introduce a thin **pre-processing + verdict** wrapper around the existing dispatch; the new commands are ordinary `command_t` entries. No new module, no new files.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Loop[main read loop] --> Pre[preprocess line]
|
||||
Pre -->|leading '#'/blank| Skip[ignore]
|
||||
Pre -->|leading '!'| Shell[run via system; set $?]
|
||||
Pre -->|else| Subst[substitute $? / $jobid / $jobcount]
|
||||
Subst --> Exec[execute_line -> find_command -> func]
|
||||
Exec --> Track[update session-failed + $?]
|
||||
Shell --> Track
|
||||
Track -->|abort mode + failed| Stop[stop loop]
|
||||
Track --> Loop
|
||||
Loop --> Verdict[main returns 1 if any failure else 0]
|
||||
```
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/controller/testsuite/test.c` — all client changes:
|
||||
- **Session/verdict state** (file-scope): `g_session_failed` (int), `g_continue_on_error` (bool), and auto-vars `g_last_status` (int), `g_last_jobid[XCE_MAXNAME_LEN]` (string), `g_last_jobcount` (int).
|
||||
- **`preprocess_and_run(char* line)`** (new): handle blank/`#`; leading `!` → `system(rest)`, set `g_last_status` from `WEXITSTATUS`; else substitute auto-vars then call `execute_line`. Update `g_session_failed`/`g_last_status` from the result; honor abort-vs-continue. Called from the main loop in place of the direct `execute_line`.
|
||||
- **`substitute_autovars(line) → expanded`** (new): replace whole-token `$?`, `$jobid`, `$jobcount` with current values; a reference with no current value yields a command failure (Req 3.6).
|
||||
- **New command handlers**: `cmd_wait_primary`, `cmd_expect_jobs`, `cmd_expect_job`, `cmd_echo`, registered in the command table(s).
|
||||
- **`cmd_set`**: add `continue-on-error` toggle.
|
||||
- **`cmd_jobsub`**: gate the "ok" report on a successful reply status + valid JobId; on success set `g_last_jobid`; otherwise print a typed error and return `<0` (bug-786).
|
||||
- **`cmd_joblist`**: set `g_last_jobcount`; expose a reusable internal count/contains helper for the assertions.
|
||||
- **`main`**: return `g_session_failed ? 1 : 0`.
|
||||
- `src/tools/test-commit-durability.sh` — convert to drive `xce_test` via piped scripts using `wait-primary`/`expect-jobs`/`$jobid` and the exit-code verdict instead of prose-grepping + sanitizer filtering.
|
||||
|
||||
### New Files
|
||||
- None.
|
||||
|
||||
> Note: the process exit code is only a trustworthy verdict if a clean (leak-free) exit is possible. Reuse the existing test run convention (`ASAN_OPTIONS=detect_leaks=0`) for the soak; leak cleanup of the client is out of scope here and tracked separately.
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Intent | Req Coverage |
|
||||
|-----------|--------|--------------|
|
||||
| Verdict & loop wrapper (`preprocess_and_run`, `g_session_failed`, exit code) | Turn command outcomes into a process verdict; abort/continue | 1.1–1.6, 6.1, 6.2 |
|
||||
| Shell-escape (`!cmd` via `system`) | Drive node lifecycle / setup from a script | 2.1–2.4 |
|
||||
| Auto-vars (`substitute_autovars`, `g_last_*`) | Reference last status / JobId / job count | 3.1–3.6 |
|
||||
| `cmd_wait_primary` | Block until a client op succeeds (failover-aware) | 4.1–4.4 |
|
||||
| `cmd_expect_jobs` / `cmd_expect_job` | Assert job count / membership; fail session on mismatch | 5.1–5.4 |
|
||||
| `cmd_echo` | Progress/section messages | 6.3 |
|
||||
| `cmd_jobsub` fix | Report success only on clean ack + valid JobId; set `$jobid` | 7.1–7.3, 3.2 |
|
||||
| `test-commit-durability.sh` driver | Reference E2E using the primitives | 8.1–8.3 |
|
||||
|
||||
### Verdict & loop wrapper
|
||||
- **State (file-scope):** `g_session_failed`, `g_continue_on_error`, `g_last_status`, `g_last_jobid[]`, `g_last_jobcount`.
|
||||
- **Contract:** every executed command updates `g_last_status` and, on failure, sets `g_session_failed`. In default mode the loop stops on first failure; under `continue-on-error` it keeps going. `main` returns non-zero iff `g_session_failed`.
|
||||
|
||||
### Shell-escape
|
||||
- A line whose first non-whitespace char is `!` runs `system(rest_of_line)`. `g_last_status = WEXITSTATUS(rc)` (or non-zero on `system` failure). A non-zero status counts as a command failure.
|
||||
|
||||
### Auto-variables
|
||||
- Whole-token substitution of `$?` (→ `g_last_status`), `$jobid` (→ `g_last_jobid`), `$jobcount` (→ `g_last_jobcount`) before tokenizing. Unset `$jobid`/`$jobcount` → command failure (no garbage). No other variable syntax is recognized.
|
||||
|
||||
### cmd_wait_primary
|
||||
- `wait-primary [timeout_ms]`: loop a cheap client op (e.g. the `joblist`/probe path through `active_ctx`, which already does master rotation) until success or the timeout (monotonic clock); return 0 on success, `<0` on timeout. Default timeout when arg omitted.
|
||||
|
||||
### cmd_expect_jobs / cmd_expect_job
|
||||
- `expect-jobs N`: obtain the current job count (shared helper with `cmd_joblist`); equal → 0, else print expected/actual and return `<0`.
|
||||
- `expect-job <jobid>`: list jobs; present → 0, else print the missing id and return `<0`.
|
||||
|
||||
### cmd_jobsub (bug-786 fix)
|
||||
- Inspect the reply: only when the controller returns a success status with a well-formed JobId does it print `jobsubmit ok: JobId: <id>` and set `g_last_jobid`. Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id.
|
||||
|
||||
## Error Handling
|
||||
- **Command failure** → `g_last_status != 0`, `g_session_failed = 1`; abort (default) or continue. Non-zero process exit at end.
|
||||
- **Shell-escape failure** → treated as a command failure via `$?`.
|
||||
- **Unset auto-var reference** → command failure with a clear message (never substitutes garbage).
|
||||
- **`wait-primary` timeout** and **assertion mismatch** → command failure with expected/actual reported.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit / behavioral (driven through `xce_test` itself; no controller needed for most)
|
||||
- Exit code: a script with all-passing commands exits 0; a script with a failing `expect-jobs` exits non-zero (Req 1.1–1.3).
|
||||
- Abort vs continue: a failing command mid-script stops execution by default; under `set continue-on-error` later commands still run and the final exit is non-zero (Req 1.4, 1.5).
|
||||
- Shell-escape: `!true` → `$?`=0, success; `!false` → `$?`≠0, session fails (Req 2.1–2.4).
|
||||
- Auto-vars: `$?` reflects the prior command; an unset `$jobid` reference fails rather than substituting garbage (Req 3.1, 3.6).
|
||||
- Ergonomics: `#` comment and blank lines are ignored; `echo` succeeds (Req 6).
|
||||
- `jobsub` reporting: a non-acked submit returns failure and does not set `$jobid`; never prints a malformed id (Req 7) — verifiable against a live single node or by simulating a non-ack.
|
||||
|
||||
### Integration / E2E (live `/xce-deploy-ha` cluster)
|
||||
- Reference driver: converted `test-commit-durability.sh` runs a scenario — `wait-primary`, `jobsub` (capture `$jobid`), `expect-jobs`, `!kill` the primary, `wait-primary`, `expect-job $jobid` — and gates on the `xce_test` exit code (Req 4, 5, 8). Adding a `!`-driven cold-restart-all + `expect-jobs` step exercises the durability path (would catch a bug-778-class regression).
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Component | Test |
|
||||
|-------------|-----------|------|
|
||||
| 1.1–1.6 | Verdict & loop wrapper | exit-code + abort/continue tests |
|
||||
| 2.1–2.4 | Shell-escape | `!true`/`!false` tests |
|
||||
| 3.1–3.6 | Auto-vars | `$?`/`$jobid` substitution + unset-fails tests |
|
||||
| 4.1–4.4 | cmd_wait_primary | live failover wait |
|
||||
| 5.1–5.4 | cmd_expect_jobs / cmd_expect_job | assertion pass/fail tests |
|
||||
| 6.1–6.3 | loop wrapper + cmd_echo | comment/blank/echo tests |
|
||||
| 7.1–7.3 | cmd_jobsub fix | non-ack returns failure; no garbage id |
|
||||
| 8.1–8.3 | test-commit-durability.sh | converted soak runs, exit-code-gated |
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue