lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

154
.kiro/specs/xce-test-e2e-primitives/design.md Normal file
View file

@ -0,0 +1,154 @@
# Design Document — xce-test-e2e-primitives
## Overview
**Purpose:** Make the `xce_test` client usable as a script-driven E2E driver: a piped command sequence whose **process exit code is the pass/fail verdict**, able to drive node lifecycle via a shell-escape, wait for failover deterministically, and assert outcomes in-tool.
**Users:** Test/CI authors and maintainers writing failover/durability/cold-restart E2E scenarios (e.g. the `test-commit-durability.sh` soak).
**Impact:** Today `xce_test` is an interactive client: `execute_line()` parses a line into argv, dispatches to a `command_t.func` (returning `int`, `<0` = error), and the main loop logs failures but **continues and exits 0**. Success is only inferable by grepping prose, and `jobsub` can print a garbage JobId (bug-786). This feature adds: session failure tracking + a real exit code; a `!` shell-escape; a fixed auto-variable substitution pass (`$?`, `$jobid`, `$jobcount`); `wait-primary`, `expect-jobs`, `expect-job`, `echo` commands; `#`/blank-line tolerance; and a `jobsub` reply-status fix. Everything lives in the single test-client TU; the shell remains the orchestrator.
### Goals
- The process exit code reflects whether any command failed (Req 1).
- A `!cmd` shell-escape lets one script drive node lifecycle; its status feeds `$?` (Req 2).
- A fixed, tool-maintained auto-variable set (`$?`, `$jobid`, `$jobcount`) — no user variables/expressions (Req 3).
- `wait-primary` blocks on the existing client failover until an op succeeds (Req 4).
- `expect-jobs`/`expect-job` assertions fail the session on mismatch (Req 5).
- `jobsub` reports success only on a clean ack with a valid JobId (Req 7).
### Non-Goals
- Structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test`; user-assignable variables, expressions, conditionals, or loops (the shell owns these).
## Boundary Commitments
### This Spec Owns
- The `xce_test` command loop's **failure tracking + process exit code**, an **abort-vs-continue** mode flag, and `#`/blank-line handling.
- A **line pre-processing step** in the client that (a) recognizes a leading `!` as a shell-escape and (b) substitutes the fixed auto-variables before tokenizing.
- The fixed **auto-variable state** (`$?`, `$jobid`, `$jobcount`) and the points that set it.
- New client commands: `wait-primary`, `expect-jobs`, `expect-job`, `echo`; the `set continue-on-error` toggle.
- The **`cmd_jobsub` reply-status correctness** fix (bug-786).
- The conversion of `src/tools/test-commit-durability.sh` to use these primitives.
### Out of Boundary
- Controller/replication behavior and opcodes (consumed unchanged).
- libxce client failover/master-rotation internals (consumed by `wait-primary`/`jobsub`/`joblist`).
- Node lifecycle (deploy/kill/restart) — performed by the shell, reachable via `!cmd`.
- Any general scripting language (variables/expressions/control flow).
### Allowed Dependencies
- The existing `command_t`/`cmd_func_t` dispatch, `execute_line`, `parse_argv`, `find_command`, and the main read loop in `test.c`.
- The existing client context (`active_ctx`), `cmd_jobsub`/`cmd_joblist` paths, and the libxce request/failover APIs they already call.
- Standard C `system()` for the shell-escape; a monotonic clock helper for `wait-primary` timeout.
### Revalidation Triggers
- Any change to the `cmd_func_t` contract (return-code semantics) → re-check all existing command handlers.
- A change to how `jobsub`/`joblist` report acks or list jobs → re-check `$jobid`/`$jobcount` capture and the assertions.
## Architecture
### Existing Architecture Analysis
- `execute_line(char* line)`: `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; returns the handler's `int` rc (`<0` error).
- Main loop: reads a line, `rc = execute_line(s)`; on `rc < 0` sets errno via `SET_ERROR` and **continues**; `main` returns 0. So failures never affect the exit code.
- Commands return `int` (`0` ok, `<0` errno-style). `cmd_jobsub`/`cmd_joblist` already drive the client + libxce failover.
### Pattern
Extend-in-place. Introduce a thin **pre-processing + verdict** wrapper around the existing dispatch; the new commands are ordinary `command_t` entries. No new module, no new files.
```mermaid
graph TD
Loop[main read loop] --> Pre[preprocess line]
Pre -->|leading '#'/blank| Skip[ignore]
Pre -->|leading '!'| Shell[run via system; set $?]
Pre -->|else| Subst[substitute $? / $jobid / $jobcount]
Subst --> Exec[execute_line -> find_command -> func]
Exec --> Track[update session-failed + $?]
Shell --> Track
Track -->|abort mode + failed| Stop[stop loop]
Track --> Loop
Loop --> Verdict[main returns 1 if any failure else 0]
```
## File Structure Plan
### Modified Files
- `src/controller/testsuite/test.c` — all client changes:
- **Session/verdict state** (file-scope): `g_session_failed` (int), `g_continue_on_error` (bool), and auto-vars `g_last_status` (int), `g_last_jobid[XCE_MAXNAME_LEN]` (string), `g_last_jobcount` (int).
- **`preprocess_and_run(char* line)`** (new): handle blank/`#`; leading `!` → `system(rest)`, set `g_last_status` from `WEXITSTATUS`; else substitute auto-vars then call `execute_line`. Update `g_session_failed`/`g_last_status` from the result; honor abort-vs-continue. Called from the main loop in place of the direct `execute_line`.
- **`substitute_autovars(line) → expanded`** (new): replace whole-token `$?`, `$jobid`, `$jobcount` with current values; a reference with no current value yields a command failure (Req 3.6).
- **New command handlers**: `cmd_wait_primary`, `cmd_expect_jobs`, `cmd_expect_job`, `cmd_echo`, registered in the command table(s).
- **`cmd_set`**: add `continue-on-error` toggle.
- **`cmd_jobsub`**: gate the "ok" report on a successful reply status + valid JobId; on success set `g_last_jobid`; otherwise print a typed error and return `<0` (bug-786).
- **`cmd_joblist`**: set `g_last_jobcount`; expose a reusable internal count/contains helper for the assertions.
- **`main`**: return `g_session_failed ? 1 : 0`.
- `src/tools/test-commit-durability.sh` — convert to drive `xce_test` via piped scripts using `wait-primary`/`expect-jobs`/`$jobid` and the exit-code verdict instead of prose-grepping + sanitizer filtering.
### New Files
- None.
> Note: the process exit code is only a trustworthy verdict if a clean (leak-free) exit is possible. Reuse the existing test run convention (`ASAN_OPTIONS=detect_leaks=0`) for the soak; leak cleanup of the client is out of scope here and tracked separately.
## Components and Interfaces
| Component | Intent | Req Coverage |
|-----------|--------|--------------|
| Verdict & loop wrapper (`preprocess_and_run`, `g_session_failed`, exit code) | Turn command outcomes into a process verdict; abort/continue | 1.1–1.6, 6.1, 6.2 |
| Shell-escape (`!cmd` via `system`) | Drive node lifecycle / setup from a script | 2.1–2.4 |
| Auto-vars (`substitute_autovars`, `g_last_*`) | Reference last status / JobId / job count | 3.1–3.6 |
| `cmd_wait_primary` | Block until a client op succeeds (failover-aware) | 4.1–4.4 |
| `cmd_expect_jobs` / `cmd_expect_job` | Assert job count / membership; fail session on mismatch | 5.1–5.4 |
| `cmd_echo` | Progress/section messages | 6.3 |
| `cmd_jobsub` fix | Report success only on clean ack + valid JobId; set `$jobid` | 7.1–7.3, 3.2 |
| `test-commit-durability.sh` driver | Reference E2E using the primitives | 8.1–8.3 |
### Verdict & loop wrapper
- **State (file-scope):** `g_session_failed`, `g_continue_on_error`, `g_last_status`, `g_last_jobid[]`, `g_last_jobcount`.
- **Contract:** every executed command updates `g_last_status` and, on failure, sets `g_session_failed`. In default mode the loop stops on first failure; under `continue-on-error` it keeps going. `main` returns non-zero iff `g_session_failed`.
### Shell-escape
- A line whose first non-whitespace char is `!` runs `system(rest_of_line)`. `g_last_status = WEXITSTATUS(rc)` (or non-zero on `system` failure). A non-zero status counts as a command failure.
### Auto-variables
- Whole-token substitution of `$?` (→ `g_last_status`), `$jobid` (→ `g_last_jobid`), `$jobcount` (→ `g_last_jobcount`) before tokenizing. Unset `$jobid`/`$jobcount` → command failure (no garbage). No other variable syntax is recognized.
### cmd_wait_primary
- `wait-primary [timeout_ms]`: loop a cheap client op (e.g. the `joblist`/probe path through `active_ctx`, which already does master rotation) until success or the timeout (monotonic clock); return 0 on success, `<0` on timeout. Default timeout when arg omitted.
### cmd_expect_jobs / cmd_expect_job
- `expect-jobs N`: obtain the current job count (shared helper with `cmd_joblist`); equal → 0, else print expected/actual and return `<0`.
- `expect-job <jobid>`: list jobs; present → 0, else print the missing id and return `<0`.
### cmd_jobsub (bug-786 fix)
- Inspect the reply: only when the controller returns a success status with a well-formed JobId does it print `jobsubmit ok: JobId: <id>` and set `g_last_jobid`. Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id.
## Error Handling
- **Command failure** → `g_last_status != 0`, `g_session_failed = 1`; abort (default) or continue. Non-zero process exit at end.
- **Shell-escape failure** → treated as a command failure via `$?`.
- **Unset auto-var reference** → command failure with a clear message (never substitutes garbage).
- **`wait-primary` timeout** and **assertion mismatch** → command failure with expected/actual reported.
## Testing Strategy
### Unit / behavioral (driven through `xce_test` itself; no controller needed for most)
- Exit code: a script with all-passing commands exits 0; a script with a failing `expect-jobs` exits non-zero (Req 1.1–1.3).
- Abort vs continue: a failing command mid-script stops execution by default; under `set continue-on-error` later commands still run and the final exit is non-zero (Req 1.4, 1.5).
- Shell-escape: `!true` → `$?`=0, success; `!false` → `$?`≠0, session fails (Req 2.1–2.4).
- Auto-vars: `$?` reflects the prior command; an unset `$jobid` reference fails rather than substituting garbage (Req 3.1, 3.6).
- Ergonomics: `#` comment and blank lines are ignored; `echo` succeeds (Req 6).
- `jobsub` reporting: a non-acked submit returns failure and does not set `$jobid`; never prints a malformed id (Req 7) — verifiable against a live single node or by simulating a non-ack.
### Integration / E2E (live `/xce-deploy-ha` cluster)
- Reference driver: converted `test-commit-durability.sh` runs a scenario — `wait-primary`, `jobsub` (capture `$jobid`), `expect-jobs`, `!kill` the primary, `wait-primary`, `expect-job $jobid` — and gates on the `xce_test` exit code (Req 4, 5, 8). Adding a `!`-driven cold-restart-all + `expect-jobs` step exercises the durability path (would catch a bug-778-class regression).
## Requirements Traceability
| Requirement | Component | Test |
|-------------|-----------|------|
| 1.1–1.6 | Verdict & loop wrapper | exit-code + abort/continue tests |
| 2.1–2.4 | Shell-escape | `!true`/`!false` tests |
| 3.1–3.6 | Auto-vars | `$?`/`$jobid` substitution + unset-fails tests |
| 4.1–4.4 | cmd_wait_primary | live failover wait |
| 5.1–5.4 | cmd_expect_jobs / cmd_expect_job | assertion pass/fail tests |
| 6.1–6.3 | loop wrapper + cmd_echo | comment/blank/echo tests |
| 7.1–7.3 | cmd_jobsub fix | non-ack returns failure; no garbage id |
| 8.1–8.3 | test-commit-durability.sh | converted soak runs, exit-code-gated |
```

94
.kiro/specs/xce-test-e2e-primitives/requirements.md Normal file
View file

@ -0,0 +1,94 @@
# Requirements Document
## Introduction
This feature adds a small set of scripting primitives to the `xce_test` client so that a complete end-to-end scenario — submit work, kill the primary, wait for failover, restart/cold-restart nodes, and assert that committed work survived — can be driven from a single piped or scripted command sequence whose **process exit code is the pass/fail verdict**. Today `xce_test` is an interactive client whose success can only be inferred by grepping human-readable output (and which can emit a malformed JobId), so E2E scripts must filter prose, suppress sanitizer noise, and poll controller logs for cluster state. These primitives let the shell remain the orchestrator (node lifecycle, loops) while `xce_test` reliably reports outcomes.
## Boundary Context
- **In scope**: command-failure tracking and a process exit code that reflects it; a `!`-prefixed shell-escape command; a fixed set of tool-maintained auto-variables (`$?`, `$jobid`, optional `$jobcount`); a `wait-primary` blocking command; `expect-jobs` / `expect-job` assertion commands; script ergonomics (`#` comments, blank lines, `echo`); correct `jobsub` success/failure reporting (bug-786); converting `src/tools/test-commit-durability.sh` to use these primitives.
- **Out of scope**: structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test` (kill/restart/deploy stay in the shell, reachable via `!cmd`); user-assignable variables, expression evaluation, conditionals, or loops (the shell owns these); any change to controller/replication behavior.
- **Adjacent expectations**: `wait-primary` and `jobsub`/`joblist` rely on the existing libxce client-side master-rotation/failover already built into `xce_test`; this feature consumes that behavior and does not change it.
## Requirements
### Requirement 1: Exit code reflects command outcomes
**Objective:** As an E2E script author, I want the `xce_test` process to exit non-zero when any command in the session failed, so that a shell can use the exit code alone as the pass/fail verdict.
#### Acceptance Criteria
1. When a command in the session fails, the xce_test client shall record the session as failed.
2. When the session ends and at least one command failed, the xce_test client shall exit with a non-zero status.
3. When the session ends and every command succeeded, the xce_test client shall exit with status zero.
4. While default (abort-on-failure) mode is active, when a command fails, the xce_test client shall stop executing subsequent commands and exit non-zero.
5. Where `set continue-on-error` has been issued, when a command fails, the xce_test client shall continue executing subsequent commands and still exit non-zero at session end.
6. The xce_test client shall send all human-readable log and diagnostic output in a way that does not corrupt the determination of the exit-code verdict.
### Requirement 2: Shell-escape command
**Objective:** As an E2E script author, I want to run an arbitrary shell command from within an `xce_test` script, so that a single script can drive node lifecycle (kill, restart, deploy) and environment setup without a separate orchestrator.
#### Acceptance Criteria
1. When a script line begins with `!`, the xce_test client shall execute the remainder of the line as a shell command.
2. When the shell command exits zero, the xce_test client shall treat the `!` command as successful.
3. If the shell command exits non-zero, then the xce_test client shall treat the `!` command as failed (subject to Requirement 1's exit-code and abort/continue behavior).
4. When a `!` command completes, the xce_test client shall set `$?` to the shell command's exit status.
### Requirement 3: Tool-maintained auto-variables
**Objective:** As an E2E script author, I want a fixed set of auto-variables that later commands can reference, so that I can assert on outcomes without parsing client output in the shell.
#### Acceptance Criteria
1. When any command completes, the xce_test client shall set `$?` to that command's status (zero on success, non-zero on failure).
2. When a `jobsub` command receives a successful acknowledgment, the xce_test client shall set `$jobid` to the acknowledged JobId.
3. When a subsequent command references `$jobid` (e.g. `expect-job $jobid`), the xce_test client shall substitute the most recently acknowledged JobId.
4. Where the `$jobcount` auto-variable is provided, when a `joblist` command completes, the xce_test client shall set `$jobcount` to the number of jobs returned.
5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables.
6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value.
### Requirement 4: Wait for an elected primary
**Objective:** As an E2E script author, I want a command that blocks until the cluster has a serving primary, so that scripts can proceed deterministically after a kill or restart without fixed sleeps or log polling.
#### Acceptance Criteria
1. When `wait-primary` is issued, the xce_test client shall repeatedly attempt a client operation against the configured masters until one succeeds.
2. When a client operation succeeds within the timeout, the xce_test client shall complete `wait-primary` successfully.
3. If no client operation succeeds within the timeout, then the xce_test client shall fail `wait-primary`.
4. Where a timeout argument is supplied (`wait-primary <timeout_ms>`), the xce_test client shall use it as the maximum wait; where it is omitted, the client shall use a default timeout.
### Requirement 5: Outcome assertions
**Objective:** As an E2E script author, I want assertion commands that fail the session on mismatch, so that durability/availability expectations are checked in-tool rather than by parsing output.
#### Acceptance Criteria
1. When `expect-jobs N` is issued and the current job count equals `N`, the xce_test client shall treat the assertion as passed.
2. If `expect-jobs N` is issued and the current job count does not equal `N`, then the xce_test client shall fail the assertion (non-zero status) and report the expected and actual counts.
3. When `expect-job <jobid>` is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed.
4. If `expect-job <jobid>` is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId.
### Requirement 6: Script ergonomics
**Objective:** As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting.
#### Acceptance Criteria
1. When a script line's first non-whitespace character is `#`, the xce_test client shall treat the line as a comment and ignore it.
2. When a script line is empty or whitespace-only, the xce_test client shall ignore it without error.
3. When `echo <message>` is issued, the xce_test client shall emit the message to output and treat the command as successful.
### Requirement 7: Correct job-submission reporting
**Objective:** As an E2E script author, I want `jobsub` to report success only when a job was genuinely accepted, so that scripts never chase a job the cluster never committed (bug-786).
#### Acceptance Criteria
1. When `jobsub` receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set `$jobid` to that JobId.
2. If `jobsub` does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status.
3. The xce_test client shall never report `jobsub` success with a malformed or uninitialized JobId.
### Requirement 8: Reference E2E driver
**Objective:** As a maintainer, I want the existing durability soak to use these primitives, so that there is a working reference for script-driven E2E and the primitives are exercised.
#### Acceptance Criteria
1. The `test-commit-durability.sh` soak shall be converted to drive `xce_test` using the exit-code verdict and the new primitives instead of grepping prose and filtering sanitizer output.
2. When the converted soak runs against a healthy cluster, it shall report pass/fail using `xce_test` exit codes and the assertion commands.
3. The converted soak shall preserve its existing coverage (rolling primary kill + restart, zero acknowledged-job loss across failover).

21
.kiro/specs/xce-test-e2e-primitives/research.md Normal file
View file

@ -0,0 +1,21 @@
# Research / Discovery — xce-test-e2e-primitives
## Discovery type
LIGHT (extension to an existing single-file test client). No external research needed.
## Key findings (from src/controller/testsuite/test.c)
- Dispatch: `execute_line(line)` → `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; handlers return `int` (`<0` = errno-style error). Command table is `command_t {name, func, doc}` arrays per context.
- Main read loop reads a line, `rc = execute_line(s)`, and on `rc<0` only `SET_ERROR(-rc)` + continues; `main` returns 0 → **failures never affect the exit code** (the core gap for scriptable E2E).
- `cmd_jobsub`/`cmd_joblist` already drive the client via `active_ctx` and libxce client-side master-rotation/failover — `wait-primary` and the assertions can reuse these paths.
- bug-786: `cmd_jobsub` can print `jobsubmit ok: JobId: 63-15--1-...` (uninitialized id) on a non-clean ack; controller then logs `handle_jobsubmit fail to lock job <garbage>`. Root: success is reported without verifying the reply status/jobid.
## Design decisions
- **Extend-in-place, single TU.** Add a `preprocess_and_run` wrapper (blank/`#`/`!`/auto-var-substitution) + file-scope verdict state; new commands as ordinary table entries; `main` returns `g_session_failed ? 1 : 0`. No new files, no new opcode, no JSON.
- **Fixed auto-vars only** (`$?`, `$jobid`, `$jobcount`) — substitution, not a variable system; unset reference fails (no garbage).
- **Shell-escape via `system()`**, status → `$?`; node lifecycle stays in the shell.
- **`wait-primary`** loops an existing failover-aware client op until success/timeout (monotonic clock).
- Exit-code trustworthiness depends on a clean exit; reuse `ASAN_OPTIONS=detect_leaks=0` for the soak (client leak cleanup is separate/out of scope).
## Risks
- `system()` shell-escape = arbitrary execution — acceptable for a dev/test client only.
- Auto-var substitution must be whole-token and conservative to avoid surprising expansions.