checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
21
.kiro/specs/xce-test-e2e-primitives/research.md
Normal file
21
.kiro/specs/xce-test-e2e-primitives/research.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# Research / Discovery — xce-test-e2e-primitives
|
||||
|
||||
## Discovery type
|
||||
LIGHT (extension to an existing single-file test client). No external research needed.
|
||||
|
||||
## Key findings (from src/controller/testsuite/test.c)
|
||||
- Dispatch: `execute_line(line)` → `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; handlers return `int` (`<0` = errno-style error). Command table is `command_t {name, func, doc}` arrays per context.
|
||||
- Main read loop reads a line, `rc = execute_line(s)`, and on `rc<0` only `SET_ERROR(-rc)` + continues; `main` returns 0 → **failures never affect the exit code** (the core gap for scriptable E2E).
|
||||
- `cmd_jobsub`/`cmd_joblist` already drive the client via `active_ctx` and libxce client-side master-rotation/failover — `wait-primary` and the assertions can reuse these paths.
|
||||
- bug-786: `cmd_jobsub` can print `jobsubmit ok: JobId: 63-15--1-...` (uninitialized id) on a non-clean ack; controller then logs `handle_jobsubmit fail to lock job <garbage>`. Root: success is reported without verifying the reply status/jobid.
|
||||
|
||||
## Design decisions
|
||||
- **Extend-in-place, single TU.** Add a `preprocess_and_run` wrapper (blank/`#`/`!`/auto-var-substitution) + file-scope verdict state; new commands as ordinary table entries; `main` returns `g_session_failed ? 1 : 0`. No new files, no new opcode, no JSON.
|
||||
- **Fixed auto-vars only** (`$?`, `$jobid`, `$jobcount`) — substitution, not a variable system; unset reference fails (no garbage).
|
||||
- **Shell-escape via `system()`**, status → `$?`; node lifecycle stays in the shell.
|
||||
- **`wait-primary`** loops an existing failover-aware client op until success/timeout (monotonic clock).
|
||||
- Exit-code trustworthiness depends on a clean exit; reuse `ASAN_OPTIONS=detect_leaks=0` for the soak (client leak cleanup is separate/out of scope).
|
||||
|
||||
## Risks
|
||||
- `system()` shell-escape = arbitrary execution — acceptable for a dev/test client only.
|
||||
- Auto-var substitution must be whole-token and conservative to avoid surprising expansions.
|
||||
Loading…
Add table
Add a link
Reference in a new issue