lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/xce-test-e2e-primitives/research.md
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

2 KiB
Raw Blame History

Research / Discovery — xce-test-e2e-primitives

Discovery type

LIGHT (extension to an existing single-file test client). No external research needed.

Key findings (from src/controller/testsuite/test.c)

  • Dispatch: execute_line(line) → parse_argv → find_command(argv[0]) → command->func(argc, argv); handlers return int (<0 = errno-style error). Command table is command_t {name, func, doc} arrays per context.
  • Main read loop reads a line, rc = execute_line(s), and on rc<0 only SET_ERROR(-rc) + continues; main returns 0 → failures never affect the exit code (the core gap for scriptable E2E).
  • cmd_jobsub/cmd_joblist already drive the client via active_ctx and libxce client-side master-rotation/failover — wait-primary and the assertions can reuse these paths.
  • bug-786: cmd_jobsub can print jobsubmit ok: JobId: 63-15--1-... (uninitialized id) on a non-clean ack; controller then logs handle_jobsubmit fail to lock job <garbage>. Root: success is reported without verifying the reply status/jobid.

Design decisions

  • Extend-in-place, single TU. Add a preprocess_and_run wrapper (blank/#/!/auto-var-substitution) + file-scope verdict state; new commands as ordinary table entries; main returns g_session_failed ? 1 : 0. No new files, no new opcode, no JSON.
  • Fixed auto-vars only ($?, $jobid, $jobcount) — substitution, not a variable system; unset reference fails (no garbage).
  • Shell-escape via system(), status → $?; node lifecycle stays in the shell.
  • wait-primary loops an existing failover-aware client op until success/timeout (monotonic clock).
  • Exit-code trustworthiness depends on a clean exit; reuse ASAN_OPTIONS=detect_leaks=0 for the soak (client leak cleanup is separate/out of scope).

Risks

  • system() shell-escape = arbitrary execution — acceptable for a dev/test client only.
  • Auto-var substitution must be whole-token and conservative to avoid surprising expansions.