4 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1721 | fixed | 2026-09-30 | controller | wrong-result | 2026-08-26T00:00:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1721: clean_failover_preservation.sh FAIL: cluster.bin missing under /ha/data (deploy prerequisite)
Description
clean_failover_preservation.sh FAIL: cluster.bin missing under /ha/data (deploy prerequisite)
UPDATE (2026-09-29): once past the ha/data precondition, two more stale assumptions surfaced in the same script: the deploy-ha.sh call used an old 4-argument signature, and both committed-LSN boundaries assumed bug-845's doubled submits (FAIL: jobs did not commit on a majority within 90s (max committed_lsn=3 < 5), then walcompare: FAIL -- committed-prefix divergence at LSN 5 with all three WALs identical at LSN 1-4).
Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (nix dev shell; interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: bash 5.3, gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (controller + harness + xceadmin in one build dir, src/xce.build.bug1871)
- Code: reproduced 2026-09-29 at
980a0054; the script has been broken sinceee8473bd(ha/data removal) - Deployment: local 3-node HA controller cluster the script deploys itself, localhost:29991-29993
Reproduction
XCE_CONTROLLER_BIN=<builddir>/controller/xce_controller XCE_FAILOVER_TOP=<empty dir> nix develop ./src --command bash src/controller/testsuite/clean_failover_preservation.sh
- Frequency: deterministic -- 1/1 runs failed at the precondition, 2026-09-29:
FAIL: cluster.bin missing under <repo>/ha/data (deploy prerequisite) - Evidence: /home/claude/tmp/xce-1721-green (commit-wait failure), /home/claude/tmp/xce-1721-green2 (walcompare past-the-tip failure, walcompare.out), /home/claude/tmp/xce-1721-green3 (passing run)
Root cause
Three stale assumptions in clean_failover_preservation.sh, each hidden behind the previous one. (1) It required /ha/data/{cluster.bin,xce.crt,xce.key} and passed that dir to deploy-ha.sh, but ee8473bd removed ha/data/ and deploy-ha.sh now generates the certs and cluster.bin into the deploy dir itself, taking <binary> <top-dir> <peers> -- so the precondition failed in every checkout, and the 4-argument call would have shifted every argument. (2) The pre-kill commit wait's boundary was 1 + 2*N_JOBS because bug-845 once wrote each JOB_SUBMIT twice; since bug-845's fix each job is one WAL entry, so a healthy cluster commits exactly 1 + N_JOBS and the wait timed out. (3) The walcompare --committed boundary repeated the same formula, so it asked for an LSN beyond every member's WAL and reported every member MISSING it.
Fix
src/controller/testsuite/clean_failover_preservation.sh: dropped the DATADIR variable and its cluster.bin / TLS-cert preconditions; calls deploy-ha.sh "$BINARY" "$TOPDIR" "$PEERS"; both COMMITTED_BOUNDARY defaults are now 1 + N_JOBS (own-term noop + one entry per job), with the bug-845 comments corrected. XCE_FAILOVER_COMMITTED still overrides it.
Verification
- Test: the script itself is the test -- before the fix it failed its precondition (
cluster.bin missing under .../ha/data); after, it passes end to end (deploy, submit, commit wait, kill primary, promotion, rejoin, walcompare, job survival) - Suites: n/a -- script-only change;
bash -nclean - Live: 3/3 passing runs, 2026-09-29: N_JOBS=2 twice (walcompare --committed 3) and N_JOBS=5 (--committed 6), each
PASS: clean failover converged; committed prefix identical; no job lost