51 lines
4 KiB
Markdown
51 lines
4 KiB
Markdown
---
|
|
id: "bug-1721"
|
|
status: "fixed"
|
|
status_updated: "2026-09-30"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-08-26T00:00:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/controller/testsuite/clean_failover_preservation.sh"]
|
|
tags: ["test-harness", "stale-fixture", "clean_failover_preservation.sh", "ha-term-safety", "pre-existing", "admin-config-tool-regression"]
|
|
related: []
|
|
fix_commits: ["f5c887dc"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1721: clean_failover_preservation.sh FAIL: cluster.bin missing under <repo>/ha/data (deploy prerequisite)
|
|
|
|
## Description
|
|
clean_failover_preservation.sh FAIL: cluster.bin missing under <repo>/ha/data (deploy prerequisite)
|
|
|
|
UPDATE (2026-09-29): once past the ha/data precondition, two more stale assumptions surfaced in the same script: the deploy-ha.sh call used an old 4-argument signature, and both committed-LSN boundaries assumed bug-845's doubled submits (`FAIL: jobs did not commit on a majority within 90s (max committed_lsn=3 < 5)`, then `walcompare: FAIL -- committed-prefix divergence at LSN 5` with all three WALs identical at LSN 1-4).
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (nix dev shell; interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: bash 5.3, gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (controller + harness + xceadmin in one build dir, src/xce.build.bug1871)
|
|
- Code: reproduced 2026-09-29 at 980a0054; the script has been broken since ee8473bd (ha/data removal)
|
|
- Deployment: local 3-node HA controller cluster the script deploys itself, localhost:29991-29993
|
|
|
|
## Reproduction
|
|
1. `XCE_CONTROLLER_BIN=<builddir>/controller/xce_controller XCE_FAILOVER_TOP=<empty dir> nix develop ./src --command bash src/controller/testsuite/clean_failover_preservation.sh`
|
|
- Frequency: deterministic -- 1/1 runs failed at the precondition, 2026-09-29: `FAIL: cluster.bin missing under <repo>/ha/data (deploy prerequisite)`
|
|
- Evidence: /home/claude/tmp/xce-1721-green (commit-wait failure), /home/claude/tmp/xce-1721-green2 (walcompare past-the-tip failure, walcompare.out), /home/claude/tmp/xce-1721-green3 (passing run)
|
|
|
|
## Root cause
|
|
Three stale assumptions in clean_failover_preservation.sh, each hidden behind the previous one. (1) It required <repo>/ha/data/{cluster.bin,xce.crt,xce.key} and passed that dir to deploy-ha.sh, but ee8473bd removed ha/data/ and deploy-ha.sh now generates the certs and cluster.bin into the deploy dir itself, taking `<binary> <top-dir> <peers>` -- so the precondition failed in every checkout, and the 4-argument call would have shifted every argument. (2) The pre-kill commit wait's boundary was 1 + 2*N_JOBS because bug-845 once wrote each JOB_SUBMIT twice; since bug-845's fix each job is one WAL entry, so a healthy cluster commits exactly 1 + N_JOBS and the wait timed out. (3) The walcompare --committed boundary repeated the same formula, so it asked for an LSN beyond every member's WAL and reported every member MISSING it.
|
|
|
|
## Fix
|
|
src/controller/testsuite/clean_failover_preservation.sh: dropped the DATADIR variable and its cluster.bin / TLS-cert preconditions; calls `deploy-ha.sh "$BINARY" "$TOPDIR" "$PEERS"`; both COMMITTED_BOUNDARY defaults are now 1 + N_JOBS (own-term noop + one entry per job), with the bug-845 comments corrected. XCE_FAILOVER_COMMITTED still overrides it.
|
|
|
|
## Verification
|
|
- Test: the script itself is the test -- before the fix it failed its precondition (`cluster.bin missing under .../ha/data`); after, it passes end to end (deploy, submit, commit wait, kill primary, promotion, rejoin, walcompare, job survival)
|
|
- Suites: n/a -- script-only change; `bash -n` clean
|
|
- Live: 3/3 passing runs, 2026-09-29: N_JOBS=2 twice (walcompare --committed 3) and N_JOBS=5 (--committed 6), each `PASS: clean failover converged; committed prefix identical; no job lost`
|
|
|
|
## References
|
|
- Fix commit: f5c887dc
|
|
- Related: bug-845 (doubled submits, fixed), ee8473bd (ha/data removal)
|