docs(ha-config-bootstrap): add manual integration test guide with correct worktree paths
This commit is contained in:
parent
6b821eb4d5
commit
e88c7bebe9
1 changed files with 293 additions and 0 deletions
293
.kiro/specs/ha-config-bootstrap/manual-tests.md
Normal file
293
.kiro/specs/ha-config-bootstrap/manual-tests.md
Normal file
|
|
@ -0,0 +1,293 @@
|
|||
# Manual Integration Tests: ha-config-bootstrap
|
||||
|
||||
This document describes how to run the integration tests for tasks 11.1–11.5.
|
||||
Run each test, record the output, and paste the filled **Results Template** at
|
||||
the bottom back to the developer.
|
||||
|
||||
## Environment
|
||||
|
||||
```
|
||||
Worktree: /home/rshen/projects/xce/worktrees/persistent_ha
|
||||
Build dir: /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
|
||||
Controller: src/xce.build/controller/xce_controller
|
||||
CLI: src/xce.build/frontend/xcecli/xce (or /tmp/xce_static after make build-cli)
|
||||
Unit tests: src/xce.build/controller/controller_unit_test
|
||||
src/xce.build/lib/xcelib_test
|
||||
```
|
||||
|
||||
## Build First
|
||||
|
||||
```bash
|
||||
# From the worktree
|
||||
/xce-build # full dev build (preferred: uses nix develop + meson)
|
||||
|
||||
# Or manually from the src directory:
|
||||
nix develop /home/rshen/projects/xce/xce/src --command bash -c \
|
||||
"cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja"
|
||||
```
|
||||
|
||||
## Unit Tests (Sanity Check Before Integration)
|
||||
|
||||
```bash
|
||||
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
|
||||
|
||||
./controller/controller_unit_test 2>/dev/null | tail -5
|
||||
./lib/xcelib_test 2>/dev/null | tail -3
|
||||
```
|
||||
|
||||
Expected: all tests PASS, no FAIL lines, exit 0.
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
All tests use temporary work directories. Each controller node needs:
|
||||
- A unique `--myname` (e.g., `ctrl-a`, `ctrl-b`)
|
||||
- Shared `--peers` list (comma-separated `name=host:port`, priority order 0 = highest)
|
||||
- A `--workdir` for persistent state (`ha-term.bin`, `data/wal/`, `data/snap/`)
|
||||
- A `--logfile` so you can capture output
|
||||
|
||||
```bash
|
||||
# One-time setup
|
||||
mkdir -p /tmp/xce-a /tmp/xce-b /tmp/xce-a/data /tmp/xce-b/data
|
||||
BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller
|
||||
PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902"
|
||||
```
|
||||
|
||||
**Start Node A (Terminal 1):**
|
||||
```bash
|
||||
$BINARY \
|
||||
--myname ctrl-a \
|
||||
--peers "$PEERS" \
|
||||
--workdir /tmp/xce-a \
|
||||
--logfile /tmp/xce-a/ctrl.log \
|
||||
--vcpus 2
|
||||
```
|
||||
|
||||
**Start Node B (Terminal 2):**
|
||||
```bash
|
||||
$BINARY \
|
||||
--myname ctrl-b \
|
||||
--peers "$PEERS" \
|
||||
--workdir /tmp/xce-b \
|
||||
--logfile /tmp/xce-b/ctrl.log \
|
||||
--vcpus 2
|
||||
```
|
||||
|
||||
Watch logs in another terminal:
|
||||
```bash
|
||||
tail -f /tmp/xce-a/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term"
|
||||
tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 11.1 — Concurrent Startup and Config-Fresh Election
|
||||
|
||||
**Requirements:** 1.1, 2.1, 2.7
|
||||
|
||||
**Scenario:** Two nodes start within 3 seconds — only one becomes primary.
|
||||
|
||||
1. Start Node A and Node B as shown above within 3 seconds of each other.
|
||||
2. Wait 10 seconds.
|
||||
3. Check the logs.
|
||||
|
||||
**Expected log patterns:**
|
||||
- Exactly **one** node logs: `am_i_primary_quorum: won` or `CTRL_PRIMARY`
|
||||
- Other node logs: `CTRL_STANDBY`
|
||||
- No node stays in `CTRL_ELECTING` loop indefinitely
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Exactly one node becomes `CTRL_PRIMARY`
|
||||
- [ ] Other node becomes `CTRL_STANDBY`
|
||||
- [ ] No crash or ASAN error
|
||||
- [ ] Both nodes started within 3s of each other (no false self-election)
|
||||
|
||||
---
|
||||
|
||||
## Test 11.2 — Failover and WAL Durability
|
||||
|
||||
**Requirements:** 5.6, 6.2, 5.5
|
||||
|
||||
**Prerequisites:** 2-node cluster running from Test 11.1.
|
||||
|
||||
### Scenario 1: Primary crash → RDHUP → standby promotes
|
||||
|
||||
1. Identify which node is primary from the logs.
|
||||
2. Kill the primary process: `kill -9 <pid>` or Ctrl-C.
|
||||
3. Watch the standby log.
|
||||
|
||||
**Expected within 1 second:**
|
||||
```
|
||||
HA: primary_ctrl_fd RDHUP → CTRL_ELECTING
|
||||
am_i_primary_quorum: won
|
||||
CTRL_PRIMARY
|
||||
```
|
||||
|
||||
### Scenario 2: WAL commit with no standbys → ETIMEDOUT
|
||||
|
||||
1. With only the surviving primary running, submit a job:
|
||||
```bash
|
||||
# Use xcecli to submit anything that triggers a WAL write
|
||||
/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \
|
||||
--server 127.0.0.1:9901 jobsubmit -c "hostname"
|
||||
```
|
||||
|
||||
**Expected in primary log:** `repl_wait_majority_ack: timeout` — NOT a crash.
|
||||
|
||||
### Scenario 3: Reconnected standby catches up
|
||||
|
||||
1. Restart the killed node.
|
||||
2. Watch logs on both nodes.
|
||||
|
||||
**Expected:**
|
||||
- Standby log: `CtrlHelloReply from primary: committed_lsn=...`
|
||||
- Standby enters `CTRL_STANDBY`
|
||||
|
||||
**Checklist:**
|
||||
- [ ] RDHUP failover detected within 1s of primary kill
|
||||
- [ ] Standby promoted to CTRL_PRIMARY
|
||||
- [ ] WAL commit with no standbys returns ETIMEDOUT (graceful degraded mode)
|
||||
- [ ] Restarted standby reconnects and enters CTRL_STANDBY
|
||||
|
||||
---
|
||||
|
||||
## Test 11.3 — Unconfigured Node Bootstrap
|
||||
|
||||
**Requirements:** 3.2, 3.5
|
||||
|
||||
**Scenario:** Node B starts with no local cluster config (config_tag=0).
|
||||
|
||||
1. Remove any existing cluster.bin from node B's workdir:
|
||||
```bash
|
||||
rm -f /tmp/xce-b/cluster.bin /tmp/xce-b/data/current 2>/dev/null
|
||||
```
|
||||
2. Start Node A first (let it become primary with a valid config).
|
||||
3. Start Node B without a config file.
|
||||
|
||||
**Expected in Node B log:**
|
||||
```
|
||||
defer_config_fetch=1
|
||||
Config fetch successful from primary (tag=...)
|
||||
CTRL_STANDBY
|
||||
```
|
||||
|
||||
**Expected in `/tmp/xce-b/`:** a `cluster.bin` or `current` pointer file appears.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Node B detects config_tag=0 and sets defer_config_fetch
|
||||
- [ ] Node B yields election to Node A (fresher config)
|
||||
- [ ] Node B successfully fetches config from primary
|
||||
- [ ] Node B enters CTRL_STANDBY with correct config
|
||||
|
||||
---
|
||||
|
||||
## Test 11.4 — Term Durability Across Restart
|
||||
|
||||
**Requirements:** 9.1, 9.2
|
||||
|
||||
**Scenario:** After election, `ha-term.bin` is written; on restart, term is restored.
|
||||
|
||||
1. Run 2-node cluster from Test 11.1, wait for election to complete.
|
||||
2. Check that `ha-term.bin` was written:
|
||||
```bash
|
||||
xxd /tmp/xce-a/ha-term.bin | head -4
|
||||
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0
|
||||
```
|
||||
3. Kill Node A.
|
||||
4. Restart Node A.
|
||||
5. Check startup log for term restore.
|
||||
|
||||
**Expected in startup log:**
|
||||
```
|
||||
HA: restored term=N from /tmp/xce-a/ha-term.bin
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] `ha-term.bin` file exists after election
|
||||
- [ ] Magic bytes are `0xDEADBEEFCAFEBABE` (little-endian in `xxd`)
|
||||
- [ ] Term field (bytes 8–15) is > 0 after a vote was granted
|
||||
- [ ] On restart: log shows `restored term=N`
|
||||
- [ ] Node re-joins correctly without double-voting
|
||||
|
||||
---
|
||||
|
||||
## Test 11.5 — Full ASAN Build Verification
|
||||
|
||||
**Requirements:** 4
|
||||
|
||||
**Steps:**
|
||||
|
||||
```bash
|
||||
# Run unit tests (already built with ASAN via /xce-build)
|
||||
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
|
||||
|
||||
./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
|
||||
./lib/xcelib_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
|
||||
```
|
||||
|
||||
**Expected output:** No lines (zero ASAN errors introduced by this feature).
|
||||
|
||||
Pre-existing leaks from `test_worker_join` in controller_unit_test are acceptable —
|
||||
they existed before this branch and are not new.
|
||||
|
||||
```bash
|
||||
# Count pre-existing vs new leaks:
|
||||
./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] `controller_unit_test` exits 0, all tests PASS
|
||||
- [ ] `xcelib_test` exits 0, all tests PASS
|
||||
- [ ] Zero `AddressSanitizer` errors (use-after-free, buffer overflow)
|
||||
- [ ] No new `LeakSanitizer` leaks from ha-config-bootstrap code paths
|
||||
|
||||
---
|
||||
|
||||
## Results Template
|
||||
|
||||
Copy, fill in, and paste back:
|
||||
|
||||
```
|
||||
## Test Run Results
|
||||
|
||||
Date:
|
||||
Commit: (run: cd /home/rshen/projects/xce/worktrees/persistent_ha && git log -1 --oneline)
|
||||
Build: /xce-build completed successfully? YES / NO
|
||||
|
||||
### Unit Tests
|
||||
- controller_unit_test: PASS / FAIL
|
||||
- xcelib_test: PASS / FAIL
|
||||
|
||||
### 11.1 Concurrent Startup
|
||||
- Result: PASS / FAIL
|
||||
- Primary elected: ctrl-a / ctrl-b
|
||||
- Time to election: ~Xs
|
||||
- Log excerpt (primary and standby):
|
||||
|
||||
### 11.2 Failover and WAL Durability
|
||||
- Scenario 1 (RDHUP failover): PASS / FAIL
|
||||
- Time to detect RDHUP: ~Xs
|
||||
- Log excerpt:
|
||||
- Scenario 2 (degraded WAL timeout): PASS / FAIL
|
||||
- Log excerpt:
|
||||
- Scenario 3 (reconnect): PASS / FAIL
|
||||
|
||||
### 11.3 Unconfigured Node Bootstrap
|
||||
- Result: PASS / FAIL
|
||||
- cluster.bin created in /tmp/xce-b/? YES / NO
|
||||
- Log excerpt:
|
||||
|
||||
### 11.4 Term Durability
|
||||
- ha-term.bin exists: YES / NO
|
||||
- xxd output (first 4 lines):
|
||||
- Restart log (term restore line):
|
||||
- Result: PASS / FAIL
|
||||
|
||||
### 11.5 ASAN Build
|
||||
- controller_unit_test ASAN errors: N
|
||||
- xcelib_test ASAN errors: N
|
||||
- Result: PASS / FAIL
|
||||
|
||||
### Additional Notes / Unexpected Behaviour
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue