diff --git a/.kiro/specs/ha-config-bootstrap/manual-tests.md b/.kiro/specs/ha-config-bootstrap/manual-tests.md new file mode 100644 index 00000000..690d1aa8 --- /dev/null +++ b/.kiro/specs/ha-config-bootstrap/manual-tests.md @@ -0,0 +1,293 @@ +# Manual Integration Tests: ha-config-bootstrap + +This document describes how to run the integration tests for tasks 11.1–11.5. +Run each test, record the output, and paste the filled **Results Template** at +the bottom back to the developer. + +## Environment + +``` +Worktree: /home/rshen/projects/xce/worktrees/persistent_ha +Build dir: /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build +Controller: src/xce.build/controller/xce_controller +CLI: src/xce.build/frontend/xcecli/xce (or /tmp/xce_static after make build-cli) +Unit tests: src/xce.build/controller/controller_unit_test + src/xce.build/lib/xcelib_test +``` + +## Build First + +```bash +# From the worktree +/xce-build # full dev build (preferred: uses nix develop + meson) + +# Or manually from the src directory: +nix develop /home/rshen/projects/xce/xce/src --command bash -c \ + "cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja" +``` + +## Unit Tests (Sanity Check Before Integration) + +```bash +cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build + +./controller/controller_unit_test 2>/dev/null | tail -5 +./lib/xcelib_test 2>/dev/null | tail -3 +``` + +Expected: all tests PASS, no FAIL lines, exit 0. + +--- + +## Test Infrastructure + +All tests use temporary work directories. Each controller node needs: +- A unique `--myname` (e.g., `ctrl-a`, `ctrl-b`) +- Shared `--peers` list (comma-separated `name=host:port`, priority order 0 = highest) +- A `--workdir` for persistent state (`ha-term.bin`, `data/wal/`, `data/snap/`) +- A `--logfile` so you can capture output + +```bash +# One-time setup +mkdir -p /tmp/xce-a /tmp/xce-b /tmp/xce-a/data /tmp/xce-b/data +BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller +PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902" +``` + +**Start Node A (Terminal 1):** +```bash +$BINARY \ + --myname ctrl-a \ + --peers "$PEERS" \ + --workdir /tmp/xce-a \ + --logfile /tmp/xce-a/ctrl.log \ + --vcpus 2 +``` + +**Start Node B (Terminal 2):** +```bash +$BINARY \ + --myname ctrl-b \ + --peers "$PEERS" \ + --workdir /tmp/xce-b \ + --logfile /tmp/xce-b/ctrl.log \ + --vcpus 2 +``` + +Watch logs in another terminal: +```bash +tail -f /tmp/xce-a/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term" +tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term" +``` + +--- + +## Test 11.1 — Concurrent Startup and Config-Fresh Election + +**Requirements:** 1.1, 2.1, 2.7 + +**Scenario:** Two nodes start within 3 seconds — only one becomes primary. + +1. Start Node A and Node B as shown above within 3 seconds of each other. +2. Wait 10 seconds. +3. Check the logs. + +**Expected log patterns:** +- Exactly **one** node logs: `am_i_primary_quorum: won` or `CTRL_PRIMARY` +- Other node logs: `CTRL_STANDBY` +- No node stays in `CTRL_ELECTING` loop indefinitely + +**Checklist:** +- [ ] Exactly one node becomes `CTRL_PRIMARY` +- [ ] Other node becomes `CTRL_STANDBY` +- [ ] No crash or ASAN error +- [ ] Both nodes started within 3s of each other (no false self-election) + +--- + +## Test 11.2 — Failover and WAL Durability + +**Requirements:** 5.6, 6.2, 5.5 + +**Prerequisites:** 2-node cluster running from Test 11.1. + +### Scenario 1: Primary crash → RDHUP → standby promotes + +1. Identify which node is primary from the logs. +2. Kill the primary process: `kill -9 ` or Ctrl-C. +3. Watch the standby log. + +**Expected within 1 second:** +``` +HA: primary_ctrl_fd RDHUP → CTRL_ELECTING +am_i_primary_quorum: won +CTRL_PRIMARY +``` + +### Scenario 2: WAL commit with no standbys → ETIMEDOUT + +1. With only the surviving primary running, submit a job: +```bash +# Use xcecli to submit anything that triggers a WAL write +/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \ + --server 127.0.0.1:9901 jobsubmit -c "hostname" +``` + +**Expected in primary log:** `repl_wait_majority_ack: timeout` — NOT a crash. + +### Scenario 3: Reconnected standby catches up + +1. Restart the killed node. +2. Watch logs on both nodes. + +**Expected:** +- Standby log: `CtrlHelloReply from primary: committed_lsn=...` +- Standby enters `CTRL_STANDBY` + +**Checklist:** +- [ ] RDHUP failover detected within 1s of primary kill +- [ ] Standby promoted to CTRL_PRIMARY +- [ ] WAL commit with no standbys returns ETIMEDOUT (graceful degraded mode) +- [ ] Restarted standby reconnects and enters CTRL_STANDBY + +--- + +## Test 11.3 — Unconfigured Node Bootstrap + +**Requirements:** 3.2, 3.5 + +**Scenario:** Node B starts with no local cluster config (config_tag=0). + +1. Remove any existing cluster.bin from node B's workdir: +```bash +rm -f /tmp/xce-b/cluster.bin /tmp/xce-b/data/current 2>/dev/null +``` +2. Start Node A first (let it become primary with a valid config). +3. Start Node B without a config file. + +**Expected in Node B log:** +``` +defer_config_fetch=1 +Config fetch successful from primary (tag=...) +CTRL_STANDBY +``` + +**Expected in `/tmp/xce-b/`:** a `cluster.bin` or `current` pointer file appears. + +**Checklist:** +- [ ] Node B detects config_tag=0 and sets defer_config_fetch +- [ ] Node B yields election to Node A (fresher config) +- [ ] Node B successfully fetches config from primary +- [ ] Node B enters CTRL_STANDBY with correct config + +--- + +## Test 11.4 — Term Durability Across Restart + +**Requirements:** 9.1, 9.2 + +**Scenario:** After election, `ha-term.bin` is written; on restart, term is restored. + +1. Run 2-node cluster from Test 11.1, wait for election to complete. +2. Check that `ha-term.bin` was written: +```bash +xxd /tmp/xce-a/ha-term.bin | head -4 +# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0 +``` +3. Kill Node A. +4. Restart Node A. +5. Check startup log for term restore. + +**Expected in startup log:** +``` +HA: restored term=N from /tmp/xce-a/ha-term.bin +``` + +**Checklist:** +- [ ] `ha-term.bin` file exists after election +- [ ] Magic bytes are `0xDEADBEEFCAFEBABE` (little-endian in `xxd`) +- [ ] Term field (bytes 8–15) is > 0 after a vote was granted +- [ ] On restart: log shows `restored term=N` +- [ ] Node re-joins correctly without double-voting + +--- + +## Test 11.5 — Full ASAN Build Verification + +**Requirements:** 4 + +**Steps:** + +```bash +# Run unit tests (already built with ASAN via /xce-build) +cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build + +./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)' +./lib/xcelib_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)' +``` + +**Expected output:** No lines (zero ASAN errors introduced by this feature). + +Pre-existing leaks from `test_worker_join` in controller_unit_test are acceptable — +they existed before this branch and are not new. + +```bash +# Count pre-existing vs new leaks: +./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l +``` + +**Checklist:** +- [ ] `controller_unit_test` exits 0, all tests PASS +- [ ] `xcelib_test` exits 0, all tests PASS +- [ ] Zero `AddressSanitizer` errors (use-after-free, buffer overflow) +- [ ] No new `LeakSanitizer` leaks from ha-config-bootstrap code paths + +--- + +## Results Template + +Copy, fill in, and paste back: + +``` +## Test Run Results + +Date: +Commit: (run: cd /home/rshen/projects/xce/worktrees/persistent_ha && git log -1 --oneline) +Build: /xce-build completed successfully? YES / NO + +### Unit Tests +- controller_unit_test: PASS / FAIL +- xcelib_test: PASS / FAIL + +### 11.1 Concurrent Startup +- Result: PASS / FAIL +- Primary elected: ctrl-a / ctrl-b +- Time to election: ~Xs +- Log excerpt (primary and standby): + +### 11.2 Failover and WAL Durability +- Scenario 1 (RDHUP failover): PASS / FAIL + - Time to detect RDHUP: ~Xs + - Log excerpt: +- Scenario 2 (degraded WAL timeout): PASS / FAIL + - Log excerpt: +- Scenario 3 (reconnect): PASS / FAIL + +### 11.3 Unconfigured Node Bootstrap +- Result: PASS / FAIL +- cluster.bin created in /tmp/xce-b/? YES / NO +- Log excerpt: + +### 11.4 Term Durability +- ha-term.bin exists: YES / NO +- xxd output (first 4 lines): +- Restart log (term restore line): +- Result: PASS / FAIL + +### 11.5 ASAN Build +- controller_unit_test ASAN errors: N +- xcelib_test ASAN errors: N +- Result: PASS / FAIL + +### Additional Notes / Unexpected Behaviour +```