diff --git a/.kiro/specs/ha-config-bootstrap/manual-tests.md b/.kiro/specs/ha-config-bootstrap/manual-tests.md index b38efad..0ce8c51 100644 --- a/.kiro/specs/ha-config-bootstrap/manual-tests.md +++ b/.kiro/specs/ha-config-bootstrap/manual-tests.md @@ -41,29 +41,33 @@ Expected: all tests PASS, no FAIL lines, exit 0. ## Test Infrastructure +> **Design constraint:** The controller rejects an even number of peer nodes +> (`--peers` list), because Raft quorum requires an odd cluster size (1, 3, 5…) +> to guarantee a safe majority. **All tests use a 3-node cluster.** + All tests use temporary work directories. Each controller node needs: -- A unique `--myname` (e.g., `ctrl-a`, `ctrl-b`) -- Shared `--peers` list (comma-separated `name=host:port`, priority order 0 = highest) +- A unique `--myname` (`ctrl-a`, `ctrl-b`, `ctrl-c`; index 0 = highest priority) +- Shared `--peers` list (`name=host:port,...`, **odd count only**) - A `--workdir` for persistent state (`ha-term.bin`, `data/wal/`, `data/snap/`) -- A `--logfile` so you can capture output -- **TLS certificates** at `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key` — the - controller reads them automatically from that fixed path (no CLI flag) +- A `--logfile` to capture output +- **TLS certificates** at `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key` + (loaded automatically from that fixed path — no CLI flag required) + +### One-time setup ```bash -# One-time setup -mkdir -p /tmp/xce-a/certs /tmp/xce-b/certs /tmp/xce-a/data /tmp/xce-b/data BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller -PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902" +PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902,ctrl-c=127.0.0.1:9903" + +mkdir -p /tmp/xce-{a,b,c}/certs /tmp/xce-{a,b,c}/data ``` ### TLS Certificate Setup (required) -The controller expects `{workdir}/certs/xce.crt` (certificate) and -`{workdir}/certs/xce.key` (private key). For local testing, generate a -self-signed cert shared by both nodes: +The controller loads `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key` +automatically. Generate one self-signed cert shared by all three nodes: ```bash -# Generate a self-signed cert (valid 10 years, localhost + 127.0.0.1 SANs) openssl req -x509 -newkey rsa:2048 -nodes \ -keyout /tmp/xce.key \ -out /tmp/xce.crt \ @@ -71,41 +75,38 @@ openssl req -x509 -newkey rsa:2048 -nodes \ -subj "/CN=xce-controller" \ -addext "subjectAltName=IP:127.0.0.1,DNS:localhost" -# Copy to both node work dirs -cp /tmp/xce.crt /tmp/xce-a/certs/xce.crt -cp /tmp/xce.key /tmp/xce-a/certs/xce.key -cp /tmp/xce.crt /tmp/xce-b/certs/xce.crt -cp /tmp/xce.key /tmp/xce-b/certs/xce.key +for node in a b c; do + cp /tmp/xce.crt /tmp/xce-${node}/certs/xce.crt + cp /tmp/xce.key /tmp/xce-${node}/certs/xce.key +done ``` -> **Note:** Both nodes must trust each other's certificate. Using the same -> self-signed cert for all nodes (same CA = same cert) is the simplest setup for -> local testing. In production each node has its own cert signed by a shared CA. +> All nodes share the same self-signed cert so they mutually trust each other. +> In production each node has its own cert signed by a shared CA. -**Start Node A (Terminal 1):** +### Start the 3-node cluster (3 terminals) + +**Terminal 1 — Node A (priority 0 = highest):** ```bash -$BINARY \ - --myname ctrl-a \ - --peers "$PEERS" \ - --workdir /tmp/xce-a \ - --logfile /tmp/xce-a/ctrl.log \ - --vcpus 2 +$BINARY --myname ctrl-a --peers "$PEERS" \ + --workdir /tmp/xce-a --logfile /tmp/xce-a/ctrl.log --vcpus 2 ``` -**Start Node B (Terminal 2):** +**Terminal 2 — Node B:** ```bash -$BINARY \ - --myname ctrl-b \ - --peers "$PEERS" \ - --workdir /tmp/xce-b \ - --logfile /tmp/xce-b/ctrl.log \ - --vcpus 2 +$BINARY --myname ctrl-b --peers "$PEERS" \ + --workdir /tmp/xce-b --logfile /tmp/xce-b/ctrl.log --vcpus 2 ``` -Watch logs in another terminal: +**Terminal 3 — Node C:** ```bash -tail -f /tmp/xce-a/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term" -tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term" +$BINARY --myname ctrl-c --peers "$PEERS" \ + --workdir /tmp/xce-c --logfile /tmp/xce-c/ctrl.log --vcpus 2 +``` + +**Watch logs (Terminal 4):** +```bash +tail -f /tmp/xce-{a,b,c}/ctrl.log | grep -E "CTRL_|election|quorum|RDHUP|ha-term|heartbeat" ``` --- @@ -114,22 +115,22 @@ tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha- **Requirements:** 1.1, 2.1, 2.7 -**Scenario:** Two nodes start within 3 seconds — only one becomes primary. +**Scenario:** Three nodes start within 3 seconds — exactly one wins election (needs majority = 2/3 votes). -1. Start Node A and Node B as shown above within 3 seconds of each other. +1. Start all 3 nodes (Terminals 1–3) within 3 seconds of each other. 2. Wait 10 seconds. -3. Check the logs. +3. Check the logs on all three. **Expected log patterns:** - Exactly **one** node logs: `am_i_primary_quorum: won` or `CTRL_PRIMARY` -- Other node logs: `CTRL_STANDBY` +- Other **two** nodes log: `CTRL_STANDBY` - No node stays in `CTRL_ELECTING` loop indefinitely **Checklist:** - [ ] Exactly one node becomes `CTRL_PRIMARY` -- [ ] Other node becomes `CTRL_STANDBY` +- [ ] Two nodes become `CTRL_STANDBY` - [ ] No crash or ASAN error -- [ ] Both nodes started within 3s of each other (no false self-election) +- [ ] All three nodes started within 3s (no false self-election) --- @@ -137,26 +138,29 @@ tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha- **Requirements:** 5.6, 6.2, 5.5 -**Prerequisites:** 2-node cluster running from Test 11.1. +**Prerequisites:** 3-node cluster running from Test 11.1. -### Scenario 1: Primary crash → RDHUP → standby promotes +### Scenario 1: Primary crash → RDHUP → one standby promotes 1. Identify which node is primary from the logs. 2. Kill the primary process: `kill -9 ` or Ctrl-C. -3. Watch the standby log. +3. Watch the two surviving standbys. -**Expected within 1 second:** +**Expected within 1 second on both standbys:** ``` HA: primary_ctrl_fd RDHUP → CTRL_ELECTING +``` +**Expected on the winner (needs 1 vote from the other surviving node):** +``` am_i_primary_quorum: won CTRL_PRIMARY ``` -### Scenario 2: WAL commit with no standbys → ETIMEDOUT +### Scenario 2: WAL commit with both standbys down → ETIMEDOUT -1. With only the surviving primary running, submit a job: +1. Kill both standbys (leaving only the primary). +2. Submit a job: ```bash -# Use xcecli to submit anything that triggers a WAL write /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \ --server 127.0.0.1:9901 jobsubmit -c "hostname" ``` @@ -165,17 +169,19 @@ CTRL_PRIMARY ### Scenario 3: Reconnected standby catches up -1. Restart the killed node. -2. Watch logs on both nodes. +1. Restart one of the killed standbys. +2. Watch logs on the restarted node. **Expected:** -- Standby log: `CtrlHelloReply from primary: committed_lsn=...` -- Standby enters `CTRL_STANDBY` +``` +CtrlHelloReply from primary: committed_lsn=... +CTRL_STANDBY +``` **Checklist:** - [ ] RDHUP failover detected within 1s of primary kill -- [ ] Standby promoted to CTRL_PRIMARY -- [ ] WAL commit with no standbys returns ETIMEDOUT (graceful degraded mode) +- [ ] One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other) +- [ ] WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode) - [ ] Restarted standby reconnects and enters CTRL_STANDBY --- @@ -184,29 +190,29 @@ CTRL_PRIMARY **Requirements:** 3.2, 3.5 -**Scenario:** Node B starts with no local cluster config (config_tag=0). +**Scenario:** Node C starts with no local cluster config (config_tag=0); Nodes A and B have a valid config. -1. Remove any existing cluster.bin from node B's workdir: +1. Start Nodes A and B first (let one become primary). +2. Remove any existing config from Node C's workdir: ```bash -rm -f /tmp/xce-b/cluster.bin /tmp/xce-b/data/current 2>/dev/null +rm -f /tmp/xce-c/cluster.bin /tmp/xce-c/data/current 2>/dev/null ``` -2. Start Node A first (let it become primary with a valid config). -3. Start Node B without a config file. +3. Start Node C without a config file (just `--peers`, no config). -**Expected in Node B log:** +**Expected in Node C log:** ``` defer_config_fetch=1 Config fetch successful from primary (tag=...) CTRL_STANDBY ``` -**Expected in `/tmp/xce-b/`:** a `cluster.bin` or `current` pointer file appears. +**Expected in `/tmp/xce-c/`:** a `cluster.bin` or `current` pointer file appears. **Checklist:** -- [ ] Node B detects config_tag=0 and sets defer_config_fetch -- [ ] Node B yields election to Node A (fresher config) -- [ ] Node B successfully fetches config from primary -- [ ] Node B enters CTRL_STANDBY with correct config +- [ ] Node C detects config_tag=0 and sets defer_config_fetch +- [ ] Node C yields election to the existing primary (fresher config) +- [ ] Node C successfully fetches config from primary +- [ ] Node C enters CTRL_STANDBY with correct config --- @@ -216,8 +222,8 @@ CTRL_STANDBY **Scenario:** After election, `ha-term.bin` is written; on restart, term is restored. -1. Run 2-node cluster from Test 11.1, wait for election to complete. -2. Check that `ha-term.bin` was written: +1. Run 3-node cluster from Test 11.1, wait for election to complete. +2. Check that `ha-term.bin` was written on the node that granted a vote: ```bash xxd /tmp/xce-a/ha-term.bin | head -4 # Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0 @@ -287,23 +293,25 @@ Build: /xce-build completed successfully? YES / NO - controller_unit_test: PASS / FAIL - xcelib_test: PASS / FAIL -### 11.1 Concurrent Startup +### 11.1 Concurrent Startup (3 nodes) - Result: PASS / FAIL -- Primary elected: ctrl-a / ctrl-b +- Primary elected: ctrl-a / ctrl-b / ctrl-c - Time to election: ~Xs -- Log excerpt (primary and standby): +- Two standbys confirmed: YES / NO +- Log excerpt (primary win + both standbys): -### 11.2 Failover and WAL Durability -- Scenario 1 (RDHUP failover): PASS / FAIL +### 11.2 Failover and WAL Durability (3 nodes) +- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS / FAIL - Time to detect RDHUP: ~Xs + - New primary: ctrl-a / ctrl-b / ctrl-c - Log excerpt: -- Scenario 2 (degraded WAL timeout): PASS / FAIL +- Scenario 2 (degraded WAL timeout, both standbys down): PASS / FAIL - Log excerpt: -- Scenario 3 (reconnect): PASS / FAIL +- Scenario 3 (reconnect one standby): PASS / FAIL -### 11.3 Unconfigured Node Bootstrap +### 11.3 Unconfigured Node Bootstrap (ctrl-c joins ctrl-a+ctrl-b) - Result: PASS / FAIL -- cluster.bin created in /tmp/xce-b/? YES / NO +- cluster.bin created in /tmp/xce-c/? YES / NO - Log excerpt: ### 11.4 Term Durability