docs(ha-config-bootstrap): update manual tests to 3-node cluster (odd peer count design requirement)
This commit is contained in:
parent
abdb195c8e
commit
bb4e23663a
1 changed files with 86 additions and 78 deletions
|
|
@ -41,29 +41,33 @@ Expected: all tests PASS, no FAIL lines, exit 0.
|
|||
|
||||
## Test Infrastructure
|
||||
|
||||
> **Design constraint:** The controller rejects an even number of peer nodes
|
||||
> (`--peers` list), because Raft quorum requires an odd cluster size (1, 3, 5…)
|
||||
> to guarantee a safe majority. **All tests use a 3-node cluster.**
|
||||
|
||||
All tests use temporary work directories. Each controller node needs:
|
||||
- A unique `--myname` (e.g., `ctrl-a`, `ctrl-b`)
|
||||
- Shared `--peers` list (comma-separated `name=host:port`, priority order 0 = highest)
|
||||
- A unique `--myname` (`ctrl-a`, `ctrl-b`, `ctrl-c`; index 0 = highest priority)
|
||||
- Shared `--peers` list (`name=host:port,...`, **odd count only**)
|
||||
- A `--workdir` for persistent state (`ha-term.bin`, `data/wal/`, `data/snap/`)
|
||||
- A `--logfile` so you can capture output
|
||||
- **TLS certificates** at `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key` — the
|
||||
controller reads them automatically from that fixed path (no CLI flag)
|
||||
- A `--logfile` to capture output
|
||||
- **TLS certificates** at `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key`
|
||||
(loaded automatically from that fixed path — no CLI flag required)
|
||||
|
||||
### One-time setup
|
||||
|
||||
```bash
|
||||
# One-time setup
|
||||
mkdir -p /tmp/xce-a/certs /tmp/xce-b/certs /tmp/xce-a/data /tmp/xce-b/data
|
||||
BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller
|
||||
PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902"
|
||||
PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902,ctrl-c=127.0.0.1:9903"
|
||||
|
||||
mkdir -p /tmp/xce-{a,b,c}/certs /tmp/xce-{a,b,c}/data
|
||||
```
|
||||
|
||||
### TLS Certificate Setup (required)
|
||||
|
||||
The controller expects `{workdir}/certs/xce.crt` (certificate) and
|
||||
`{workdir}/certs/xce.key` (private key). For local testing, generate a
|
||||
self-signed cert shared by both nodes:
|
||||
The controller loads `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key`
|
||||
automatically. Generate one self-signed cert shared by all three nodes:
|
||||
|
||||
```bash
|
||||
# Generate a self-signed cert (valid 10 years, localhost + 127.0.0.1 SANs)
|
||||
openssl req -x509 -newkey rsa:2048 -nodes \
|
||||
-keyout /tmp/xce.key \
|
||||
-out /tmp/xce.crt \
|
||||
|
|
@ -71,41 +75,38 @@ openssl req -x509 -newkey rsa:2048 -nodes \
|
|||
-subj "/CN=xce-controller" \
|
||||
-addext "subjectAltName=IP:127.0.0.1,DNS:localhost"
|
||||
|
||||
# Copy to both node work dirs
|
||||
cp /tmp/xce.crt /tmp/xce-a/certs/xce.crt
|
||||
cp /tmp/xce.key /tmp/xce-a/certs/xce.key
|
||||
cp /tmp/xce.crt /tmp/xce-b/certs/xce.crt
|
||||
cp /tmp/xce.key /tmp/xce-b/certs/xce.key
|
||||
for node in a b c; do
|
||||
cp /tmp/xce.crt /tmp/xce-${node}/certs/xce.crt
|
||||
cp /tmp/xce.key /tmp/xce-${node}/certs/xce.key
|
||||
done
|
||||
```
|
||||
|
||||
> **Note:** Both nodes must trust each other's certificate. Using the same
|
||||
> self-signed cert for all nodes (same CA = same cert) is the simplest setup for
|
||||
> local testing. In production each node has its own cert signed by a shared CA.
|
||||
> All nodes share the same self-signed cert so they mutually trust each other.
|
||||
> In production each node has its own cert signed by a shared CA.
|
||||
|
||||
**Start Node A (Terminal 1):**
|
||||
### Start the 3-node cluster (3 terminals)
|
||||
|
||||
**Terminal 1 — Node A (priority 0 = highest):**
|
||||
```bash
|
||||
$BINARY \
|
||||
--myname ctrl-a \
|
||||
--peers "$PEERS" \
|
||||
--workdir /tmp/xce-a \
|
||||
--logfile /tmp/xce-a/ctrl.log \
|
||||
--vcpus 2
|
||||
$BINARY --myname ctrl-a --peers "$PEERS" \
|
||||
--workdir /tmp/xce-a --logfile /tmp/xce-a/ctrl.log --vcpus 2
|
||||
```
|
||||
|
||||
**Start Node B (Terminal 2):**
|
||||
**Terminal 2 — Node B:**
|
||||
```bash
|
||||
$BINARY \
|
||||
--myname ctrl-b \
|
||||
--peers "$PEERS" \
|
||||
--workdir /tmp/xce-b \
|
||||
--logfile /tmp/xce-b/ctrl.log \
|
||||
--vcpus 2
|
||||
$BINARY --myname ctrl-b --peers "$PEERS" \
|
||||
--workdir /tmp/xce-b --logfile /tmp/xce-b/ctrl.log --vcpus 2
|
||||
```
|
||||
|
||||
Watch logs in another terminal:
|
||||
**Terminal 3 — Node C:**
|
||||
```bash
|
||||
tail -f /tmp/xce-a/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term"
|
||||
tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-term"
|
||||
$BINARY --myname ctrl-c --peers "$PEERS" \
|
||||
--workdir /tmp/xce-c --logfile /tmp/xce-c/ctrl.log --vcpus 2
|
||||
```
|
||||
|
||||
**Watch logs (Terminal 4):**
|
||||
```bash
|
||||
tail -f /tmp/xce-{a,b,c}/ctrl.log | grep -E "CTRL_|election|quorum|RDHUP|ha-term|heartbeat"
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -114,22 +115,22 @@ tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-
|
|||
|
||||
**Requirements:** 1.1, 2.1, 2.7
|
||||
|
||||
**Scenario:** Two nodes start within 3 seconds — only one becomes primary.
|
||||
**Scenario:** Three nodes start within 3 seconds — exactly one wins election (needs majority = 2/3 votes).
|
||||
|
||||
1. Start Node A and Node B as shown above within 3 seconds of each other.
|
||||
1. Start all 3 nodes (Terminals 1–3) within 3 seconds of each other.
|
||||
2. Wait 10 seconds.
|
||||
3. Check the logs.
|
||||
3. Check the logs on all three.
|
||||
|
||||
**Expected log patterns:**
|
||||
- Exactly **one** node logs: `am_i_primary_quorum: won` or `CTRL_PRIMARY`
|
||||
- Other node logs: `CTRL_STANDBY`
|
||||
- Other **two** nodes log: `CTRL_STANDBY`
|
||||
- No node stays in `CTRL_ELECTING` loop indefinitely
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Exactly one node becomes `CTRL_PRIMARY`
|
||||
- [ ] Other node becomes `CTRL_STANDBY`
|
||||
- [ ] Two nodes become `CTRL_STANDBY`
|
||||
- [ ] No crash or ASAN error
|
||||
- [ ] Both nodes started within 3s of each other (no false self-election)
|
||||
- [ ] All three nodes started within 3s (no false self-election)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -137,26 +138,29 @@ tail -f /tmp/xce-b/ctrl.log | grep -E "CTRL_|election|heartbeat|quorum|RDHUP|ha-
|
|||
|
||||
**Requirements:** 5.6, 6.2, 5.5
|
||||
|
||||
**Prerequisites:** 2-node cluster running from Test 11.1.
|
||||
**Prerequisites:** 3-node cluster running from Test 11.1.
|
||||
|
||||
### Scenario 1: Primary crash → RDHUP → standby promotes
|
||||
### Scenario 1: Primary crash → RDHUP → one standby promotes
|
||||
|
||||
1. Identify which node is primary from the logs.
|
||||
2. Kill the primary process: `kill -9 <pid>` or Ctrl-C.
|
||||
3. Watch the standby log.
|
||||
3. Watch the two surviving standbys.
|
||||
|
||||
**Expected within 1 second:**
|
||||
**Expected within 1 second on both standbys:**
|
||||
```
|
||||
HA: primary_ctrl_fd RDHUP → CTRL_ELECTING
|
||||
```
|
||||
**Expected on the winner (needs 1 vote from the other surviving node):**
|
||||
```
|
||||
am_i_primary_quorum: won
|
||||
CTRL_PRIMARY
|
||||
```
|
||||
|
||||
### Scenario 2: WAL commit with no standbys → ETIMEDOUT
|
||||
### Scenario 2: WAL commit with both standbys down → ETIMEDOUT
|
||||
|
||||
1. With only the surviving primary running, submit a job:
|
||||
1. Kill both standbys (leaving only the primary).
|
||||
2. Submit a job:
|
||||
```bash
|
||||
# Use xcecli to submit anything that triggers a WAL write
|
||||
/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \
|
||||
--server 127.0.0.1:9901 jobsubmit -c "hostname"
|
||||
```
|
||||
|
|
@ -165,17 +169,19 @@ CTRL_PRIMARY
|
|||
|
||||
### Scenario 3: Reconnected standby catches up
|
||||
|
||||
1. Restart the killed node.
|
||||
2. Watch logs on both nodes.
|
||||
1. Restart one of the killed standbys.
|
||||
2. Watch logs on the restarted node.
|
||||
|
||||
**Expected:**
|
||||
- Standby log: `CtrlHelloReply from primary: committed_lsn=...`
|
||||
- Standby enters `CTRL_STANDBY`
|
||||
```
|
||||
CtrlHelloReply from primary: committed_lsn=...
|
||||
CTRL_STANDBY
|
||||
```
|
||||
|
||||
**Checklist:**
|
||||
- [ ] RDHUP failover detected within 1s of primary kill
|
||||
- [ ] Standby promoted to CTRL_PRIMARY
|
||||
- [ ] WAL commit with no standbys returns ETIMEDOUT (graceful degraded mode)
|
||||
- [ ] One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other)
|
||||
- [ ] WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode)
|
||||
- [ ] Restarted standby reconnects and enters CTRL_STANDBY
|
||||
|
||||
---
|
||||
|
|
@ -184,29 +190,29 @@ CTRL_PRIMARY
|
|||
|
||||
**Requirements:** 3.2, 3.5
|
||||
|
||||
**Scenario:** Node B starts with no local cluster config (config_tag=0).
|
||||
**Scenario:** Node C starts with no local cluster config (config_tag=0); Nodes A and B have a valid config.
|
||||
|
||||
1. Remove any existing cluster.bin from node B's workdir:
|
||||
1. Start Nodes A and B first (let one become primary).
|
||||
2. Remove any existing config from Node C's workdir:
|
||||
```bash
|
||||
rm -f /tmp/xce-b/cluster.bin /tmp/xce-b/data/current 2>/dev/null
|
||||
rm -f /tmp/xce-c/cluster.bin /tmp/xce-c/data/current 2>/dev/null
|
||||
```
|
||||
2. Start Node A first (let it become primary with a valid config).
|
||||
3. Start Node B without a config file.
|
||||
3. Start Node C without a config file (just `--peers`, no config).
|
||||
|
||||
**Expected in Node B log:**
|
||||
**Expected in Node C log:**
|
||||
```
|
||||
defer_config_fetch=1
|
||||
Config fetch successful from primary (tag=...)
|
||||
CTRL_STANDBY
|
||||
```
|
||||
|
||||
**Expected in `/tmp/xce-b/`:** a `cluster.bin` or `current` pointer file appears.
|
||||
**Expected in `/tmp/xce-c/`:** a `cluster.bin` or `current` pointer file appears.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] Node B detects config_tag=0 and sets defer_config_fetch
|
||||
- [ ] Node B yields election to Node A (fresher config)
|
||||
- [ ] Node B successfully fetches config from primary
|
||||
- [ ] Node B enters CTRL_STANDBY with correct config
|
||||
- [ ] Node C detects config_tag=0 and sets defer_config_fetch
|
||||
- [ ] Node C yields election to the existing primary (fresher config)
|
||||
- [ ] Node C successfully fetches config from primary
|
||||
- [ ] Node C enters CTRL_STANDBY with correct config
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -216,8 +222,8 @@ CTRL_STANDBY
|
|||
|
||||
**Scenario:** After election, `ha-term.bin` is written; on restart, term is restored.
|
||||
|
||||
1. Run 2-node cluster from Test 11.1, wait for election to complete.
|
||||
2. Check that `ha-term.bin` was written:
|
||||
1. Run 3-node cluster from Test 11.1, wait for election to complete.
|
||||
2. Check that `ha-term.bin` was written on the node that granted a vote:
|
||||
```bash
|
||||
xxd /tmp/xce-a/ha-term.bin | head -4
|
||||
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0
|
||||
|
|
@ -287,23 +293,25 @@ Build: /xce-build completed successfully? YES / NO
|
|||
- controller_unit_test: PASS / FAIL
|
||||
- xcelib_test: PASS / FAIL
|
||||
|
||||
### 11.1 Concurrent Startup
|
||||
### 11.1 Concurrent Startup (3 nodes)
|
||||
- Result: PASS / FAIL
|
||||
- Primary elected: ctrl-a / ctrl-b
|
||||
- Primary elected: ctrl-a / ctrl-b / ctrl-c
|
||||
- Time to election: ~Xs
|
||||
- Log excerpt (primary and standby):
|
||||
- Two standbys confirmed: YES / NO
|
||||
- Log excerpt (primary win + both standbys):
|
||||
|
||||
### 11.2 Failover and WAL Durability
|
||||
- Scenario 1 (RDHUP failover): PASS / FAIL
|
||||
### 11.2 Failover and WAL Durability (3 nodes)
|
||||
- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS / FAIL
|
||||
- Time to detect RDHUP: ~Xs
|
||||
- New primary: ctrl-a / ctrl-b / ctrl-c
|
||||
- Log excerpt:
|
||||
- Scenario 2 (degraded WAL timeout): PASS / FAIL
|
||||
- Scenario 2 (degraded WAL timeout, both standbys down): PASS / FAIL
|
||||
- Log excerpt:
|
||||
- Scenario 3 (reconnect): PASS / FAIL
|
||||
- Scenario 3 (reconnect one standby): PASS / FAIL
|
||||
|
||||
### 11.3 Unconfigured Node Bootstrap
|
||||
### 11.3 Unconfigured Node Bootstrap (ctrl-c joins ctrl-a+ctrl-b)
|
||||
- Result: PASS / FAIL
|
||||
- cluster.bin created in /tmp/xce-b/? YES / NO
|
||||
- cluster.bin created in /tmp/xce-c/? YES / NO
|
||||
- Log excerpt:
|
||||
|
||||
### 11.4 Term Durability
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue