From 25c71adffbb1bbf267b30160ef2540c5c01c8cc7 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Fri, 29 May 2026 11:02:45 +0000 Subject: [PATCH] docs(ha-config-bootstrap): Test 11.1 now covers config replication verification --- .../specs/ha-config-bootstrap/manual-tests.md | 52 +++++++++++++------ 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/.kiro/specs/ha-config-bootstrap/manual-tests.md b/.kiro/specs/ha-config-bootstrap/manual-tests.md index da34913..e5d04f9 100644 --- a/.kiro/specs/ha-config-bootstrap/manual-tests.md +++ b/.kiro/specs/ha-config-bootstrap/manual-tests.md @@ -144,26 +144,42 @@ tail -f /tmp/xce-{a,b,c}/ctrl.log | grep -E "CTRL_|election|quorum|RDHUP|ha-term --- -## Test 11.1 — Concurrent Startup and Config-Fresh Election +## Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication -**Requirements:** 1.1, 2.1, 2.7 +**Requirements:** 1.1, 2.1, 2.7, 3.2, 3.5 -**Scenario:** Three nodes start within 3 seconds — exactly one wins election (needs majority = 2/3 votes). +**Scenario:** ctrl-a has the initial config; ctrl-b and ctrl-c start with +`config_tag=0` (no config). All three start within 3s — ctrl-a wins election +(freshest config), then pushes the config to ctrl-b and ctrl-c via replication. + +This test verifies election AND config bootstrap replication in a single run. 1. Start all 3 nodes (Terminals 1–3) within 3 seconds of each other. -2. Wait 10 seconds. +2. Wait 15 seconds. 3. Check the logs on all three. -**Expected log patterns:** -- Exactly **one** node logs: `am_i_primary_quorum: won` or `CTRL_PRIMARY` -- Other **two** nodes log: `CTRL_STANDBY` -- No node stays in `CTRL_ELECTING` loop indefinitely +**Expected log patterns — Election:** +- ctrl-a logs: `am_i_primary_quorum: won` → `CTRL_PRIMARY` +- ctrl-b and ctrl-c log: `defer_config_fetch=1` → yield to fresher peer → `CTRL_STANDBY` + +**Expected log patterns — Config replication:** +- ctrl-a logs: `Config committed on majority (tag=...)` +- ctrl-b and ctrl-c log: `Config fetch successful from primary (tag=...)` +- Config files appear in both standby workdirs: + ```bash + ls /tmp/xce-b/ctrl-b/conf/ + ls /tmp/xce-c/ctrl-c/conf/ + # Expected: cluster.bin. and current + ``` **Checklist:** -- [ ] Exactly one node becomes `CTRL_PRIMARY` -- [ ] Two nodes become `CTRL_STANDBY` +- [ ] Exactly ctrl-a becomes `CTRL_PRIMARY` +- [ ] ctrl-b and ctrl-c become `CTRL_STANDBY` - [ ] No crash or ASAN error - [ ] All three nodes started within 3s (no false self-election) +- [ ] Config replicated to ctrl-b: `cluster.bin.*` exists in `/tmp/xce-b/ctrl-b/conf/` +- [ ] Config replicated to ctrl-c: `cluster.bin.*` exists in `/tmp/xce-c/ctrl-c/conf/` +- [ ] Config tag on standbys matches primary's tag --- @@ -326,12 +342,16 @@ Build: /xce-build completed successfully? YES / NO - controller_unit_test: PASS / FAIL - xcelib_test: PASS / FAIL -### 11.1 Concurrent Startup (3 nodes) -- Result: PASS / FAIL -- Primary elected: ctrl-a / ctrl-b / ctrl-c -- Time to election: ~Xs -- Two standbys confirmed: YES / NO -- Log excerpt (primary win + both standbys): +### 11.1 Concurrent Startup + Config Replication (3 nodes) +- Election result: PASS / FAIL + - Primary elected: ctrl-a / ctrl-b / ctrl-c + - Time to election: ~Xs + - Two standbys confirmed: YES / NO +- Config replication result: PASS / FAIL + - cluster.bin exists in /tmp/xce-b/ctrl-b/conf/: YES / NO + - cluster.bin exists in /tmp/xce-c/ctrl-c/conf/: YES / NO + - Config tag on standbys matches primary: YES / NO +- Log excerpt (primary win + config push + standby fetch): ### 11.2 Failover and WAL Durability (3 nodes) - Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS / FAIL