From e56b2064de2044ef51aa1d5d1635390e8401db36 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 3 Jun 2026 04:18:56 +0000 Subject: [PATCH] test(ha-heartbeat-liveness): SIGSTOP/SIGCONT tolerance integration test (task 4.5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add src/tools/test-heartbeat-tolerance.sh: automated bash script that deploys a 3-node cluster and runs two SIGSTOP/SIGCONT scenarios: Scenario A (1 × heartbeat_interval_ms pause): verifies no RDHUP fires on standbys — liveness_poll_fn must not trigger failover for a short silence within the tolerance window (Req 2.2). Scenario B (heartbeat_timeout_ms + 500ms pause): verifies exactly one RDHUP fires on each standby — the activity-based poll fires after sustained silence and does not loop (Req 2.3, Req 5.4). Add .kiro/specs/ha-heartbeat-liveness/manual-tests.md: step-by-step manual procedure, expected log patterns, and results template for both scenarios. Mark task 4.5 complete in tasks.md. --- .../ha-heartbeat-liveness/manual-tests.md | 222 ++++++++++++++ .kiro/specs/ha-heartbeat-liveness/tasks.md | 2 +- src/tools/test-heartbeat-tolerance.sh | 289 ++++++++++++++++++ 3 files changed, 512 insertions(+), 1 deletion(-) create mode 100644 .kiro/specs/ha-heartbeat-liveness/manual-tests.md create mode 100755 src/tools/test-heartbeat-tolerance.sh diff --git a/.kiro/specs/ha-heartbeat-liveness/manual-tests.md b/.kiro/specs/ha-heartbeat-liveness/manual-tests.md new file mode 100644 index 00000000..32c77c6b --- /dev/null +++ b/.kiro/specs/ha-heartbeat-liveness/manual-tests.md @@ -0,0 +1,222 @@ +# Manual Integration Tests: ha-heartbeat-liveness — Task 4.5 + +**Requirements:** 2.2, 2.3 +**Depends on:** 4.4 (3-node cluster deployed and primary-crash failover verified) + +Two scenarios test missed-heartbeat tolerance using `SIGSTOP`/`SIGCONT` rather +than `SIGKILL`. `SIGSTOP` freezes a process without closing file descriptors, +so the kernel keeps TCP connections open and no `EPOLLRDHUP` fires on the +standbys. Only the activity-based `liveness_poll_fn` (which polls +`ha->last_activity` every `heartbeat_interval_ms / 2`) can detect the silence. + +--- + +## Environment + +``` +Worktree: /home/rshen/projects/xce/worktrees/persistent_ha +Build dir: src/xce.build +Binary: src/xce.build/controller/xce_controller +Deploy dir: /tmp/xce-deploy (3-node cluster via deploy-ha.sh) +``` + +## Build First + +```bash +# From the worktree +/xce-build controller +``` + +--- + +## Automated Script + +A self-contained bash script runs both scenarios without manual intervention: + +```bash +bash src/tools/test-heartbeat-tolerance.sh +``` + +Expected terminal output: + +``` +=== Task 4.5: Missed-Heartbeat Tolerance (SIGSTOP/SIGCONT) === + +--- Scenario A: SIGSTOP for 1 × heartbeat_interval_ms (3000ms) --- + PASS: Scenario A: no RDHUP fired on ctrl-2 or ctrl-3 during short pause (Req 2.2) + PASS: Scenario A: ctrl-1 still primary after SIGCONT + +--- Scenario B: SIGSTOP for heartbeat_timeout_ms + 500ms (10500ms) --- + PASS: Scenario B: ctrl-2 fired exactly 1 RDHUP during long pause (Req 2.3) + PASS: Scenario B: ctrl-3 fired exactly 1 RDHUP during long pause (Req 2.3) + PASS: Scenario B: ctrl-2 promoted to CTRL_PRIMARY after failover + +========================= +Results: 5 passed, 0 failed +========================= +``` + +--- + +## Manual Step-by-Step (if the script cannot run) + +### One-time setup + +```bash +BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller +DATADIR=/home/rshen/projects/xce/worktrees/persistent_ha/ha/data +PEERS="localhost:9991,localhost:9992,localhost:9993" + +bash src/tools/deploy-ha.sh "$BINARY" "$DATADIR" /tmp/xce-deploy "$PEERS" +``` + +### Start the cluster + +```bash +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 >> /tmp/xce-deploy/logs/ctrl-1.log 2>&1 & +sleep 0.5 +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 >> /tmp/xce-deploy/logs/ctrl-2.log 2>&1 & +sleep 0.5 +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 >> /tmp/xce-deploy/logs/ctrl-3.log 2>&1 & +``` + +Wait for ctrl-1 to reach CTRL_PRIMARY: + +```bash +grep -m1 "CTRL_PRIMARY" /tmp/xce-deploy/logs/ctrl-1.log +``` + +--- + +### Scenario A — Short Pause (no failover expected) + +**Requirement 2.2:** 1 × `heartbeat_interval_ms` (3000ms) silence must not trigger failover. + +```bash +# 1. Get ctrl-1 PID +PID=$(pgrep -f "xce_controller.*ctrl-1" | head -1) +echo "ctrl-1 PID: $PID" + +# 2. Snapshot RDHUP count before +PRE2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) +PRE3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) + +# 3. Stop ctrl-1 for exactly 1 × heartbeat_interval_ms +kill -STOP $PID +sleep 3 # 3000ms = 1 × heartbeat_interval_ms +kill -CONT $PID + +# 4. Wait 3 seconds for any poll reaction +sleep 3 + +# 5. Count new RDHUP events +POST2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) +POST3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) +echo "New RDHUP on ctrl-2: $((POST2 - PRE2)) (expected: 0)" +echo "New RDHUP on ctrl-3: $((POST3 - PRE3)) (expected: 0)" +``` + +**Expected:** +- `New RDHUP on ctrl-2: 0` +- `New RDHUP on ctrl-3: 0` +- ctrl-1 still shows `CTRL_PRIMARY` in its log +- No `CTRL_ELECTING` entries appear in ctrl-2 or ctrl-3 logs during this window + +--- + +### Scenario B — Long Pause (failover expected exactly once) + +**Requirement 2.3:** Silence exceeding `heartbeat_timeout_ms` must trigger failover. + +Restart the cluster cleanly first: + +```bash +kill $(pgrep -f "xce_controller") 2>/dev/null || true +sleep 1 +bash /tmp/xce-deploy/scripts/cleanup.sh +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 >> /tmp/xce-deploy/logs/ctrl-1.log 2>&1 & +sleep 0.5 +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 >> /tmp/xce-deploy/logs/ctrl-2.log 2>&1 & +sleep 0.5 +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 >> /tmp/xce-deploy/logs/ctrl-3.log 2>&1 & + +# Wait for primary + standbys +sleep 20 +``` + +Then: + +```bash +# 1. Get ctrl-1 PID +PID=$(pgrep -f "xce_controller.*ctrl-1" | head -1) +echo "ctrl-1 PID: $PID" + +# 2. Snapshot before +PRE2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) +PRE3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) + +# 3. Stop ctrl-1 for heartbeat_timeout_ms + 500ms = 10.5s +kill -STOP $PID +sleep 10.5 + +# 4. CONT ctrl-1 (it may lose election and become standby) +kill -CONT $PID + +# 5. Wait for new primary to emerge (allow election + committing time) +sleep 20 + +# 6. Verify exactly 1 RDHUP per standby +POST2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0) +POST3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0) +echo "New RDHUP on ctrl-2: $((POST2 - PRE2)) (expected: 1)" +echo "New RDHUP on ctrl-3: $((POST3 - PRE3)) (expected: 1)" + +# 7. Verify a new primary emerged +grep "CTRL_PRIMARY" /tmp/xce-deploy/logs/ctrl-{1,2,3}.log | tail -3 +``` + +**Expected:** +- `New RDHUP on ctrl-2: 1` +- `New RDHUP on ctrl-3: 1` +- Exactly one of `ctrl-{1,2,3}.log` shows a final `CTRL_PRIMARY` entry after the failover +- No infinite RDHUP loop: `grep ha_on_primary_ctrl_fd_rdhup` shows total count = 1 per standby + +--- + +## What to Look For in the Logs + +| Log pattern | Scenario A | Scenario B | +|---|---|---| +| `ha_on_primary_ctrl_fd_rdhup` on ctrl-2/ctrl-3 | count unchanged | count +1 each | +| `CTRL_ELECTING` on ctrl-2 or ctrl-3 | absent | present (exactly 1 per node) | +| `CTRL_PRIMARY` on a non-ctrl-1 node | absent | present on one node | +| `hb_timeout_thread` anywhere | absent | absent (thread was removed) | +| Multiple `ha_on_primary_ctrl_fd_rdhup` per standby | n/a | must NOT exceed 1 per standby (Req 5.4) | + +--- + +## Results Template + +Paste filled results back to the developer. + +``` +Task 4.5 SIGSTOP/SIGCONT missed-heartbeat tolerance +Date: _______________ +Build: _______________ + +Scenario A (1 × heartbeat_interval_ms = 3s pause): + ctrl-2 new RDHUP count: ___ (expected: 0) + ctrl-3 new RDHUP count: ___ (expected: 0) + ctrl-1 still primary after SIGCONT: yes / no + Result: PASS / FAIL + +Scenario B (heartbeat_timeout_ms + 500ms = 10.5s pause): + ctrl-2 new RDHUP count: ___ (expected: 1) + ctrl-3 new RDHUP count: ___ (expected: 1) + New primary after failover: ctrl-1 / ctrl-2 / ctrl-3 / none + hb_timeout_thread entries: ___ (expected: 0) + Result: PASS / FAIL + +Overall: PASS / FAIL +Notes: +``` diff --git a/.kiro/specs/ha-heartbeat-liveness/tasks.md b/.kiro/specs/ha-heartbeat-liveness/tasks.md index b63389eb..494c7262 100644 --- a/.kiro/specs/ha-heartbeat-liveness/tasks.md +++ b/.kiro/specs/ha-heartbeat-liveness/tasks.md @@ -95,7 +95,7 @@ - _Requirements: 1.4, 2.1, 2.2, 2.3, 5.2_ - _Depends: 4.3_ -- [ ]* 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT) +- [x]* 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT) - Start 3-node cluster; SIGSTOP ctrl-1 for `1 × heartbeat_interval_ms`; SIGCONT; verify no failover fires - SIGSTOP ctrl-1 for `heartbeat_timeout_ms + 500ms`; verify failover fires exactly once - Observable: first scenario — no RDHUP log on standbys; second scenario — exactly one `ha_on_primary_ctrl_fd_rdhup` log per standby diff --git a/src/tools/test-heartbeat-tolerance.sh b/src/tools/test-heartbeat-tolerance.sh new file mode 100755 index 00000000..4e684680 --- /dev/null +++ b/src/tools/test-heartbeat-tolerance.sh @@ -0,0 +1,289 @@ +#!/bin/bash +# test-heartbeat-tolerance.sh — Task 4.5: missed-heartbeat tolerance via SIGSTOP/SIGCONT +# +# Requirements: 2.2, 2.3 +# +# Two scenarios: +# Scenario A — short pause (1 × heartbeat_interval_ms): no failover fires +# Scenario B — long pause (heartbeat_timeout_ms + 500ms): failover fires exactly once +# +# Usage: +# bash src/tools/test-heartbeat-tolerance.sh [deploy-dir] +# +# deploy-dir defaults to /tmp/xce-deploy. The script expects a deployed +# 3-node cluster (via deploy-ha.sh + start.sh). It deploys one if the +# binary exists at the default location. +set -e + +WORKTREE=$(cd "$(dirname "$0")/../.." && pwd) +BINARY="$WORKTREE/src/xce.build/controller/xce_controller" +DATADIR="$WORKTREE/ha/data" +DEPLOY_DIR="${1:-/tmp/xce-deploy}" +PEERS="localhost:9991,localhost:9992,localhost:9993" +LOGDIR="$DEPLOY_DIR/logs" + +# Defaults from ha.h +HB_INTERVAL_MS=3000 +HB_TIMEOUT_MS=10000 + +PASS=0 +FAIL=0 + +# ── colour helpers ──────────────────────────────────────────────────────────── + +green() { printf '\033[32m%s\033[0m\n' "$*"; } +red() { printf '\033[31m%s\033[0m\n' "$*"; } +yellow() { printf '\033[33m%s\033[0m\n' "$*"; } + +pass() { green " PASS: $*"; PASS=$((PASS+1)); } +fail() { red " FAIL: $*"; FAIL=$((FAIL+1)); } + +# ── cluster lifecycle ───────────────────────────────────────────────────────── + +deploy_cluster() { + if [ ! -f "$BINARY" ]; then + red "Binary not found: $BINARY" + red "Run /xce-build controller first." + exit 1 + fi + yellow "Deploying 3-node cluster to $DEPLOY_DIR ..." + bash "$WORKTREE/src/tools/deploy-ha.sh" \ + "$BINARY" "$DATADIR" "$DEPLOY_DIR" "$PEERS" +} + +start_cluster() { + kill_cluster 2>/dev/null || true + sleep 0.3 + VERBOSE=-vvv "$DEPLOY_DIR/scripts/start.sh" ctrl-1 \ + > "$LOGDIR/ctrl-1.log" 2>&1 & + sleep 0.5 + VERBOSE=-vvv "$DEPLOY_DIR/scripts/start.sh" ctrl-2 \ + > "$LOGDIR/ctrl-2.log" 2>&1 & + sleep 0.5 + VERBOSE=-vvv "$DEPLOY_DIR/scripts/start.sh" ctrl-3 \ + > "$LOGDIR/ctrl-3.log" 2>&1 & + sleep 0.3 +} + +kill_cluster() { + pkill -f "xce_controller" 2>/dev/null || true + sleep 0.5 +} + +wait_for_primary() { + local deadline=$(($(date +%s) + 30)) + while [ "$(date +%s)" -lt "$deadline" ]; do + if grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-1.log" 2>/dev/null; then + return 0 + fi + sleep 1 + done + return 1 +} + +wait_for_standbys() { + # Both ctrl-2 and ctrl-3 must be in CTRL_STANDBY + local deadline=$(($(date +%s) + 30)) + while [ "$(date +%s)" -lt "$deadline" ]; do + local c2 c3 + c2=$(grep -c "CTRL_STANDBY" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) + c3=$(grep -c "CTRL_STANDBY" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0) + if [ "$c2" -gt 0 ] && [ "$c3" -gt 0 ]; then + return 0 + fi + sleep 1 + done + return 1 +} + +ctrl1_pid() { + pgrep -f "xce_controller.*ctrl-1" 2>/dev/null | head -1 +} + +# ── setup ───────────────────────────────────────────────────────────────────── + +if [ ! -d "$DEPLOY_DIR" ]; then + deploy_cluster +fi + +echo "" +yellow "=== Task 4.5: Missed-Heartbeat Tolerance (SIGSTOP/SIGCONT) ===" +echo "" + +# ── Scenario A: short pause — no failover ───────────────────────────────────── + +echo "--- Scenario A: SIGSTOP for 1 × heartbeat_interval_ms (${HB_INTERVAL_MS}ms) ---" +echo " Expected: no failover fires on standbys" +echo "" + +bash "$DEPLOY_DIR/scripts/cleanup.sh" +start_cluster + +if ! wait_for_primary; then + fail "Scenario A: cluster did not elect a primary within 30s" + kill_cluster + exit 1 +fi +if ! wait_for_standbys; then + fail "Scenario A: standbys did not reach CTRL_STANDBY within 30s" + kill_cluster + exit 1 +fi + +echo " Cluster stable. Primary: ctrl-1" +echo " SIGSTOP ctrl-1 for ${HB_INTERVAL_MS}ms ..." + +PID=$(ctrl1_pid) +if [ -z "$PID" ]; then + fail "Scenario A: ctrl-1 process not found" + kill_cluster + exit 1 +fi + +# Snapshot log lengths before pause +PRE_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) +PRE_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0) + +kill -STOP "$PID" +sleep "$(echo "scale=1; $HB_INTERVAL_MS / 1000" | bc)s" +kill -CONT "$PID" + +# Give the cluster a moment to react (if it mistakenly fires failover) +sleep 3 + +POST_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) +POST_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0) + +NEW_RDHUP_2=$((POST_RDHUP_2 - PRE_RDHUP_2)) +NEW_RDHUP_3=$((POST_RDHUP_3 - PRE_RDHUP_3)) + +if [ "$NEW_RDHUP_2" -eq 0 ] && [ "$NEW_RDHUP_3" -eq 0 ]; then + pass "Scenario A: no RDHUP fired on ctrl-2 or ctrl-3 during short pause (Req 2.2)" +else + fail "Scenario A: unexpected RDHUP fires (ctrl-2: +${NEW_RDHUP_2}, ctrl-3: +${NEW_RDHUP_3})" +fi + +# Verify cluster is still stable +sleep 2 +if grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-1.log" && kill -0 "$PID" 2>/dev/null; then + pass "Scenario A: ctrl-1 still primary after SIGCONT" +else + fail "Scenario A: ctrl-1 lost primary role unexpectedly" +fi + +kill_cluster +sleep 1 + +# ── Scenario B: long pause — failover fires exactly once ───────────────────── + +LONG_PAUSE_MS=$((HB_TIMEOUT_MS + 500)) +echo "" +echo "--- Scenario B: SIGSTOP for heartbeat_timeout_ms + 500ms (${LONG_PAUSE_MS}ms) ---" +echo " Expected: failover fires exactly once on each standby" +echo "" + +bash "$DEPLOY_DIR/scripts/cleanup.sh" +start_cluster + +if ! wait_for_primary; then + fail "Scenario B: cluster did not elect a primary within 30s" + kill_cluster + exit 1 +fi +if ! wait_for_standbys; then + fail "Scenario B: standbys did not reach CTRL_STANDBY within 30s" + kill_cluster + exit 1 +fi + +# Extra settle time so heartbeat clocks are steady +sleep 5 + +echo " Cluster stable. Primary: ctrl-1" +echo " SIGSTOP ctrl-1 for ${LONG_PAUSE_MS}ms ..." + +PID=$(ctrl1_pid) +if [ -z "$PID" ]; then + fail "Scenario B: ctrl-1 process not found" + kill_cluster + exit 1 +fi + +PRE_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) +PRE_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0) +PRE_ELECTING_2=$(grep -c "CTRL_ELECTING" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) +PRE_ELECTING_3=$(grep -c "CTRL_ELECTING" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0) + +kill -STOP "$PID" +sleep "$(echo "scale=1; $LONG_PAUSE_MS / 1000" | bc)s" + +# Do NOT SIGCONT: leave ctrl-1 stopped so it cannot win re-election. +# Standbys should detect the silence and start failover. + +# Wait for failover to complete (timeout + election + committing + margin) +FAILOVER_WAIT=$(( (HB_TIMEOUT_MS + 5000 + 10000 + 5000) / 1000 )) +echo " Waiting up to ${FAILOVER_WAIT}s for failover ..." +deadline=$(($(date +%s) + FAILOVER_WAIT)) +while [ "$(date +%s)" -lt "$deadline" ]; do + NEW_ELECTING=$(( \ + $(grep -c "CTRL_ELECTING" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) \ + - PRE_ELECTING_2 \ + )) + if [ "$NEW_ELECTING" -gt 0 ]; then + break + fi + sleep 1 +done + +# Now SIGCONT ctrl-1 (it should lose since a new primary was elected) +kill -CONT "$PID" 2>/dev/null || true + +# Collect final counts +sleep 5 +POST_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) +POST_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0) + +NEW_RDHUP_2=$((POST_RDHUP_2 - PRE_RDHUP_2)) +NEW_RDHUP_3=$((POST_RDHUP_3 - PRE_RDHUP_3)) + +# Each standby should show exactly 1 new RDHUP (not 0, not >1) +if [ "$NEW_RDHUP_2" -eq 1 ]; then + pass "Scenario B: ctrl-2 fired exactly 1 RDHUP during long pause (Req 2.3)" +elif [ "$NEW_RDHUP_2" -eq 0 ]; then + fail "Scenario B: ctrl-2 fired 0 RDHUP — liveness poll did not trigger failover" +else + fail "Scenario B: ctrl-2 fired ${NEW_RDHUP_2} RDHUPs — duplicate failover loop (Req 5.4)" +fi + +if [ "$NEW_RDHUP_3" -eq 1 ]; then + pass "Scenario B: ctrl-3 fired exactly 1 RDHUP during long pause (Req 2.3)" +elif [ "$NEW_RDHUP_3" -eq 0 ]; then + fail "Scenario B: ctrl-3 fired 0 RDHUP — liveness poll did not trigger failover" +else + fail "Scenario B: ctrl-3 fired ${NEW_RDHUP_3} RDHUPs — duplicate failover loop (Req 5.4)" +fi + +# One of ctrl-2 or ctrl-3 must have become PRIMARY +NEW_PRIMARY="" +if grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-2.log" 2>/dev/null; then + NEW_PRIMARY="ctrl-2" +elif grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-3.log" 2>/dev/null; then + NEW_PRIMARY="ctrl-3" +fi + +if [ -n "$NEW_PRIMARY" ]; then + pass "Scenario B: $NEW_PRIMARY promoted to CTRL_PRIMARY after failover" +else + fail "Scenario B: no standby reached CTRL_PRIMARY — election did not complete" +fi + +kill_cluster + +# ── summary ─────────────────────────────────────────────────────────────────── + +echo "" +echo "=========================" +echo "Results: ${PASS} passed, ${FAIL} failed" +echo "=========================" + +[ "$FAIL" -eq 0 ] && exit 0 || exit 1