test(ha-heartbeat-liveness): SIGSTOP/SIGCONT tolerance integration test (task 4.5)
Add src/tools/test-heartbeat-tolerance.sh: automated bash script that deploys a 3-node cluster and runs two SIGSTOP/SIGCONT scenarios: Scenario A (1 × heartbeat_interval_ms pause): verifies no RDHUP fires on standbys — liveness_poll_fn must not trigger failover for a short silence within the tolerance window (Req 2.2). Scenario B (heartbeat_timeout_ms + 500ms pause): verifies exactly one RDHUP fires on each standby — the activity-based poll fires after sustained silence and does not loop (Req 2.3, Req 5.4). Add .kiro/specs/ha-heartbeat-liveness/manual-tests.md: step-by-step manual procedure, expected log patterns, and results template for both scenarios. Mark task 4.5 complete in tasks.md.
This commit is contained in:
parent
529552c127
commit
e56b2064de
3 changed files with 512 additions and 1 deletions
222
.kiro/specs/ha-heartbeat-liveness/manual-tests.md
Normal file
222
.kiro/specs/ha-heartbeat-liveness/manual-tests.md
Normal file
|
|
@ -0,0 +1,222 @@
|
|||
# Manual Integration Tests: ha-heartbeat-liveness — Task 4.5
|
||||
|
||||
**Requirements:** 2.2, 2.3
|
||||
**Depends on:** 4.4 (3-node cluster deployed and primary-crash failover verified)
|
||||
|
||||
Two scenarios test missed-heartbeat tolerance using `SIGSTOP`/`SIGCONT` rather
|
||||
than `SIGKILL`. `SIGSTOP` freezes a process without closing file descriptors,
|
||||
so the kernel keeps TCP connections open and no `EPOLLRDHUP` fires on the
|
||||
standbys. Only the activity-based `liveness_poll_fn` (which polls
|
||||
`ha->last_activity` every `heartbeat_interval_ms / 2`) can detect the silence.
|
||||
|
||||
---
|
||||
|
||||
## Environment
|
||||
|
||||
```
|
||||
Worktree: /home/rshen/projects/xce/worktrees/persistent_ha
|
||||
Build dir: src/xce.build
|
||||
Binary: src/xce.build/controller/xce_controller
|
||||
Deploy dir: /tmp/xce-deploy (3-node cluster via deploy-ha.sh)
|
||||
```
|
||||
|
||||
## Build First
|
||||
|
||||
```bash
|
||||
# From the worktree
|
||||
/xce-build controller
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Automated Script
|
||||
|
||||
A self-contained bash script runs both scenarios without manual intervention:
|
||||
|
||||
```bash
|
||||
bash src/tools/test-heartbeat-tolerance.sh
|
||||
```
|
||||
|
||||
Expected terminal output:
|
||||
|
||||
```
|
||||
=== Task 4.5: Missed-Heartbeat Tolerance (SIGSTOP/SIGCONT) ===
|
||||
|
||||
--- Scenario A: SIGSTOP for 1 × heartbeat_interval_ms (3000ms) ---
|
||||
PASS: Scenario A: no RDHUP fired on ctrl-2 or ctrl-3 during short pause (Req 2.2)
|
||||
PASS: Scenario A: ctrl-1 still primary after SIGCONT
|
||||
|
||||
--- Scenario B: SIGSTOP for heartbeat_timeout_ms + 500ms (10500ms) ---
|
||||
PASS: Scenario B: ctrl-2 fired exactly 1 RDHUP during long pause (Req 2.3)
|
||||
PASS: Scenario B: ctrl-3 fired exactly 1 RDHUP during long pause (Req 2.3)
|
||||
PASS: Scenario B: ctrl-2 promoted to CTRL_PRIMARY after failover
|
||||
|
||||
=========================
|
||||
Results: 5 passed, 0 failed
|
||||
=========================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manual Step-by-Step (if the script cannot run)
|
||||
|
||||
### One-time setup
|
||||
|
||||
```bash
|
||||
BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller
|
||||
DATADIR=/home/rshen/projects/xce/worktrees/persistent_ha/ha/data
|
||||
PEERS="localhost:9991,localhost:9992,localhost:9993"
|
||||
|
||||
bash src/tools/deploy-ha.sh "$BINARY" "$DATADIR" /tmp/xce-deploy "$PEERS"
|
||||
```
|
||||
|
||||
### Start the cluster
|
||||
|
||||
```bash
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 >> /tmp/xce-deploy/logs/ctrl-1.log 2>&1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 >> /tmp/xce-deploy/logs/ctrl-2.log 2>&1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 >> /tmp/xce-deploy/logs/ctrl-3.log 2>&1 &
|
||||
```
|
||||
|
||||
Wait for ctrl-1 to reach CTRL_PRIMARY:
|
||||
|
||||
```bash
|
||||
grep -m1 "CTRL_PRIMARY" /tmp/xce-deploy/logs/ctrl-1.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scenario A — Short Pause (no failover expected)
|
||||
|
||||
**Requirement 2.2:** 1 × `heartbeat_interval_ms` (3000ms) silence must not trigger failover.
|
||||
|
||||
```bash
|
||||
# 1. Get ctrl-1 PID
|
||||
PID=$(pgrep -f "xce_controller.*ctrl-1" | head -1)
|
||||
echo "ctrl-1 PID: $PID"
|
||||
|
||||
# 2. Snapshot RDHUP count before
|
||||
PRE2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0)
|
||||
PRE3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0)
|
||||
|
||||
# 3. Stop ctrl-1 for exactly 1 × heartbeat_interval_ms
|
||||
kill -STOP $PID
|
||||
sleep 3 # 3000ms = 1 × heartbeat_interval_ms
|
||||
kill -CONT $PID
|
||||
|
||||
# 4. Wait 3 seconds for any poll reaction
|
||||
sleep 3
|
||||
|
||||
# 5. Count new RDHUP events
|
||||
POST2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0)
|
||||
POST3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0)
|
||||
echo "New RDHUP on ctrl-2: $((POST2 - PRE2)) (expected: 0)"
|
||||
echo "New RDHUP on ctrl-3: $((POST3 - PRE3)) (expected: 0)"
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
- `New RDHUP on ctrl-2: 0`
|
||||
- `New RDHUP on ctrl-3: 0`
|
||||
- ctrl-1 still shows `CTRL_PRIMARY` in its log
|
||||
- No `CTRL_ELECTING` entries appear in ctrl-2 or ctrl-3 logs during this window
|
||||
|
||||
---
|
||||
|
||||
### Scenario B — Long Pause (failover expected exactly once)
|
||||
|
||||
**Requirement 2.3:** Silence exceeding `heartbeat_timeout_ms` must trigger failover.
|
||||
|
||||
Restart the cluster cleanly first:
|
||||
|
||||
```bash
|
||||
kill $(pgrep -f "xce_controller") 2>/dev/null || true
|
||||
sleep 1
|
||||
bash /tmp/xce-deploy/scripts/cleanup.sh
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 >> /tmp/xce-deploy/logs/ctrl-1.log 2>&1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 >> /tmp/xce-deploy/logs/ctrl-2.log 2>&1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 >> /tmp/xce-deploy/logs/ctrl-3.log 2>&1 &
|
||||
|
||||
# Wait for primary + standbys
|
||||
sleep 20
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
# 1. Get ctrl-1 PID
|
||||
PID=$(pgrep -f "xce_controller.*ctrl-1" | head -1)
|
||||
echo "ctrl-1 PID: $PID"
|
||||
|
||||
# 2. Snapshot before
|
||||
PRE2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0)
|
||||
PRE3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0)
|
||||
|
||||
# 3. Stop ctrl-1 for heartbeat_timeout_ms + 500ms = 10.5s
|
||||
kill -STOP $PID
|
||||
sleep 10.5
|
||||
|
||||
# 4. CONT ctrl-1 (it may lose election and become standby)
|
||||
kill -CONT $PID
|
||||
|
||||
# 5. Wait for new primary to emerge (allow election + committing time)
|
||||
sleep 20
|
||||
|
||||
# 6. Verify exactly 1 RDHUP per standby
|
||||
POST2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-2.log 2>/dev/null || echo 0)
|
||||
POST3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" /tmp/xce-deploy/logs/ctrl-3.log 2>/dev/null || echo 0)
|
||||
echo "New RDHUP on ctrl-2: $((POST2 - PRE2)) (expected: 1)"
|
||||
echo "New RDHUP on ctrl-3: $((POST3 - PRE3)) (expected: 1)"
|
||||
|
||||
# 7. Verify a new primary emerged
|
||||
grep "CTRL_PRIMARY" /tmp/xce-deploy/logs/ctrl-{1,2,3}.log | tail -3
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
- `New RDHUP on ctrl-2: 1`
|
||||
- `New RDHUP on ctrl-3: 1`
|
||||
- Exactly one of `ctrl-{1,2,3}.log` shows a final `CTRL_PRIMARY` entry after the failover
|
||||
- No infinite RDHUP loop: `grep ha_on_primary_ctrl_fd_rdhup` shows total count = 1 per standby
|
||||
|
||||
---
|
||||
|
||||
## What to Look For in the Logs
|
||||
|
||||
| Log pattern | Scenario A | Scenario B |
|
||||
|---|---|---|
|
||||
| `ha_on_primary_ctrl_fd_rdhup` on ctrl-2/ctrl-3 | count unchanged | count +1 each |
|
||||
| `CTRL_ELECTING` on ctrl-2 or ctrl-3 | absent | present (exactly 1 per node) |
|
||||
| `CTRL_PRIMARY` on a non-ctrl-1 node | absent | present on one node |
|
||||
| `hb_timeout_thread` anywhere | absent | absent (thread was removed) |
|
||||
| Multiple `ha_on_primary_ctrl_fd_rdhup` per standby | n/a | must NOT exceed 1 per standby (Req 5.4) |
|
||||
|
||||
---
|
||||
|
||||
## Results Template
|
||||
|
||||
Paste filled results back to the developer.
|
||||
|
||||
```
|
||||
Task 4.5 SIGSTOP/SIGCONT missed-heartbeat tolerance
|
||||
Date: _______________
|
||||
Build: _______________
|
||||
|
||||
Scenario A (1 × heartbeat_interval_ms = 3s pause):
|
||||
ctrl-2 new RDHUP count: ___ (expected: 0)
|
||||
ctrl-3 new RDHUP count: ___ (expected: 0)
|
||||
ctrl-1 still primary after SIGCONT: yes / no
|
||||
Result: PASS / FAIL
|
||||
|
||||
Scenario B (heartbeat_timeout_ms + 500ms = 10.5s pause):
|
||||
ctrl-2 new RDHUP count: ___ (expected: 1)
|
||||
ctrl-3 new RDHUP count: ___ (expected: 1)
|
||||
New primary after failover: ctrl-1 / ctrl-2 / ctrl-3 / none
|
||||
hb_timeout_thread entries: ___ (expected: 0)
|
||||
Result: PASS / FAIL
|
||||
|
||||
Overall: PASS / FAIL
|
||||
Notes:
|
||||
```
|
||||
|
|
@ -95,7 +95,7 @@
|
|||
- _Requirements: 1.4, 2.1, 2.2, 2.3, 5.2_
|
||||
- _Depends: 4.3_
|
||||
|
||||
- [ ]* 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT)
|
||||
- [x]* 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT)
|
||||
- Start 3-node cluster; SIGSTOP ctrl-1 for `1 × heartbeat_interval_ms`; SIGCONT; verify no failover fires
|
||||
- SIGSTOP ctrl-1 for `heartbeat_timeout_ms + 500ms`; verify failover fires exactly once
|
||||
- Observable: first scenario — no RDHUP log on standbys; second scenario — exactly one `ha_on_primary_ctrl_fd_rdhup` log per standby
|
||||
|
|
|
|||
289
src/tools/test-heartbeat-tolerance.sh
Executable file
289
src/tools/test-heartbeat-tolerance.sh
Executable file
|
|
@ -0,0 +1,289 @@
|
|||
#!/bin/bash
|
||||
# test-heartbeat-tolerance.sh — Task 4.5: missed-heartbeat tolerance via SIGSTOP/SIGCONT
|
||||
#
|
||||
# Requirements: 2.2, 2.3
|
||||
#
|
||||
# Two scenarios:
|
||||
# Scenario A — short pause (1 × heartbeat_interval_ms): no failover fires
|
||||
# Scenario B — long pause (heartbeat_timeout_ms + 500ms): failover fires exactly once
|
||||
#
|
||||
# Usage:
|
||||
# bash src/tools/test-heartbeat-tolerance.sh [deploy-dir]
|
||||
#
|
||||
# deploy-dir defaults to /tmp/xce-deploy. The script expects a deployed
|
||||
# 3-node cluster (via deploy-ha.sh + start.sh). It deploys one if the
|
||||
# binary exists at the default location.
|
||||
set -e
|
||||
|
||||
WORKTREE=$(cd "$(dirname "$0")/../.." && pwd)
|
||||
BINARY="$WORKTREE/src/xce.build/controller/xce_controller"
|
||||
DATADIR="$WORKTREE/ha/data"
|
||||
DEPLOY_DIR="${1:-/tmp/xce-deploy}"
|
||||
PEERS="localhost:9991,localhost:9992,localhost:9993"
|
||||
LOGDIR="$DEPLOY_DIR/logs"
|
||||
|
||||
# Defaults from ha.h
|
||||
HB_INTERVAL_MS=3000
|
||||
HB_TIMEOUT_MS=10000
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
# ── colour helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
green() { printf '\033[32m%s\033[0m\n' "$*"; }
|
||||
red() { printf '\033[31m%s\033[0m\n' "$*"; }
|
||||
yellow() { printf '\033[33m%s\033[0m\n' "$*"; }
|
||||
|
||||
pass() { green " PASS: $*"; PASS=$((PASS+1)); }
|
||||
fail() { red " FAIL: $*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
# ── cluster lifecycle ─────────────────────────────────────────────────────────
|
||||
|
||||
deploy_cluster() {
|
||||
if [ ! -f "$BINARY" ]; then
|
||||
red "Binary not found: $BINARY"
|
||||
red "Run /xce-build controller first."
|
||||
exit 1
|
||||
fi
|
||||
yellow "Deploying 3-node cluster to $DEPLOY_DIR ..."
|
||||
bash "$WORKTREE/src/tools/deploy-ha.sh" \
|
||||
"$BINARY" "$DATADIR" "$DEPLOY_DIR" "$PEERS"
|
||||
}
|
||||
|
||||
start_cluster() {
|
||||
kill_cluster 2>/dev/null || true
|
||||
sleep 0.3
|
||||
VERBOSE=-vvv "$DEPLOY_DIR/scripts/start.sh" ctrl-1 \
|
||||
> "$LOGDIR/ctrl-1.log" 2>&1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv "$DEPLOY_DIR/scripts/start.sh" ctrl-2 \
|
||||
> "$LOGDIR/ctrl-2.log" 2>&1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv "$DEPLOY_DIR/scripts/start.sh" ctrl-3 \
|
||||
> "$LOGDIR/ctrl-3.log" 2>&1 &
|
||||
sleep 0.3
|
||||
}
|
||||
|
||||
kill_cluster() {
|
||||
pkill -f "xce_controller" 2>/dev/null || true
|
||||
sleep 0.5
|
||||
}
|
||||
|
||||
wait_for_primary() {
|
||||
local deadline=$(($(date +%s) + 30))
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
if grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-1.log" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_standbys() {
|
||||
# Both ctrl-2 and ctrl-3 must be in CTRL_STANDBY
|
||||
local deadline=$(($(date +%s) + 30))
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
local c2 c3
|
||||
c2=$(grep -c "CTRL_STANDBY" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0)
|
||||
c3=$(grep -c "CTRL_STANDBY" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0)
|
||||
if [ "$c2" -gt 0 ] && [ "$c3" -gt 0 ]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
ctrl1_pid() {
|
||||
pgrep -f "xce_controller.*ctrl-1" 2>/dev/null | head -1
|
||||
}
|
||||
|
||||
# ── setup ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
if [ ! -d "$DEPLOY_DIR" ]; then
|
||||
deploy_cluster
|
||||
fi
|
||||
|
||||
echo ""
|
||||
yellow "=== Task 4.5: Missed-Heartbeat Tolerance (SIGSTOP/SIGCONT) ==="
|
||||
echo ""
|
||||
|
||||
# ── Scenario A: short pause — no failover ─────────────────────────────────────
|
||||
|
||||
echo "--- Scenario A: SIGSTOP for 1 × heartbeat_interval_ms (${HB_INTERVAL_MS}ms) ---"
|
||||
echo " Expected: no failover fires on standbys"
|
||||
echo ""
|
||||
|
||||
bash "$DEPLOY_DIR/scripts/cleanup.sh"
|
||||
start_cluster
|
||||
|
||||
if ! wait_for_primary; then
|
||||
fail "Scenario A: cluster did not elect a primary within 30s"
|
||||
kill_cluster
|
||||
exit 1
|
||||
fi
|
||||
if ! wait_for_standbys; then
|
||||
fail "Scenario A: standbys did not reach CTRL_STANDBY within 30s"
|
||||
kill_cluster
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo " Cluster stable. Primary: ctrl-1"
|
||||
echo " SIGSTOP ctrl-1 for ${HB_INTERVAL_MS}ms ..."
|
||||
|
||||
PID=$(ctrl1_pid)
|
||||
if [ -z "$PID" ]; then
|
||||
fail "Scenario A: ctrl-1 process not found"
|
||||
kill_cluster
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Snapshot log lengths before pause
|
||||
PRE_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0)
|
||||
PRE_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0)
|
||||
|
||||
kill -STOP "$PID"
|
||||
sleep "$(echo "scale=1; $HB_INTERVAL_MS / 1000" | bc)s"
|
||||
kill -CONT "$PID"
|
||||
|
||||
# Give the cluster a moment to react (if it mistakenly fires failover)
|
||||
sleep 3
|
||||
|
||||
POST_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0)
|
||||
POST_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0)
|
||||
|
||||
NEW_RDHUP_2=$((POST_RDHUP_2 - PRE_RDHUP_2))
|
||||
NEW_RDHUP_3=$((POST_RDHUP_3 - PRE_RDHUP_3))
|
||||
|
||||
if [ "$NEW_RDHUP_2" -eq 0 ] && [ "$NEW_RDHUP_3" -eq 0 ]; then
|
||||
pass "Scenario A: no RDHUP fired on ctrl-2 or ctrl-3 during short pause (Req 2.2)"
|
||||
else
|
||||
fail "Scenario A: unexpected RDHUP fires (ctrl-2: +${NEW_RDHUP_2}, ctrl-3: +${NEW_RDHUP_3})"
|
||||
fi
|
||||
|
||||
# Verify cluster is still stable
|
||||
sleep 2
|
||||
if grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-1.log" && kill -0 "$PID" 2>/dev/null; then
|
||||
pass "Scenario A: ctrl-1 still primary after SIGCONT"
|
||||
else
|
||||
fail "Scenario A: ctrl-1 lost primary role unexpectedly"
|
||||
fi
|
||||
|
||||
kill_cluster
|
||||
sleep 1
|
||||
|
||||
# ── Scenario B: long pause — failover fires exactly once ─────────────────────
|
||||
|
||||
LONG_PAUSE_MS=$((HB_TIMEOUT_MS + 500))
|
||||
echo ""
|
||||
echo "--- Scenario B: SIGSTOP for heartbeat_timeout_ms + 500ms (${LONG_PAUSE_MS}ms) ---"
|
||||
echo " Expected: failover fires exactly once on each standby"
|
||||
echo ""
|
||||
|
||||
bash "$DEPLOY_DIR/scripts/cleanup.sh"
|
||||
start_cluster
|
||||
|
||||
if ! wait_for_primary; then
|
||||
fail "Scenario B: cluster did not elect a primary within 30s"
|
||||
kill_cluster
|
||||
exit 1
|
||||
fi
|
||||
if ! wait_for_standbys; then
|
||||
fail "Scenario B: standbys did not reach CTRL_STANDBY within 30s"
|
||||
kill_cluster
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Extra settle time so heartbeat clocks are steady
|
||||
sleep 5
|
||||
|
||||
echo " Cluster stable. Primary: ctrl-1"
|
||||
echo " SIGSTOP ctrl-1 for ${LONG_PAUSE_MS}ms ..."
|
||||
|
||||
PID=$(ctrl1_pid)
|
||||
if [ -z "$PID" ]; then
|
||||
fail "Scenario B: ctrl-1 process not found"
|
||||
kill_cluster
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PRE_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0)
|
||||
PRE_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0)
|
||||
PRE_ELECTING_2=$(grep -c "CTRL_ELECTING" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0)
|
||||
PRE_ELECTING_3=$(grep -c "CTRL_ELECTING" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0)
|
||||
|
||||
kill -STOP "$PID"
|
||||
sleep "$(echo "scale=1; $LONG_PAUSE_MS / 1000" | bc)s"
|
||||
|
||||
# Do NOT SIGCONT: leave ctrl-1 stopped so it cannot win re-election.
|
||||
# Standbys should detect the silence and start failover.
|
||||
|
||||
# Wait for failover to complete (timeout + election + committing + margin)
|
||||
FAILOVER_WAIT=$(( (HB_TIMEOUT_MS + 5000 + 10000 + 5000) / 1000 ))
|
||||
echo " Waiting up to ${FAILOVER_WAIT}s for failover ..."
|
||||
deadline=$(($(date +%s) + FAILOVER_WAIT))
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
NEW_ELECTING=$(( \
|
||||
$(grep -c "CTRL_ELECTING" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0) \
|
||||
- PRE_ELECTING_2 \
|
||||
))
|
||||
if [ "$NEW_ELECTING" -gt 0 ]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Now SIGCONT ctrl-1 (it should lose since a new primary was elected)
|
||||
kill -CONT "$PID" 2>/dev/null || true
|
||||
|
||||
# Collect final counts
|
||||
sleep 5
|
||||
POST_RDHUP_2=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-2.log" 2>/dev/null || echo 0)
|
||||
POST_RDHUP_3=$(grep -c "ha_on_primary_ctrl_fd_rdhup" "$LOGDIR/ctrl-3.log" 2>/dev/null || echo 0)
|
||||
|
||||
NEW_RDHUP_2=$((POST_RDHUP_2 - PRE_RDHUP_2))
|
||||
NEW_RDHUP_3=$((POST_RDHUP_3 - PRE_RDHUP_3))
|
||||
|
||||
# Each standby should show exactly 1 new RDHUP (not 0, not >1)
|
||||
if [ "$NEW_RDHUP_2" -eq 1 ]; then
|
||||
pass "Scenario B: ctrl-2 fired exactly 1 RDHUP during long pause (Req 2.3)"
|
||||
elif [ "$NEW_RDHUP_2" -eq 0 ]; then
|
||||
fail "Scenario B: ctrl-2 fired 0 RDHUP — liveness poll did not trigger failover"
|
||||
else
|
||||
fail "Scenario B: ctrl-2 fired ${NEW_RDHUP_2} RDHUPs — duplicate failover loop (Req 5.4)"
|
||||
fi
|
||||
|
||||
if [ "$NEW_RDHUP_3" -eq 1 ]; then
|
||||
pass "Scenario B: ctrl-3 fired exactly 1 RDHUP during long pause (Req 2.3)"
|
||||
elif [ "$NEW_RDHUP_3" -eq 0 ]; then
|
||||
fail "Scenario B: ctrl-3 fired 0 RDHUP — liveness poll did not trigger failover"
|
||||
else
|
||||
fail "Scenario B: ctrl-3 fired ${NEW_RDHUP_3} RDHUPs — duplicate failover loop (Req 5.4)"
|
||||
fi
|
||||
|
||||
# One of ctrl-2 or ctrl-3 must have become PRIMARY
|
||||
NEW_PRIMARY=""
|
||||
if grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-2.log" 2>/dev/null; then
|
||||
NEW_PRIMARY="ctrl-2"
|
||||
elif grep -q "CTRL_PRIMARY" "$LOGDIR/ctrl-3.log" 2>/dev/null; then
|
||||
NEW_PRIMARY="ctrl-3"
|
||||
fi
|
||||
|
||||
if [ -n "$NEW_PRIMARY" ]; then
|
||||
pass "Scenario B: $NEW_PRIMARY promoted to CTRL_PRIMARY after failover"
|
||||
else
|
||||
fail "Scenario B: no standby reached CTRL_PRIMARY — election did not complete"
|
||||
fi
|
||||
|
||||
kill_cluster
|
||||
|
||||
# ── summary ───────────────────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "========================="
|
||||
echo "Results: ${PASS} passed, ${FAIL} failed"
|
||||
echo "========================="
|
||||
|
||||
[ "$FAIL" -eq 0 ] && exit 0 || exit 1
|
||||
Loading…
Add table
Add a link
Reference in a new issue