myxce/.kiro/specs/ha-config-bootstrap/manual-tests.md

Manual Integration Tests: ha-config-bootstrap

This document describes how to run the integration tests for tasks 11.1–11.5. Run each test, record the output, and paste the filled Results Template at the bottom back to the developer.

Environment

Worktree:   /home/rshen/projects/xce/worktrees/persistent_ha
Build dir:  /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
Controller: src/xce.build/controller/xce_controller
CLI:        src/xce.build/frontend/xcecli/xce   (or /tmp/xce_static after make build-cli)
Unit tests: src/xce.build/controller/controller_unit_test
            src/xce.build/lib/xcelib_test

Build First

# From the worktree
/xce-build           # full dev build (preferred: uses nix develop + meson)

# Or manually from the src directory:
nix develop /home/rshen/projects/xce/xce/src --command bash -c \
  "cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja"

Unit Tests (Sanity Check Before Integration)

cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>/dev/null | tail -5
./lib/xcelib_test      2>/dev/null | tail -3

Expected: all tests PASS, no FAIL lines, exit 0.

---

Test Infrastructure

Design constraint: The controller rejects an even number of peer nodes (--peers list), because Raft quorum requires an odd cluster size (1, 3, 5…) to guarantee a safe majority. All tests use a 3-node cluster.

All tests use temporary work directories. Each controller node needs:

• A unique --myname (ctrl-a, ctrl-b, ctrl-c; index 0 = highest priority)
• Shared --peers list (name=host:port,..., odd count only)
• A --workdir for persistent state (ha-term.bin, data/wal/, data/snap/)
• A --logfile to capture output
• TLS certificates at {workdir}/certs/xce.crt and {workdir}/certs/xce.key (loaded automatically from that fixed path — no CLI flag required)

One-time setup

BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller
PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902,ctrl-c=127.0.0.1:9903"

mkdir -p /tmp/xce-{a,b,c}/certs /tmp/xce-{a,b,c}/data

TLS Certificate Setup (required)

The controller loads {workdir}/certs/xce.crt and {workdir}/certs/xce.key automatically. Generate one self-signed cert shared by all three nodes:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/xce.key \
  -out    /tmp/xce.crt \
  -days 3650 \
  -subj "/CN=xce-controller" \
  -addext "subjectAltName=IP:127.0.0.1,DNS:localhost"

for node in a b c; do
  cp /tmp/xce.crt /tmp/xce-${node}/certs/xce.crt
  cp /tmp/xce.key /tmp/xce-${node}/certs/xce.key
done

All nodes share the same self-signed cert so they mutually trust each other. In production each node has its own cert signed by a shared CA.

Initial Cluster Configuration (primary node only)

Thanks to config replication, only the primary node (ctrl-a) needs the initial config file. Nodes B and C start with config_tag=0, defer the config fetch, yield the election to the primary, and automatically receive the config via push_config_to_standbys() after catch-up.

The controller looks for config at:

{workdir}/{myname}/conf/current                    ← pointer file
{workdir}/{myname}/conf/cluster.bin.<hex_tag>      ← capnp config blob

The <hex_tag> is the first 12 hex chars of the SHA-256 of the config blob (= first 6 bytes). The controller validates exactly 12 lowercase hex characters.

Install config on Node A only:

TAG=$(sha256sum /path/to/your_config.bin | awk '{print $1}' | cut -c1-12)
echo "Config tag: $TAG"

CONF_DIR="/tmp/xce-a/ctrl-a/conf"
mkdir -p "$CONF_DIR"
cp /path/to/your_config.bin "${CONF_DIR}/cluster.bin.${TAG}"
printf "cluster.bin.${TAG}" > "${CONF_DIR}/current"

printf (not echo) avoids a trailing newline that would corrupt the pointer.

Nodes B and C need no config pre-installed — they will receive it from the primary automatically after election completes.

Verify before starting (common mistake: wrong subdirectory):

# The path MUST include the node name as a subdirectory:
#   {workdir}/{myname}/conf/current  ✓  /tmp/xce-a/ctrl-a/conf/current
#   {workdir}/conf/current           ✗  /tmp/xce-a/conf/current  ← WRONG

ls -la /tmp/xce-a/ctrl-a/conf/
# Must show: current  cluster.bin.<tag>

If you see the files at /tmp/xce-a/conf/ instead, move them:

mkdir -p /tmp/xce-a/ctrl-a/conf
mv /tmp/xce-a/conf/* /tmp/xce-a/ctrl-a/conf/

Start the 3-node cluster (3 terminals)

Terminal 1 — Node A (priority 0 = highest):

$BINARY --myname ctrl-a --peers "$PEERS" \
        --workdir /tmp/xce-a --logfile /tmp/xce-a/ctrl.log --vcpus 2

Terminal 2 — Node B:

$BINARY --myname ctrl-b --peers "$PEERS" \
        --workdir /tmp/xce-b --logfile /tmp/xce-b/ctrl.log --vcpus 2

Terminal 3 — Node C:

$BINARY --myname ctrl-c --peers "$PEERS" \
        --workdir /tmp/xce-c --logfile /tmp/xce-c/ctrl.log --vcpus 2

Watch logs (Terminal 4):

tail -f /tmp/xce-{a,b,c}/ctrl.log | grep -E "CTRL_|election|quorum|RDHUP|ha-term|heartbeat"

---

Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication

Requirements: 1.1, 2.1, 2.7, 3.2, 3.5

Scenario: ctrl-a has the initial config; ctrl-b and ctrl-c start with config_tag=0 (no config). All three start within 3s — ctrl-a wins election (freshest config), then pushes the config to ctrl-b and ctrl-c via replication.

This test verifies election AND config bootstrap replication in a single run.

1. Start all 3 nodes (Terminals 1–3) within 3 seconds of each other.
2. Wait 15 seconds.
3. Check the logs on all three.

Expected log patterns — Election:

• ctrl-a logs: am_i_primary_quorum: won → CTRL_PRIMARY
• ctrl-b and ctrl-c log: defer_config_fetch=1 → yield to fresher peer → CTRL_STANDBY

Expected log patterns — Config replication:

• ctrl-a logs: Config committed on majority (tag=<hex>...)
• ctrl-b and ctrl-c log: Config fetch successful from primary (tag=<hex>...)
• Config files appear in both standby workdirs:

  ls /tmp/xce-b/ctrl-b/conf/
  ls /tmp/xce-c/ctrl-c/conf/
  # Expected: cluster.bin.<hex_tag> and current
  

Checklist:

• Exactly ctrl-a becomes CTRL_PRIMARY
• ctrl-b and ctrl-c become CTRL_STANDBY
• No crash or ASAN error
• All three nodes started within 3s (no false self-election)
• Config replicated to ctrl-b: cluster.bin.* exists in /tmp/xce-b/ctrl-b/conf/
• Config replicated to ctrl-c: cluster.bin.* exists in /tmp/xce-c/ctrl-c/conf/
• Config tag on standbys matches primary's tag

---

Test 11.2 — Failover and WAL Durability

Requirements: 5.6, 6.2, 5.5

Prerequisites: 3-node cluster running from Test 11.1.

Scenario 1: Primary crash → RDHUP → one standby promotes

1. Identify which node is primary from the logs.
2. Kill the primary process: kill -9 <pid> or Ctrl-C.
3. Watch the two surviving standbys.

Expected within 1 second on both standbys:

HA: primary_ctrl_fd RDHUP → CTRL_ELECTING

Expected on the winner (needs 1 vote from the other surviving node):

am_i_primary_quorum: won
CTRL_PRIMARY

Scenario 2: WAL commit with both standbys down → ETIMEDOUT

1. Kill both standbys (leaving only the primary).
2. Submit a job:

/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \
  --server 127.0.0.1:9901 jobsubmit -c "hostname"

Expected in primary log: repl_wait_majority_ack: timeout — NOT a crash.

Scenario 3: Reconnected standby catches up

1. Restart one of the killed standbys.
2. Watch logs on the restarted node.

Expected:

CtrlHelloReply from primary: committed_lsn=...
CTRL_STANDBY

Checklist:

• RDHUP failover detected within 1s of primary kill
• One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other)
• WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode)
• Restarted standby reconnects and enters CTRL_STANDBY

---

Test 11.3 — Unconfigured Node Bootstrap

Requirements: 3.2, 3.5

Scenario: Node C starts with no local cluster config (config_tag=0); Nodes A and B have a valid config.

1. Start Nodes A and B first (let one become primary).
2. Remove any existing config from Node C's workdir:

rm -f /tmp/xce-c/cluster.bin /tmp/xce-c/data/current 2>/dev/null

3. Start Node C without a config file (just --peers, no config).

Expected in Node C log:

defer_config_fetch=1
Config fetch successful from primary (tag=...)
CTRL_STANDBY

Expected in /tmp/xce-c/: a cluster.bin or current pointer file appears.

Checklist:

• Node C detects config_tag=0 and sets defer_config_fetch
• Node C yields election to the existing primary (fresher config)
• Node C successfully fetches config from primary
• Node C enters CTRL_STANDBY with correct config

---

Test 11.4 — Term Durability Across Restart

Requirements: 9.1, 9.2

Scenario: After election, ha-term.bin is written; on restart, term is restored.

1. Run 3-node cluster from Test 11.1, wait for election to complete.
2. Check that ha-term.bin was written on the node that granted a vote:

xxd /tmp/xce-a/ha-term.bin | head -4
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0

3. Kill Node A.
4. Restart Node A.
5. Check startup log for term restore.

Expected in startup log:

HA: restored term=N from /tmp/xce-a/ha-term.bin

Checklist:

• ha-term.bin file exists after election
• Magic bytes are 0xDEADBEEFCAFEBABE (little-endian in xxd)
• Term field (bytes 8–15) is > 0 after a vote was granted
• On restart: log shows restored term=N
• Node re-joins correctly without double-voting

---

Test 11.5 — Full ASAN Build Verification

Requirements: 4

Steps:

# Run unit tests (already built with ASAN via /xce-build)
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
./lib/xcelib_test      2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'

Expected output: No lines (zero ASAN errors introduced by this feature).

Pre-existing leaks from test_worker_join in controller_unit_test are acceptable — they existed before this branch and are not new.

# Count pre-existing vs new leaks:
./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l

Checklist:

• controller_unit_test exits 0, all tests PASS
• xcelib_test exits 0, all tests PASS
• Zero AddressSanitizer errors (use-after-free, buffer overflow)
• No new LeakSanitizer leaks from ha-config-bootstrap code paths

---

Results Template

Copy, fill in, and paste back:

## Test Run Results

Date:
Commit: (run: cd /home/rshen/projects/xce/worktrees/persistent_ha && git log -1 --oneline)
Build: /xce-build completed successfully? YES / NO

### Unit Tests
- controller_unit_test: PASS / FAIL
- xcelib_test: PASS / FAIL

### 11.1 Concurrent Startup + Config Replication (3 nodes)
- Election result: PASS / FAIL
  - Primary elected: ctrl-a / ctrl-b / ctrl-c
  - Time to election: ~Xs
  - Two standbys confirmed: YES / NO
- Config replication result: PASS / FAIL
  - cluster.bin exists in /tmp/xce-b/ctrl-b/conf/: YES / NO
  - cluster.bin exists in /tmp/xce-c/ctrl-c/conf/: YES / NO
  - Config tag on standbys matches primary: YES / NO
- Log excerpt (primary win + config push + standby fetch):

### 11.2 Failover and WAL Durability (3 nodes)
- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS / FAIL
  - Time to detect RDHUP: ~Xs
  - New primary: ctrl-a / ctrl-b / ctrl-c
  - Log excerpt:
- Scenario 2 (degraded WAL timeout, both standbys down): PASS / FAIL
  - Log excerpt:
- Scenario 3 (reconnect one standby): PASS / FAIL

### 11.3 Unconfigured Node Bootstrap (ctrl-c joins ctrl-a+ctrl-b)
- Result: PASS / FAIL
- cluster.bin created in /tmp/xce-c/? YES / NO
- Log excerpt:

### 11.4 Term Durability
- ha-term.bin exists: YES / NO
- xxd output (first 4 lines):
- Restart log (term restore line):
- Result: PASS / FAIL

### 11.5 ASAN Build
- controller_unit_test ASAN errors: N
- xcelib_test ASAN errors: N
- Result: PASS / FAIL

### Additional Notes / Unexpected Behaviour