myxce/.kiro/specs/ha-config-bootstrap/manual-tests.md

# Manual Integration Tests: ha-config-bootstrap

This document describes how to run the integration tests for tasks 11.1–11.5.
Run each test, record the output, and paste the filled **Results Template** at
the bottom back to the developer.

## Environment

```
Worktree:   /home/rshen/projects/xce/worktrees/persistent_ha
Build dir:  /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
Controller: src/xce.build/controller/xce_controller
CLI:        src/xce.build/frontend/xcecli/xce   (or /tmp/xce_static after make build-cli)
Unit tests: src/xce.build/controller/controller_unit_test
            src/xce.build/lib/xcelib_test
```

## Build First

```bash
# From the worktree
/xce-build           # full dev build (preferred: uses nix develop + meson)

# Or manually from the src directory:
nix develop /home/rshen/projects/xce/xce/src --command bash -c \
  "cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja"
```

## Unit Tests (Sanity Check Before Integration)

```bash
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>/dev/null | tail -5
./lib/xcelib_test      2>/dev/null | tail -3
```

Expected: all tests PASS, no FAIL lines, exit 0.

---

## Test Infrastructure

> **Design constraint:** The controller rejects an even number of peer nodes
> (`--peers` list), because Raft quorum requires an odd cluster size (1, 3, 5…)
> to guarantee a safe majority. **All tests use a 3-node cluster.**

All tests use temporary work directories. Each controller node needs:
- A unique `--myname` (`ctrl-a`, `ctrl-b`, `ctrl-c`; index 0 = highest priority)
- Shared `--peers` list (`name=host:port,...`, **odd count only**)
- A `--workdir` for persistent state (`ha-term.bin`, `data/wal/`, `data/snap/`)
- A `--logfile` to capture output
- **TLS certificates** at `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key`
  (loaded automatically from that fixed path — no CLI flag required)

### One-time setup

```bash
BINARY=/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/controller/xce_controller
PEERS="ctrl-a=127.0.0.1:9901,ctrl-b=127.0.0.1:9902,ctrl-c=127.0.0.1:9903"

mkdir -p /tmp/xce-{a,b,c}/certs /tmp/xce-{a,b,c}/data
```

### TLS Certificate Setup (required)

The controller loads `{workdir}/certs/xce.crt` and `{workdir}/certs/xce.key`
automatically. Generate one self-signed cert shared by all three nodes:

```bash
openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/xce.key \
  -out    /tmp/xce.crt \
  -days 3650 \
  -subj "/CN=xce-controller" \
  -addext "subjectAltName=IP:127.0.0.1,DNS:localhost"

for node in a b c; do
  cp /tmp/xce.crt /tmp/xce-${node}/certs/xce.crt
  cp /tmp/xce.key /tmp/xce-${node}/certs/xce.key
done
```

> All nodes share the same self-signed cert so they mutually trust each other.
> In production each node has its own cert signed by a shared CA.

### Initial Cluster Configuration (primary node only)

Thanks to config replication, **only the primary node (ctrl-a) needs the initial
config file.** Nodes B and C start with `config_tag=0`, defer the config fetch,
yield the election to the primary, and automatically receive the config via
`push_config_to_standbys()` after catch-up.

The controller looks for config at:
```
{workdir}/{myname}/conf/current                    ← pointer file
{workdir}/{myname}/conf/cluster.bin.<hex_tag>      ← capnp config blob
```

The `<hex_tag>` is the first **12 hex chars** of the SHA-256 of the config blob
(= first 6 bytes). The controller validates exactly 12 lowercase hex characters.

**Install config on Node A only:**

```bash
TAG=$(sha256sum /path/to/your_config.bin | awk '{print $1}' | cut -c1-12)
echo "Config tag: $TAG"

CONF_DIR="/tmp/xce-a/ctrl-a/conf"
mkdir -p "$CONF_DIR"
cp /path/to/your_config.bin "${CONF_DIR}/cluster.bin.${TAG}"
printf "cluster.bin.${TAG}" > "${CONF_DIR}/current"
```

> `printf` (not `echo`) avoids a trailing newline that would corrupt the pointer.
>
> Nodes B and C need **no** config pre-installed — they will receive it from
> the primary automatically after election completes.

**Verify before starting (common mistake: wrong subdirectory):**

```bash
# The path MUST include the node name as a subdirectory:
#   {workdir}/{myname}/conf/current  ✓  /tmp/xce-a/ctrl-a/conf/current
#   {workdir}/conf/current           ✗  /tmp/xce-a/conf/current  ← WRONG

ls -la /tmp/xce-a/ctrl-a/conf/
# Must show: current  cluster.bin.<tag>
```

If you see the files at `/tmp/xce-a/conf/` instead, move them:
```bash
mkdir -p /tmp/xce-a/ctrl-a/conf
mv /tmp/xce-a/conf/* /tmp/xce-a/ctrl-a/conf/
```

### Start the 3-node cluster (3 terminals)

**Terminal 1 — Node A (priority 0 = highest):**
```bash
$BINARY --myname ctrl-a --peers "$PEERS" \
        --workdir /tmp/xce-a --logfile /tmp/xce-a/ctrl.log --vcpus 2
```

**Terminal 2 — Node B:**
```bash
$BINARY --myname ctrl-b --peers "$PEERS" \
        --workdir /tmp/xce-b --logfile /tmp/xce-b/ctrl.log --vcpus 2
```

**Terminal 3 — Node C:**
```bash
$BINARY --myname ctrl-c --peers "$PEERS" \
        --workdir /tmp/xce-c --logfile /tmp/xce-c/ctrl.log --vcpus 2
```

**Watch logs (Terminal 4):**
```bash
tail -f /tmp/xce-{a,b,c}/ctrl.log | grep -E "CTRL_|election|quorum|RDHUP|ha-term|heartbeat"
```

---

## Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication

**Requirements:** 1.1, 2.1, 2.7, 3.2, 3.5

**Scenario:** ctrl-a has the initial config; ctrl-b and ctrl-c start with
`config_tag=0` (no config). All three start within 3s — ctrl-a wins election
(freshest config), then pushes the config to ctrl-b and ctrl-c via replication.

This test verifies election AND config bootstrap replication in a single run.

1. Start all 3 nodes (Terminals 1–3) within 3 seconds of each other.
2. Wait 15 seconds.
3. Check the logs on all three.

**Expected log patterns — Election:**
- ctrl-a logs: `am_i_primary_quorum: won` → `CTRL_PRIMARY`
- ctrl-b and ctrl-c log: `defer_config_fetch=1` → yield to fresher peer → `CTRL_STANDBY`

**Expected log patterns — Config replication:**
- ctrl-a logs: `Config committed on majority (tag=<hex>...)`
- ctrl-b and ctrl-c log: `Config fetch successful from primary (tag=<hex>...)`
- Config files appear in both standby workdirs:
  ```bash
  ls /tmp/xce-b/ctrl-b/conf/
  ls /tmp/xce-c/ctrl-c/conf/
  # Expected: cluster.bin.<hex_tag> and current
  ```

**Checklist:**
- [ ] Exactly ctrl-a becomes `CTRL_PRIMARY`
- [ ] ctrl-b and ctrl-c become `CTRL_STANDBY`
- [ ] No crash or ASAN error
- [ ] All three nodes started within 3s (no false self-election)
- [ ] Config replicated to ctrl-b: `cluster.bin.*` exists in `/tmp/xce-b/ctrl-b/conf/`
- [ ] Config replicated to ctrl-c: `cluster.bin.*` exists in `/tmp/xce-c/ctrl-c/conf/`
- [ ] Config tag on standbys matches primary's tag

---

## Test 11.2 — Failover and WAL Durability

**Requirements:** 5.6, 6.2, 5.5

**Prerequisites:** 3-node cluster running from Test 11.1.

### Scenario 1: Primary crash → RDHUP → one standby promotes

1. Identify which node is primary from the logs.
2. Kill the primary process: `kill -9 <pid>` or Ctrl-C.
3. Watch the two surviving standbys.

**Expected within 1 second on both standbys:**
```
HA: primary_ctrl_fd RDHUP → CTRL_ELECTING
```
**Expected on the winner (needs 1 vote from the other surviving node):**
```
am_i_primary_quorum: won
CTRL_PRIMARY
```

### Scenario 2: WAL commit with both standbys down → ETIMEDOUT

1. Kill both standbys (leaving only the primary).
2. Submit a job:
```bash
/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \
  --server 127.0.0.1:9901 jobsubmit -c "hostname"
```

**Expected in primary log:** `repl_wait_majority_ack: timeout` — NOT a crash.

### Scenario 3: Reconnected standby catches up

1. Restart one of the killed standbys.
2. Watch logs on the restarted node.

**Expected:**
```
CtrlHelloReply from primary: committed_lsn=...
CTRL_STANDBY
```

**Checklist:**
- [ ] RDHUP failover detected within 1s of primary kill
- [ ] One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other)
- [ ] WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode)
- [ ] Restarted standby reconnects and enters CTRL_STANDBY

---

## Test 11.3 — Unconfigured Node Bootstrap

**Requirements:** 3.2, 3.5

**Scenario:** Node C starts with no local cluster config (config_tag=0); Nodes A and B have a valid config.

1. Start Nodes A and B first (let one become primary).
2. Remove any existing config from Node C's workdir:
```bash
rm -f /tmp/xce-c/cluster.bin /tmp/xce-c/data/current 2>/dev/null
```
3. Start Node C without a config file (just `--peers`, no config).

**Expected in Node C log:**
```
defer_config_fetch=1
Config fetch successful from primary (tag=...)
CTRL_STANDBY
```

**Expected in `/tmp/xce-c/`:** a `cluster.bin` or `current` pointer file appears.

**Checklist:**
- [ ] Node C detects config_tag=0 and sets defer_config_fetch
- [ ] Node C yields election to the existing primary (fresher config)
- [ ] Node C successfully fetches config from primary
- [ ] Node C enters CTRL_STANDBY with correct config

---

## Test 11.4 — Term Durability Across Restart

**Requirements:** 9.1, 9.2

**Scenario:** After election, `ha-term.bin` is written; on restart, term is restored.

1. Run 3-node cluster from Test 11.1, wait for election to complete.
2. Check that `ha-term.bin` was written on the node that granted a vote:
```bash
xxd /tmp/xce-a/ha-term.bin | head -4
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0
```
3. Kill Node A.
4. Restart Node A.
5. Check startup log for term restore.

**Expected in startup log:**
```
HA: restored term=N from /tmp/xce-a/ha-term.bin
```

**Checklist:**
- [ ] `ha-term.bin` file exists after election
- [ ] Magic bytes are `0xDEADBEEFCAFEBABE` (little-endian in `xxd`)
- [ ] Term field (bytes 8–15) is > 0 after a vote was granted
- [ ] On restart: log shows `restored term=N`
- [ ] Node re-joins correctly without double-voting

---

## Test 11.5 — Full ASAN Build Verification

**Requirements:** 4

**Steps:**

```bash
# Run unit tests (already built with ASAN via /xce-build)
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
./lib/xcelib_test      2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
```

**Expected output:** No lines (zero ASAN errors introduced by this feature).

Pre-existing leaks from `test_worker_join` in controller_unit_test are acceptable —
they existed before this branch and are not new.

```bash
# Count pre-existing vs new leaks:
./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l
```

**Checklist:**
- [ ] `controller_unit_test` exits 0, all tests PASS
- [ ] `xcelib_test` exits 0, all tests PASS
- [ ] Zero `AddressSanitizer` errors (use-after-free, buffer overflow)
- [ ] No new `LeakSanitizer` leaks from ha-config-bootstrap code paths

---

## Results Template

Copy, fill in, and paste back:

```
## Test Run Results

Date:
Commit: (run: cd /home/rshen/projects/xce/worktrees/persistent_ha && git log -1 --oneline)
Build: /xce-build completed successfully? YES / NO

### Unit Tests
- controller_unit_test: PASS / FAIL
- xcelib_test: PASS / FAIL

### 11.1 Concurrent Startup + Config Replication (3 nodes)
- Election result: PASS / FAIL
  - Primary elected: ctrl-a / ctrl-b / ctrl-c
  - Time to election: ~Xs
  - Two standbys confirmed: YES / NO
- Config replication result: PASS / FAIL
  - cluster.bin exists in /tmp/xce-b/ctrl-b/conf/: YES / NO
  - cluster.bin exists in /tmp/xce-c/ctrl-c/conf/: YES / NO
  - Config tag on standbys matches primary: YES / NO
- Log excerpt (primary win + config push + standby fetch):

### 11.2 Failover and WAL Durability (3 nodes)
- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS / FAIL
  - Time to detect RDHUP: ~Xs
  - New primary: ctrl-a / ctrl-b / ctrl-c
  - Log excerpt:
- Scenario 2 (degraded WAL timeout, both standbys down): PASS / FAIL
  - Log excerpt:
- Scenario 3 (reconnect one standby): PASS / FAIL

### 11.3 Unconfigured Node Bootstrap (ctrl-c joins ctrl-a+ctrl-b)
- Result: PASS / FAIL
- cluster.bin created in /tmp/xce-c/? YES / NO
- Log excerpt:

### 11.4 Term Durability
- ha-term.bin exists: YES / NO
- xxd output (first 4 lines):
- Restart log (term restore line):
- Result: PASS / FAIL

### 11.5 ASAN Build
- controller_unit_test ASAN errors: N
- xcelib_test ASAN errors: N
- Result: PASS / FAIL

### Additional Notes / Unexpected Behaviour
```