myxce/.kiro/specs/ha-config-bootstrap/manual-tests.md

# Manual Integration Tests: ha-config-bootstrap

This document describes how to run the integration tests for tasks 11.1–11.5.
Run each test, record the output, and paste the filled **Results Template** at
the bottom back to the developer.

## Environment

```
Worktree:   /home/rshen/projects/xce/worktrees/persistent_ha
Build dir:  /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
Controller: src/xce.build/controller/xce_controller
CLI:        src/xce.build/frontend/xcecli/xce   (or /tmp/xce_static after make build-cli)
Unit tests: src/xce.build/controller/controller_unit_test
            src/xce.build/lib/xcelib_test
```

## Build First

```bash
# From the worktree
/xce-build           # full dev build (preferred: uses nix develop + meson)

# Or manually from the src directory:
nix develop /home/rshen/projects/xce/xce/src --command bash -c \
  "cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja"
```

## Unit Tests (Sanity Check Before Integration)

```bash
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>/dev/null | tail -5
./lib/xcelib_test      2>/dev/null | tail -3
```

Expected: all tests PASS, no FAIL lines, exit 0.

---

## Test Infrastructure

> **Design constraint:** The controller rejects an even number of peer nodes
> (`--peers` list), because Raft quorum requires an odd cluster size (1, 3, 5…)
> to guarantee a safe majority. **All tests use a 3-node cluster.**

### Deploy via `/xce-deploy-ha` skill

All tests use the `/xce-deploy-ha` skill which automates deployment:

```bash
# 1. Build (if not already built)
/xce-build controller

# 2. Deploy 3-node cluster
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
```

This deploys to **`/tmp/xce-deploy/`** with:
- `certs/` — shared TLS cert and key (copied from `ha/data/`)
- `controller/ctrl-1/conf/` — initial config (primary only; standbys receive via push)
- `controller/ctrl-{1,2,3}/data/` — WAL, snapshots, `ha-term.bin`
- `logs/ctrl-{1,2,3}.log` — per-node log files
- `scripts/start.sh` and `scripts/cleanup.sh` — lifecycle helpers

Nodes are named `ctrl-1`, `ctrl-2`, `ctrl-3` (index 0 = highest priority = ctrl-1).

### Start and verify

```bash
# Start all 3 (staggered by 0.5s)
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &

# Wait 15s for election + config replication, then verify
grep -i "CTRL_PRIMARY\|raft.*leader\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
find /tmp/xce-deploy/controller/ctrl-2/conf /tmp/xce-deploy/controller/ctrl-3/conf -type f | sort
```

### Clean restart (preserves config, wipes data)

```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
bash /tmp/xce-deploy/scripts/cleanup.sh
# Then restart nodes as above
```

**What cleanup.sh preserves:** `controller/ctrl-1/conf/*`, `$TOP/certs/`
**What cleanup.sh removes:** all `controller/ctrl-{1..N}/data/` dirs, `controller/ctrl-{2..N}/conf/*`, all `logs/*.log`

### Manual restart with config preserved (for failover testing)

To restart without wiping standby configs (so standbys retain `config_tag > 0`):

```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
rm -rf /tmp/xce-deploy/controller/ctrl-{1,2,3}/data /tmp/xce-deploy/logs/*.log
mkdir -p /tmp/xce-deploy/controller/ctrl-{1,2,3}/data
# Then restart nodes as above
```

---

## Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication

**Requirements:** 1.1, 2.1, 2.7, 3.2, 3.5

**Scenario:** ctrl-1 has the initial config; ctrl-2 and ctrl-3 start with
`config_tag=0` (no config). All three start concurrently — ctrl-1 wins election
(freshest config), then pushes the config to ctrl-2 and ctrl-3 via replication.

**Steps:** Deploy and start as described in Test Infrastructure above.

**Expected log patterns — Election (current implementation):**
- ctrl-1: `raft_cb_become_leader` → `CTRL_PRIMARY_COMMITTING` → `CTRL_PRIMARY`
- ctrl-2/ctrl-3: `myapp_run: CTRL_STANDBY — waiting for primary to connect and push config`

**Expected log patterns — Config replication (primary push model):**
- ctrl-1: `Config committed on majority (tag=<hex>, pushed=2/2 standbys)`
- ctrl-2/ctrl-3: `handle_repl_config_push: config saved and loaded (tag=<hex>)`
- Config files appear in both standby workdirs:
  ```bash
  ls /tmp/xce-deploy/controller/ctrl-2/conf/
  ls /tmp/xce-deploy/controller/ctrl-3/conf/
  # Expected: cluster.bin.<hex_tag> and current
  ```

**Checklist:**
- [ ] Exactly ctrl-1 becomes `CTRL_PRIMARY`
- [ ] ctrl-2 and ctrl-3 become `CTRL_STANDBY`
- [ ] No crash or ASAN error
- [ ] Config replicated to ctrl-2: `cluster.bin.*` exists in `/tmp/xce-deploy/controller/ctrl-2/conf/`
- [ ] Config replicated to ctrl-3: `cluster.bin.*` exists in `/tmp/xce-deploy/controller/ctrl-3/conf/`
- [ ] Config tag on standbys matches primary's tag

---

## Test 11.2 — Failover and WAL Durability

**Requirements:** 5.6, 6.2, 5.5

**Prerequisites:** 3-node cluster running from Test 11.1.

### Scenario 1: Primary crash → liveness timeout → re-election

1. Identify which node is primary: `grep CTRL_PRIMARY /tmp/xce-deploy/logs/ctrl-*.log`
2. Kill the primary process: `kill -9 <pid>`.
3. Watch the two surviving standbys: `tail -f /tmp/xce-deploy/logs/ctrl-{2,3}.log`

**Expected on ctrl-2 and ctrl-3 (both standbys, within ~10s of primary kill):**
```
HA: liveness poll: no primary activity for 10s → failover
HA: primary_ctrl_fd RDHUP → CTRL_ELECTING
```
**Expected on the winner (standby with config_tag > 0 campaigns and wins):**
```
raft_cb_become_leader
CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY
```

> **Note (2026-06-03):** Failover election requires standbys to have a
> non-zero `config_tag` loaded at startup. After the initial deployment,
> standbys receive config via `handle_repl_config_push`. On subsequent
> restarts, if `cleanup.sh` wipes `ctrl-{2,3}/conf/`, the standbys restart
> with `config_tag=0` and participate as voters only (they will not campaign).
> Preserve `ctrl-{2,3}/conf/` across restarts for failover to complete.
> See test variant below in results.

### Scenario 2: WAL commit with both standbys down → ETIMEDOUT

1. Kill both standbys (leaving only the primary).
2. Submit a job:
```bash
/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \
  --server 127.0.0.1:9901 jobsubmit -c "hostname"
```

**Expected in primary log:** `repl_wait_majority_ack: timeout` — NOT a crash.

### Scenario 3: Reconnected standby catches up

1. Restart one of the killed standbys.
2. Watch logs on the restarted node.

**Expected:**
```
CtrlHelloReply from primary: committed_lsn=...
CTRL_STANDBY
```

**Checklist:**
- [ ] RDHUP failover detected within 1s of primary kill
- [ ] One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other)
- [ ] WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode)
- [ ] Restarted standby reconnects and enters CTRL_STANDBY

---

## Test 11.3 — Unconfigured Node Bootstrap

**Requirements:** 3.2, 3.5

**Scenario:** ctrl-3 starts with no local cluster config (config_tag=0); ctrl-1 and ctrl-2 already hold a valid config from Test 11.1.

1. Start ctrl-1 and ctrl-2 first (let one become primary).
2. Ensure ctrl-3 has no conf: `rm -rf /tmp/xce-deploy/controller/ctrl-3/conf/* 2>/dev/null` (or rely on fresh deploy where only ctrl-1 has conf).
3. Start ctrl-3.

**Expected in ctrl-3 log:**
```
ha_start_election_async: no local config (tag=0, n_peers=3)
myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
handle_repl_config_push: config saved and loaded (tag=...)
```

**Expected in `/tmp/xce-deploy/controller/ctrl-3/conf/`:** a `cluster.bin.<tag>` and `current` pointer file appear after primary push.

**Checklist:**
- [ ] ctrl-3 detects config_tag=0, participates as voter only
- [ ] ctrl-3 yields election to the existing primary (fresher config)
- [ ] ctrl-3 receives config via `handle_repl_config_push` from primary
- [ ] ctrl-3 enters CTRL_STANDBY with correct config
- [ ] Config files appear in `/tmp/xce-deploy/controller/ctrl-3/conf/` after push

---

## Test 11.4 — Term Durability Across Restart

**Requirements:** 9.1, 9.2

**Scenario:** After election, `ha-term.bin` is written; on restart, term is restored.

1. Run 3-node cluster from Test 11.1, wait for election to complete.
2. Check that `ha-term.bin` was written on the primary:
```bash
xxd /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin | head -4
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0
```
3. Kill ctrl-1.
4. Restart ctrl-1: `VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &`
5. Check startup log for term restore.

**Expected in startup log:**
```
HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
```
(See `cluster_init_phase1` log line)

**Checklist:**
- [ ] `ha-term.bin` file exists in `/tmp/xce-deploy/controller/ctrl-1/data/` after election
- [ ] Magic bytes are `0xDEADBEEFCAFEBABE` (appears as `beba feca efbe adde` in `xxd` little-endian)
- [ ] Term field (bytes 8–15) is > 0 after a vote was granted
- [ ] On restart: log shows `HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin`
- [ ] Node re-joins correctly without double-voting

---

## Test 11.5 — Full ASAN Build Verification

**Requirements:** 4

**Steps:**

```bash
# Run unit tests (already built with ASAN via /xce-build)
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
./lib/xcelib_test      2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
```

**Expected output:** No lines (zero ASAN errors introduced by this feature).

Pre-existing leaks from `test_worker_join` in controller_unit_test are acceptable —
they existed before this branch and are not new.

```bash
# Count pre-existing vs new leaks:
./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l
```

**Checklist:**
- [ ] `controller_unit_test` exits 0, all tests PASS
- [ ] `xcelib_test` exits 0, all tests PASS
- [ ] Zero `AddressSanitizer` errors (use-after-free, buffer overflow)
- [ ] No new `LeakSanitizer` leaks from ha-config-bootstrap code paths

---

## Results Template

Copy, fill in, and paste back:

```
## Test Run Results

Date: 2026-06-03
Commit: 374f3bf chore(ha-config-bootstrap): gap analysis audit + fix parent task headers
Build: /xce-build controller completed successfully? YES

### Unit Tests
- controller_unit_test: PASS (1/1, 157s)
- xcelib_test: PASS

### 11.1 Concurrent Startup + Config Replication (3 nodes)
- Election result: PASS
  - Primary elected: ctrl-1
  - Time to election: ~16s (first boot: term=1 standby → phase2 → term=2 leader → term=3 PRIMARY)
  - Two standbys confirmed: YES
- Config replication result: PASS
  - cluster.bin exists in /tmp/xce-deploy/controller/ctrl-2/conf/: YES (cluster.bin.4986624fd309)
  - cluster.bin exists in /tmp/xce-deploy/controller/ctrl-3/conf/: YES (cluster.bin.4986624fd309)
  - Config tag on standbys matches primary: YES (4986624fd309142c)
- Log excerpt:
  ctrl-1: `raft_cb_become_leader → CTRL_PRIMARY_COMMITTING (term=2)`
  ctrl-1: `Config committed on majority (tag=4986624fd309142c, pushed=2/2 standbys)`
  ctrl-1: `ha_do_promote: → CTRL_PRIMARY (term=3)`
  ctrl-2: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)`
  ctrl-3: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)`

### 11.2 Failover and WAL Durability (3 nodes)
- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS (with note)
  - Time to detect liveness loss: ~11s (liveness poll timeout 10s)
  - Both standbys correctly detected: `liveness poll: no primary activity → failover`
  - Both standbys transitioned: `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING`
  - Note: With clean-restart (configs preserved), failover election succeeds.
    With cleanup.sh (wipes standby confs), standbys restart with tag=0 and
    cannot campaign (voter-only mode). See additional notes.
  - Log excerpt (detection):
    ctrl-2: `liveness_poll_fn: no primary activity for 11s → failover`
    ctrl-2: `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING`
    ctrl-3: same pattern
- Scenario 2 (degraded WAL timeout, both standbys down): NOT TESTED (requires xcecli)
- Scenario 3 (reconnect one standby): NOT TESTED (covered by Test 11.1 re-run)

### 11.3 Unconfigured Node Bootstrap (ctrl-2/ctrl-3 join ctrl-1+cluster)
- Result: PASS
- cluster.bin created in /tmp/xce-deploy/controller/ctrl-2/conf/? YES
- cluster.bin created in /tmp/xce-deploy/controller/ctrl-3/conf/? YES
- Log excerpt:
  ctrl-2: `ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only`
  ctrl-2: `myapp_run: CTRL_STANDBY — waiting for primary to connect and push config`
  ctrl-2: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)`
  ctrl-3: same pattern

### 11.4 Term Durability
- ha-term.bin exists: YES (all 3 nodes)
- xxd output (ctrl-1, first 2 lines):
  00000000: beba feca efbe adde 0200 0000 0000 0000  ................
  00000010: 0200 0000 0000 0000 6374 726c 2d31 0000  ........ctrl-1..
  Magic: 0xDEADBEEFCAFEBABE (little-endian) ✓
  Term: 2 ✓
  voted_for: "ctrl-1" ✓
- Restart log (term restore line):
  `cluster_init_phase1: HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin`
- Result: PASS

### 11.5 ASAN Build
- controller_unit_test ASAN errors: 0 (only pre-existing LeakSanitizer from test_worker_join/makeInstance)
- xcelib_test ASAN errors: 0
- Result: PASS

### Additional Notes / Unexpected Behaviour
1. **Architecture change from original design**: The election path now uses
   `ha_start_election_async` + raft library (election-iomgr-refactor), not
   `am_i_primary_quorum`. Config flows via primary push (`handle_repl_config_push`)
   not standby fetch (`do_one_config_fetch`).

2. **Config tag persistence on restarts**: After initial deployment, standbys
   receive config via push. On clean restart (data wipe + preserve conf),
   standbys load `config_tag > 0` from disk and can campaign in failover
   elections. On full cleanup (`cleanup.sh` wipes conf/), standbys restart
   with `config_tag=0` and participate as voters only.

3. **Liveness detection**: Works correctly — both standbys detect primary loss
   within ~10s (liveness_poll_fn timeout). The `ha_arm_heartbeat_timeout_vtask`
   delaymgr-based mechanism replaces the old watchdog pthread.

4. **Dead code removed during this session**: `am_i_primary()`, `probe_thread()`,
   `wait_any_probe()`, `ha_watchdog_thread()`, `ha_start_watchdog()`,
   `ha_stop_watchdog()`, `ha_install_sigusr1()`, `ha_check_heartbeat_nocluster()`,
   `start_heartbeat_listener()`, `stop_heartbeat_listener()`, and associated
   `test_am_i_primary` suite. All ~630 lines removed.
```