Rongsong Shen fa66d1ba9c finalize configuration replication

2026-06-09 16:40:13 +08:00

15 KiB

Raw Permalink Blame History

Manual Integration Tests: ha-config-bootstrap

This document describes how to run the integration tests for tasks 11.1–11.5. Run each test, record the output, and paste the filled Results Template at the bottom back to the developer.

Environment

Worktree:   /home/rshen/projects/xce/worktrees/persistent_ha
Build dir:  /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build
Controller: src/xce.build/controller/xce_controller
CLI:        src/xce.build/frontend/xcecli/xce   (or /tmp/xce_static after make build-cli)
Unit tests: src/xce.build/controller/controller_unit_test
            src/xce.build/lib/xcelib_test

Build First

# From the worktree
/xce-build           # full dev build (preferred: uses nix develop + meson)

# Or manually from the src directory:
nix develop /home/rshen/projects/xce/xce/src --command bash -c \
  "cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build && ninja"

Unit Tests (Sanity Check Before Integration)

cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>/dev/null | tail -5
./lib/xcelib_test      2>/dev/null | tail -3

Expected: all tests PASS, no FAIL lines, exit 0.

Test Infrastructure

Design constraint: The controller rejects an even number of peer nodes (--peers list), because Raft quorum requires an odd cluster size (1, 3, 5…) to guarantee a safe majority. All tests use a 3-node cluster.

Deploy via `/xce-deploy-ha` skill

All tests use the /xce-deploy-ha skill which automates deployment:

# 1. Build (if not already built)
/xce-build controller

# 2. Deploy 3-node cluster
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

This deploys to /tmp/xce-deploy/ with:

certs/ — shared TLS cert and key (copied from ha/data/)
controller/ctrl-1/conf/ — initial config (primary only; standbys receive via push)
controller/ctrl-{1,2,3}/data/ — WAL, snapshots, ha-term.bin
logs/ctrl-{1,2,3}.log — per-node log files
scripts/start.sh and scripts/cleanup.sh — lifecycle helpers

Nodes are named ctrl-1, ctrl-2, ctrl-3 (index 0 = highest priority = ctrl-1).

Start and verify

# Start all 3 (staggered by 0.5s)
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &

# Wait 15s for election + config replication, then verify
grep -i "CTRL_PRIMARY\|raft.*leader\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
find /tmp/xce-deploy/controller/ctrl-2/conf /tmp/xce-deploy/controller/ctrl-3/conf -type f | sort

Clean restart (preserves config, wipes data)

kill -9 $(pgrep xce_controller) 2>/dev/null || true
bash /tmp/xce-deploy/scripts/cleanup.sh
# Then restart nodes as above

What cleanup.sh preserves: controller/ctrl-1/conf/*, $TOP/certs/ What cleanup.sh removes: all controller/ctrl-{1..N}/data/ dirs, controller/ctrl-{2..N}/conf/*, all logs/*.log

Manual restart with config preserved (for failover testing)

To restart without wiping standby configs (so standbys retain config_tag > 0):

kill -9 $(pgrep xce_controller) 2>/dev/null || true
rm -rf /tmp/xce-deploy/controller/ctrl-{1,2,3}/data /tmp/xce-deploy/logs/*.log
mkdir -p /tmp/xce-deploy/controller/ctrl-{1,2,3}/data
# Then restart nodes as above

Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication

Requirements: 1.1, 2.1, 2.7, 3.2, 3.5

Scenario: ctrl-1 has the initial config; ctrl-2 and ctrl-3 start with config_tag=0 (no config). All three start concurrently — ctrl-1 wins election (freshest config), then pushes the config to ctrl-2 and ctrl-3 via replication.

Steps: Deploy and start as described in Test Infrastructure above.

Expected log patterns — Election (current implementation):

ctrl-1: raft_cb_become_leader → CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY
ctrl-2/ctrl-3: myapp_run: CTRL_STANDBY — waiting for primary to connect and push config

Expected log patterns — Config replication (primary push model):

ctrl-1: Config committed on majority (tag=<hex>, pushed=2/2 standbys)
ctrl-2/ctrl-3: handle_repl_config_push: config saved and loaded (tag=<hex>)

Config files appear in both standby workdirs:

ls /tmp/xce-deploy/controller/ctrl-2/conf/
ls /tmp/xce-deploy/controller/ctrl-3/conf/
# Expected: cluster.bin.<hex_tag> and current

Checklist:

Exactly ctrl-1 becomes CTRL_PRIMARY
ctrl-2 and ctrl-3 become CTRL_STANDBY
No crash or ASAN error
Config replicated to ctrl-2: cluster.bin.* exists in /tmp/xce-deploy/controller/ctrl-2/conf/
Config replicated to ctrl-3: cluster.bin.* exists in /tmp/xce-deploy/controller/ctrl-3/conf/
Config tag on standbys matches primary's tag

Test 11.2 — Failover and WAL Durability

Requirements: 5.6, 6.2, 5.5

Prerequisites: 3-node cluster running from Test 11.1.

Scenario 1: Primary crash → liveness timeout → re-election

Identify which node is primary: grep CTRL_PRIMARY /tmp/xce-deploy/logs/ctrl-*.log
Kill the primary process: kill -9 <pid>.
Watch the two surviving standbys: tail -f /tmp/xce-deploy/logs/ctrl-{2,3}.log

Expected on ctrl-2 and ctrl-3 (both standbys, within ~10s of primary kill):

HA: liveness poll: no primary activity for 10s → failover
HA: primary_ctrl_fd RDHUP → CTRL_ELECTING

Expected on the winner (standby with config_tag > 0 campaigns and wins):

raft_cb_become_leader
CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY

Note (2026-06-03): Failover election requires standbys to have a non-zero config_tag loaded at startup. After the initial deployment, standbys receive config via handle_repl_config_push. On subsequent restarts, if cleanup.sh wipes ctrl-{2,3}/conf/, the standbys restart with config_tag=0 and participate as voters only (they will not campaign). Preserve ctrl-{2,3}/conf/ across restarts for failover to complete. See test variant below in results.

Scenario 2: WAL commit with both standbys down → ETIMEDOUT

Kill both standbys (leaving only the primary).
Submit a job:

/home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build/frontend/xcecli/xce \
  --server 127.0.0.1:9901 jobsubmit -c "hostname"

Expected in primary log: repl_wait_majority_ack: timeout — NOT a crash.

Scenario 3: Reconnected standby catches up

Restart one of the killed standbys.
Watch logs on the restarted node.

Expected:

CtrlHelloReply from primary: committed_lsn=...
CTRL_STANDBY

Checklist:

RDHUP failover detected within 1s of primary kill
One surviving standby promoted to CTRL_PRIMARY (needed 1 vote from other)
WAL commit with both standbys down returns ETIMEDOUT (graceful degraded mode)
Restarted standby reconnects and enters CTRL_STANDBY

Test 11.3 — Unconfigured Node Bootstrap

Requirements: 3.2, 3.5

Scenario: ctrl-3 starts with no local cluster config (config_tag=0); ctrl-1 and ctrl-2 already hold a valid config from Test 11.1.

Start ctrl-1 and ctrl-2 first (let one become primary).
Ensure ctrl-3 has no conf: rm -rf /tmp/xce-deploy/controller/ctrl-3/conf/* 2>/dev/null (or rely on fresh deploy where only ctrl-1 has conf).
Start ctrl-3.

Expected in ctrl-3 log:

ha_start_election_async: no local config (tag=0, n_peers=3)
myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
handle_repl_config_push: config saved and loaded (tag=...)

Expected in /tmp/xce-deploy/controller/ctrl-3/conf/: a cluster.bin.<tag> and current pointer file appear after primary push.

Checklist:

ctrl-3 detects config_tag=0, participates as voter only
ctrl-3 yields election to the existing primary (fresher config)
ctrl-3 receives config via handle_repl_config_push from primary
ctrl-3 enters CTRL_STANDBY with correct config
Config files appear in /tmp/xce-deploy/controller/ctrl-3/conf/ after push

Test 11.4 — Term Durability Across Restart

Requirements: 9.1, 9.2

Scenario: After election, ha-term.bin is written; on restart, term is restored.

Run 3-node cluster from Test 11.1, wait for election to complete.
Check that ha-term.bin was written on the primary:

xxd /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin | head -4
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0

Kill ctrl-1.
Restart ctrl-1: VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
Check startup log for term restore.

Expected in startup log:

HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin

(See cluster_init_phase1 log line)

Checklist:

ha-term.bin file exists in /tmp/xce-deploy/controller/ctrl-1/data/ after election
Magic bytes are 0xDEADBEEFCAFEBABE (appears as beba feca efbe adde in xxd little-endian)
Term field (bytes 8–15) is > 0 after a vote was granted
On restart: log shows HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
Node re-joins correctly without double-voting

Test 11.5 — Full ASAN Build Verification

Requirements: 4

Steps:

# Run unit tests (already built with ASAN via /xce-build)
cd /home/rshen/projects/xce/worktrees/persistent_ha/src/xce.build

./controller/controller_unit_test 2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'
./lib/xcelib_test      2>&1 | grep -E 'ERROR: (LeakSanitizer|AddressSanitizer)'

Expected output: No lines (zero ASAN errors introduced by this feature).

Pre-existing leaks from test_worker_join in controller_unit_test are acceptable — they existed before this branch and are not new.

# Count pre-existing vs new leaks:
./controller/controller_unit_test 2>&1 | grep 'LeakSanitizer' | wc -l

Checklist:

controller_unit_test exits 0, all tests PASS
xcelib_test exits 0, all tests PASS
Zero AddressSanitizer errors (use-after-free, buffer overflow)
No new LeakSanitizer leaks from ha-config-bootstrap code paths

Results Template

Copy, fill in, and paste back:

## Test Run Results

Date: 2026-06-03
Commit: 374f3bf chore(ha-config-bootstrap): gap analysis audit + fix parent task headers
Build: /xce-build controller completed successfully? YES

### Unit Tests
- controller_unit_test: PASS (1/1, 157s)
- xcelib_test: PASS

### 11.1 Concurrent Startup + Config Replication (3 nodes)
- Election result: PASS
  - Primary elected: ctrl-1
  - Time to election: ~16s (first boot: term=1 standby → phase2 → term=2 leader → term=3 PRIMARY)
  - Two standbys confirmed: YES
- Config replication result: PASS
  - cluster.bin exists in /tmp/xce-deploy/controller/ctrl-2/conf/: YES (cluster.bin.4986624fd309)
  - cluster.bin exists in /tmp/xce-deploy/controller/ctrl-3/conf/: YES (cluster.bin.4986624fd309)
  - Config tag on standbys matches primary: YES (4986624fd309142c)
- Log excerpt:
  ctrl-1: `raft_cb_become_leader → CTRL_PRIMARY_COMMITTING (term=2)`
  ctrl-1: `Config committed on majority (tag=4986624fd309142c, pushed=2/2 standbys)`
  ctrl-1: `ha_do_promote: → CTRL_PRIMARY (term=3)`
  ctrl-2: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)`
  ctrl-3: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)`

### 11.2 Failover and WAL Durability (3 nodes)
- Scenario 1 (RDHUP failover, 1 of 2 survivors promotes): PASS (with note)
  - Time to detect liveness loss: ~11s (liveness poll timeout 10s)
  - Both standbys correctly detected: `liveness poll: no primary activity → failover`
  - Both standbys transitioned: `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING`
  - Note: With clean-restart (configs preserved), failover election succeeds.
    With cleanup.sh (wipes standby confs), standbys restart with tag=0 and
    cannot campaign (voter-only mode). See additional notes.
  - Log excerpt (detection):
    ctrl-2: `liveness_poll_fn: no primary activity for 11s → failover`
    ctrl-2: `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING`
    ctrl-3: same pattern
- Scenario 2 (degraded WAL timeout, both standbys down): NOT TESTED (requires xcecli)
- Scenario 3 (reconnect one standby): NOT TESTED (covered by Test 11.1 re-run)

### 11.3 Unconfigured Node Bootstrap (ctrl-2/ctrl-3 join ctrl-1+cluster)
- Result: PASS
- cluster.bin created in /tmp/xce-deploy/controller/ctrl-2/conf/? YES
- cluster.bin created in /tmp/xce-deploy/controller/ctrl-3/conf/? YES
- Log excerpt:
  ctrl-2: `ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only`
  ctrl-2: `myapp_run: CTRL_STANDBY — waiting for primary to connect and push config`
  ctrl-2: `handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)`
  ctrl-3: same pattern

### 11.4 Term Durability
- ha-term.bin exists: YES (all 3 nodes)
- xxd output (ctrl-1, first 2 lines):
  00000000: beba feca efbe adde 0200 0000 0000 0000  ................
  00000010: 0200 0000 0000 0000 6374 726c 2d31 0000  ........ctrl-1..
  Magic: 0xDEADBEEFCAFEBABE (little-endian) ✓
  Term: 2 ✓
  voted_for: "ctrl-1" ✓
- Restart log (term restore line):
  `cluster_init_phase1: HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin`
- Result: PASS

### 11.5 ASAN Build
- controller_unit_test ASAN errors: 0 (only pre-existing LeakSanitizer from test_worker_join/makeInstance)
- xcelib_test ASAN errors: 0
- Result: PASS

### Additional Notes / Unexpected Behaviour
1. **Architecture change from original design**: The election path now uses
   `ha_start_election_async` + raft library (election-iomgr-refactor), not
   `am_i_primary_quorum`. Config flows via primary push (`handle_repl_config_push`)
   not standby fetch (`do_one_config_fetch`).

2. **Config tag persistence on restarts**: After initial deployment, standbys
   receive config via push. On clean restart (data wipe + preserve conf),
   standbys load `config_tag > 0` from disk and can campaign in failover
   elections. On full cleanup (`cleanup.sh` wipes conf/), standbys restart
   with `config_tag=0` and participate as voters only.

3. **Liveness detection**: Works correctly — both standbys detect primary loss
   within ~10s (liveness_poll_fn timeout). The `ha_arm_heartbeat_timeout_vtask`
   delaymgr-based mechanism replaces the old watchdog pthread.

4. **Dead code removed during this session**: `am_i_primary()`, `probe_thread()`,
   `wait_any_probe()`, `ha_watchdog_thread()`, `ha_start_watchdog()`,
   `ha_stop_watchdog()`, `ha_install_sigusr1()`, `ha_check_heartbeat_nocluster()`,
   `start_heartbeat_listener()`, `stop_heartbeat_listener()`, and associated
   `test_am_i_primary` suite. All ~630 lines removed.

15 KiB Raw Permalink Blame History Unescape Escape

Manual Integration Tests: ha-config-bootstrap

Environment

Build First

Unit Tests (Sanity Check Before Integration)

Test Infrastructure

Deploy via /xce-deploy-ha skill

Start and verify

Clean restart (preserves config, wipes data)

Manual restart with config preserved (for failover testing)

Test 11.1 — Concurrent Startup, Config-Fresh Election, and Config Replication

Test 11.2 — Failover and WAL Durability

Scenario 1: Primary crash → liveness timeout → re-election

Scenario 2: WAL commit with both standbys down → ETIMEDOUT

Scenario 3: Reconnected standby catches up

Test 11.3 — Unconfigured Node Bootstrap

Test 11.4 — Term Durability Across Restart

Test 11.5 — Full ASAN Build Verification

Results Template

15 KiB

Raw Permalink Blame History

Deploy via `/xce-deploy-ha` skill