diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md new file mode 100644 index 00000000..678758a4 --- /dev/null +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -0,0 +1,140 @@ +--- +name: xce-deploy-ha +description: "Deploy and manage a local XCE HA controller cluster for testing. Usage: /xce-deploy-ha [peers] — e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993" +--- + +# XCE HA Cluster Deployment + +Deploy a multi-node XCE controller cluster for local testing using +`src/tools/deploy-ha.sh`. Covers: build, deploy, start, verify, stop, and cleanup. + +## Prerequisites + +- Build must be current: run `/xce-build controller` first if binary is stale +- TLS certs and a `cluster.bin` must exist under `ha/data/`: + - `ha/data/xce.crt` + - `ha/data/xce.key` + - `ha/data/cluster.bin` + +## Arguments + +Arguments are space-separated `host:port` pairs, one per node, in priority order +(index 0 = highest priority = primary candidate): + +``` +/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993 +``` + +If no arguments are given, default to `localhost:9991 localhost:9992 localhost:9993`. + +## Step 1 — Resolve paths + +```bash +BINARY="src/xce.build/controller/xce_controller" +DATADIR="ha/data" +TOP="/tmp/xce-deploy" +``` + +Verify the binary exists: +```bash +ls -lh "$BINARY" +``` + +If missing, tell the user to run `/xce-build controller` first. + +## Step 2 — Stop any running instances + +```bash +kill -9 $(pgrep xce_controller) 2>/dev/null || true +``` + +## Step 3 — Deploy cluster + +Join peers as a comma-separated string and call the deploy script: + +```bash +PEERS="localhost:9991,localhost:9992,localhost:9993" # from args +bash src/tools/deploy-ha.sh "$BINARY" "$DATADIR" "$TOP" "$PEERS" +``` + +This: +- Creates `$TOP/ctrl-{1..N}/conf`, `ctrl-{1..N}/data`, `ctrl-{1..N}/certs`, `logs/` +- Copies `xce.crt` and `xce.key` into each controller's `certs/` dir +- Places `cluster.bin.` + `conf/current` in `ctrl-1/conf/` only + (standbys receive config from primary via `OPCode_replConfigPush`) +- Generates `$TOP/scripts/start.sh` and `cleanup.sh` + +## Step 4 — Start all nodes + +Start each controller in the background. Use `-vvv` for raft trace logs: + +```bash +VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 & +sleep 0.5 +VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 & +sleep 0.5 +VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 & +``` + +The startup stagger (0.5s between nodes) allows higher-priority nodes to +begin their election before lower-priority ones start. + +## Step 5 — Verify cluster health + +Wait ~15 seconds for election and config replication, then check: + +```bash +# 1. Confirm ctrl-1 became primary and pushed config +grep -i "CTRL_PRIMARY\|push_config\|Config committed" "$TOP/logs/ctrl-1.log" | grep -v DEBUG3 + +# 2. Confirm ctrl-2 and ctrl-3 received config +find "$TOP/ctrl-2/conf" "$TOP/ctrl-3/conf" -type f | sort + +# 3. Confirm standbys set primary_ctrl_fd +grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctrl-3.log" +``` + +**Healthy cluster signs:** +- `ctrl-1.log`: `→ CTRL_PRIMARY_COMMITTING` then `→ CTRL_PRIMARY` +- `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)` +- `ctrl-2/conf/cluster.bin.*` and `ctrl-3/conf/cluster.bin.*` exist +- `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded` + +## Step 6 — Raft trace (DEBUG3 logs with -vvv) + +With `-vvv`, raft vote events appear at `DEBUG3`: + +``` +RAFT[p0] term=2 event=election_start +RAFT[p0] term=2 event=vote_reply_in +RAFT[p0] term=2 event=quorum_reached +RAFT[p0] term=2 event=became_leader +``` + +## Clean restart + +To wipe state and redeploy from scratch: + +```bash +kill -9 $(pgrep xce_controller) 2>/dev/null || true +for i in 1 2 3; do + rm -rf "$TOP/ctrl-$i/data/"* + rm -f "$TOP/logs/ctrl-$i.log" +done +# Then re-run steps 3–5 +``` + +Or use the generated cleanup script (preserves primary config but wipes data/logs): +```bash +bash "$TOP/scripts/cleanup.sh" +``` + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `cluster_open_listener: no port` | Peers missing from `--peers` | Ensure `name=host:port` format in generated start.sh | +| `ctrl-2/3/conf` empty | `push_config_to_standbys` bug or `OPCode_replConfigPush` not handled | Verify binary is latest build | +| `repl_ack_decode_and_dispatch: unexpected opcode` | Heartbeat NOP on promoted ep | Fixed in current build; update binary | +| `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports | +| `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above | diff --git a/src/controller/CLAUDE.md b/src/controller/CLAUDE.md index e9227d34..32764c15 100644 --- a/src/controller/CLAUDE.md +++ b/src/controller/CLAUDE.md @@ -61,9 +61,55 @@ XCE uses an asymmetric heartbeat model — different from standard Raft: - Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`. - `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect. +## HA Cluster Deployment + +Use the `/xce-deploy-ha` skill to deploy a local 3-node HA cluster for testing: + +``` +/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993 +``` + +**Prerequisites:** +- Build binary: `/xce-build controller` +- TLS certs and `cluster.bin` at `ha/data/` (already in repo) + +**Quick deploy (from repo root):** +```bash +# Deploy + start all 3 nodes with raft trace logging +bash src/tools/deploy-ha.sh \ + src/xce.build/controller/xce_controller \ + ha/data \ + /tmp/xce-deploy \ + "localhost:9991,localhost:9992,localhost:9993" + +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 & +sleep 0.5 +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 & +sleep 0.5 +VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 & +``` + +**Verify health (after ~15s):** +```bash +# Primary elected and config pushed +grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3 +# Config files present on standbys +find /tmp/xce-deploy/ctrl-2/conf /tmp/xce-deploy/ctrl-3/conf -type f +``` + +**Clean restart:** +```bash +kill -9 $(pgrep xce_controller) 2>/dev/null || true +for i in 1 2 3; do rm -rf /tmp/xce-deploy/ctrl-$i/data/* /tmp/xce-deploy/logs/ctrl-$i.log; done +# Then redeploy +``` + +See `.claude/skills/xce-deploy-ha/SKILL.md` for full troubleshooting reference. + ## Key Files - `src/controller/` — Controller source files - `include/` — Controller-specific headers +- `src/tools/deploy-ha.sh` — HA cluster deploy script ## See Also - Shared library: `src/lib/CLAUDE.md`