docs(ha): add xce-deploy-ha skill and update controller CLAUDE.md
- .claude/skills/xce-deploy-ha/SKILL.md: new skill for deploying a local 3-node XCE HA controller cluster. Covers build prerequisites, deploy via src/tools/deploy-ha.sh, node startup with -vvv raft trace, health verification checks, clean restart procedure, and a troubleshooting table for common failure modes. - src/controller/CLAUDE.md: add HA Cluster Deployment section with quick-deploy commands, health check one-liners, and reference to the xce-deploy-ha skill. Add src/tools/deploy-ha.sh to key files.
This commit is contained in:
parent
3d076ccda6
commit
484b110009
2 changed files with 186 additions and 0 deletions
140
.claude/skills/xce-deploy-ha/SKILL.md
Normal file
140
.claude/skills/xce-deploy-ha/SKILL.md
Normal file
|
|
@ -0,0 +1,140 @@
|
|||
---
|
||||
name: xce-deploy-ha
|
||||
description: "Deploy and manage a local XCE HA controller cluster for testing. Usage: /xce-deploy-ha [peers] — e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993"
|
||||
---
|
||||
|
||||
# XCE HA Cluster Deployment
|
||||
|
||||
Deploy a multi-node XCE controller cluster for local testing using
|
||||
`src/tools/deploy-ha.sh`. Covers: build, deploy, start, verify, stop, and cleanup.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Build must be current: run `/xce-build controller` first if binary is stale
|
||||
- TLS certs and a `cluster.bin` must exist under `ha/data/`:
|
||||
- `ha/data/xce.crt`
|
||||
- `ha/data/xce.key`
|
||||
- `ha/data/cluster.bin`
|
||||
|
||||
## Arguments
|
||||
|
||||
Arguments are space-separated `host:port` pairs, one per node, in priority order
|
||||
(index 0 = highest priority = primary candidate):
|
||||
|
||||
```
|
||||
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
||||
```
|
||||
|
||||
If no arguments are given, default to `localhost:9991 localhost:9992 localhost:9993`.
|
||||
|
||||
## Step 1 — Resolve paths
|
||||
|
||||
```bash
|
||||
BINARY="src/xce.build/controller/xce_controller"
|
||||
DATADIR="ha/data"
|
||||
TOP="/tmp/xce-deploy"
|
||||
```
|
||||
|
||||
Verify the binary exists:
|
||||
```bash
|
||||
ls -lh "$BINARY"
|
||||
```
|
||||
|
||||
If missing, tell the user to run `/xce-build controller` first.
|
||||
|
||||
## Step 2 — Stop any running instances
|
||||
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
```
|
||||
|
||||
## Step 3 — Deploy cluster
|
||||
|
||||
Join peers as a comma-separated string and call the deploy script:
|
||||
|
||||
```bash
|
||||
PEERS="localhost:9991,localhost:9992,localhost:9993" # from args
|
||||
bash src/tools/deploy-ha.sh "$BINARY" "$DATADIR" "$TOP" "$PEERS"
|
||||
```
|
||||
|
||||
This:
|
||||
- Creates `$TOP/ctrl-{1..N}/conf`, `ctrl-{1..N}/data`, `ctrl-{1..N}/certs`, `logs/`
|
||||
- Copies `xce.crt` and `xce.key` into each controller's `certs/` dir
|
||||
- Places `cluster.bin.<tag>` + `conf/current` in `ctrl-1/conf/` only
|
||||
(standbys receive config from primary via `OPCode_replConfigPush`)
|
||||
- Generates `$TOP/scripts/start.sh` and `cleanup.sh`
|
||||
|
||||
## Step 4 — Start all nodes
|
||||
|
||||
Start each controller in the background. Use `-vvv` for raft trace logs:
|
||||
|
||||
```bash
|
||||
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 &
|
||||
```
|
||||
|
||||
The startup stagger (0.5s between nodes) allows higher-priority nodes to
|
||||
begin their election before lower-priority ones start.
|
||||
|
||||
## Step 5 — Verify cluster health
|
||||
|
||||
Wait ~15 seconds for election and config replication, then check:
|
||||
|
||||
```bash
|
||||
# 1. Confirm ctrl-1 became primary and pushed config
|
||||
grep -i "CTRL_PRIMARY\|push_config\|Config committed" "$TOP/logs/ctrl-1.log" | grep -v DEBUG3
|
||||
|
||||
# 2. Confirm ctrl-2 and ctrl-3 received config
|
||||
find "$TOP/ctrl-2/conf" "$TOP/ctrl-3/conf" -type f | sort
|
||||
|
||||
# 3. Confirm standbys set primary_ctrl_fd
|
||||
grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctrl-3.log"
|
||||
```
|
||||
|
||||
**Healthy cluster signs:**
|
||||
- `ctrl-1.log`: `→ CTRL_PRIMARY_COMMITTING` then `→ CTRL_PRIMARY`
|
||||
- `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)`
|
||||
- `ctrl-2/conf/cluster.bin.*` and `ctrl-3/conf/cluster.bin.*` exist
|
||||
- `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded`
|
||||
|
||||
## Step 6 — Raft trace (DEBUG3 logs with -vvv)
|
||||
|
||||
With `-vvv`, raft vote events appear at `DEBUG3`:
|
||||
|
||||
```
|
||||
RAFT[p0] term=2 event=election_start
|
||||
RAFT[p0] term=2 event=vote_reply_in
|
||||
RAFT[p0] term=2 event=quorum_reached
|
||||
RAFT[p0] term=2 event=became_leader
|
||||
```
|
||||
|
||||
## Clean restart
|
||||
|
||||
To wipe state and redeploy from scratch:
|
||||
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
for i in 1 2 3; do
|
||||
rm -rf "$TOP/ctrl-$i/data/"*
|
||||
rm -f "$TOP/logs/ctrl-$i.log"
|
||||
done
|
||||
# Then re-run steps 3–5
|
||||
```
|
||||
|
||||
Or use the generated cleanup script (preserves primary config but wipes data/logs):
|
||||
```bash
|
||||
bash "$TOP/scripts/cleanup.sh"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `cluster_open_listener: no port` | Peers missing from `--peers` | Ensure `name=host:port` format in generated start.sh |
|
||||
| `ctrl-2/3/conf` empty | `push_config_to_standbys` bug or `OPCode_replConfigPush` not handled | Verify binary is latest build |
|
||||
| `repl_ack_decode_and_dispatch: unexpected opcode` | Heartbeat NOP on promoted ep | Fixed in current build; update binary |
|
||||
| `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports |
|
||||
| `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above |
|
||||
|
|
@ -61,9 +61,55 @@ XCE uses an asymmetric heartbeat model — different from standard Raft:
|
|||
- Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`.
|
||||
- `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect.
|
||||
|
||||
## HA Cluster Deployment
|
||||
|
||||
Use the `/xce-deploy-ha` skill to deploy a local 3-node HA cluster for testing:
|
||||
|
||||
```
|
||||
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
||||
```
|
||||
|
||||
**Prerequisites:**
|
||||
- Build binary: `/xce-build controller`
|
||||
- TLS certs and `cluster.bin` at `ha/data/` (already in repo)
|
||||
|
||||
**Quick deploy (from repo root):**
|
||||
```bash
|
||||
# Deploy + start all 3 nodes with raft trace logging
|
||||
bash src/tools/deploy-ha.sh \
|
||||
src/xce.build/controller/xce_controller \
|
||||
ha/data \
|
||||
/tmp/xce-deploy \
|
||||
"localhost:9991,localhost:9992,localhost:9993"
|
||||
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
|
||||
```
|
||||
|
||||
**Verify health (after ~15s):**
|
||||
```bash
|
||||
# Primary elected and config pushed
|
||||
grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
|
||||
# Config files present on standbys
|
||||
find /tmp/xce-deploy/ctrl-2/conf /tmp/xce-deploy/ctrl-3/conf -type f
|
||||
```
|
||||
|
||||
**Clean restart:**
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
for i in 1 2 3; do rm -rf /tmp/xce-deploy/ctrl-$i/data/* /tmp/xce-deploy/logs/ctrl-$i.log; done
|
||||
# Then redeploy
|
||||
```
|
||||
|
||||
See `.claude/skills/xce-deploy-ha/SKILL.md` for full troubleshooting reference.
|
||||
|
||||
## Key Files
|
||||
- `src/controller/` — Controller source files
|
||||
- `include/` — Controller-specific headers
|
||||
- `src/tools/deploy-ha.sh` — HA cluster deploy script
|
||||
|
||||
## See Also
|
||||
- Shared library: `src/lib/CLAUDE.md`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue