docs(ha): add xce-deploy-ha skill and update controller CLAUDE.md

- .claude/skills/xce-deploy-ha/SKILL.md: new skill for deploying a
  local 3-node XCE HA controller cluster. Covers build prerequisites,
  deploy via src/tools/deploy-ha.sh, node startup with -vvv raft trace,
  health verification checks, clean restart procedure, and a
  troubleshooting table for common failure modes.
- src/controller/CLAUDE.md: add HA Cluster Deployment section with
  quick-deploy commands, health check one-liners, and reference to the
  xce-deploy-ha skill. Add src/tools/deploy-ha.sh to key files.
This commit is contained in:
Rongsong Shen 2026-06-01 06:01:13 +00:00
parent 3d076ccda6
commit 484b110009
2 changed files with 186 additions and 0 deletions

View file

@ -61,9 +61,55 @@ XCE uses an asymmetric heartbeat model — different from standard Raft:
- Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`.
- `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect.
## HA Cluster Deployment
Use the `/xce-deploy-ha` skill to deploy a local 3-node HA cluster for testing:
```
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
```
**Prerequisites:**
- Build binary: `/xce-build controller`
- TLS certs and `cluster.bin` at `ha/data/` (already in repo)
**Quick deploy (from repo root):**
```bash
# Deploy + start all 3 nodes with raft trace logging
bash src/tools/deploy-ha.sh \
src/xce.build/controller/xce_controller \
ha/data \
/tmp/xce-deploy \
"localhost:9991,localhost:9992,localhost:9993"
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
```
**Verify health (after ~15s):**
```bash
# Primary elected and config pushed
grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
# Config files present on standbys
find /tmp/xce-deploy/ctrl-2/conf /tmp/xce-deploy/ctrl-3/conf -type f
```
**Clean restart:**
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
for i in 1 2 3; do rm -rf /tmp/xce-deploy/ctrl-$i/data/* /tmp/xce-deploy/logs/ctrl-$i.log; done
# Then redeploy
```
See `.claude/skills/xce-deploy-ha/SKILL.md` for full troubleshooting reference.
## Key Files
- `src/controller/` — Controller source files
- `include/` — Controller-specific headers
- `src/tools/deploy-ha.sh` — HA cluster deploy script
## See Also
- Shared library: `src/lib/CLAUDE.md`