finalize configuration replication

This commit is contained in:
Rongsong Shen 2026-06-09 16:40:13 +08:00
parent e1517bbf28
commit fa66d1ba9c
30 changed files with 8867 additions and 1560 deletions

View file

@ -1,6 +1,40 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "Edit|Write|Bash",
"hooks": []
},
{
"matcher": "Read",
"hooks": [
{
"type": "command",
"command": "node \"$CLAUDE_PROJECT_DIR/.wolf/hooks/post-read.js\"",
"timeout": 5
}
]
},
{
"matcher": "Write|Edit|MultiEdit",
"hooks": [
{
"type": "command",
"command": "node \"$CLAUDE_PROJECT_DIR/.wolf/hooks/post-write.js\"",
"timeout": 10
}
]
}
],
"SessionStart": [
{
"matcher": "",
"hooks": []
},
{
"matcher": "",
"hooks": []
},
{
"matcher": "",
"hooks": [
@ -34,28 +68,6 @@
]
}
],
"PostToolUse": [
{
"matcher": "Read",
"hooks": [
{
"type": "command",
"command": "node \"$CLAUDE_PROJECT_DIR/.wolf/hooks/post-read.js\"",
"timeout": 5
}
]
},
{
"matcher": "Write|Edit|MultiEdit",
"hooks": [
{
"type": "command",
"command": "node \"$CLAUDE_PROJECT_DIR/.wolf/hooks/post-write.js\"",
"timeout": 10
}
]
}
],
"Stop": [
{
"matcher": "",

View file

@ -58,11 +58,11 @@ bash src/tools/deploy-ha.sh "$BINARY" "$DATADIR" "$TOP" "$PEERS"
```
This:
- Creates `$TOP/ctrl-{1..N}/conf`, `ctrl-{1..N}/data`, `ctrl-{1..N}/certs`, `logs/`
- Copies `xce.crt` and `xce.key` into each controller's `certs/` dir
- Places `cluster.bin.<tag>` + `conf/current` in `ctrl-1/conf/` only
- Creates `$TOP/certs/`, `$TOP/controller/ctrl-{1..N}/conf`, `$TOP/controller/ctrl-{1..N}/data`, `$TOP/logs/`
- Copies `xce.crt` and `xce.key` into the shared `$TOP/certs/` dir
- Places `cluster.bin.<tag>` + `conf/current` in `controller/ctrl-1/conf/` only
(standbys receive config from primary via `OPCode_replConfigPush`)
- Generates `$TOP/scripts/start.sh` and `cleanup.sh`
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh`
## Step 4 — Start all nodes
@ -88,7 +88,7 @@ Wait ~15 seconds for election and config replication, then check:
grep -i "CTRL_PRIMARY\|push_config\|Config committed" "$TOP/logs/ctrl-1.log" | grep -v DEBUG3
# 2. Confirm ctrl-2 and ctrl-3 received config
find "$TOP/ctrl-2/conf" "$TOP/ctrl-3/conf" -type f | sort
find "$TOP/controller/ctrl-2/conf" "$TOP/controller/ctrl-3/conf" -type f | sort
# 3. Confirm standbys set primary_ctrl_fd
grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctrl-3.log"
@ -97,7 +97,7 @@ grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctr
**Healthy cluster signs:**
- `ctrl-1.log`: `→ CTRL_PRIMARY_COMMITTING` then `→ CTRL_PRIMARY`
- `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)`
- `ctrl-2/conf/cluster.bin.*` and `ctrl-3/conf/cluster.bin.*` exist
- `controller/ctrl-2/conf/cluster.bin.*` and `controller/ctrl-3/conf/cluster.bin.*` exist
- `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded`
## Step 6 — Raft trace (DEBUG3 logs with -vvv)
@ -120,8 +120,8 @@ Stop all instances, clean up state, and restart without redeploying:
kill -9 $(pgrep xce_controller) 2>/dev/null || true
# 2. Run the generated cleanup script
# Keeps: ctrl-1/conf/* (primary config + current pointer), all certs
# Wipes: ctrl-{1..N}/data/*, ctrl-{2..N}/conf/*, logs/*.log
# Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log
bash "$TOP/scripts/cleanup.sh"
# 3. Restart all nodes
@ -133,13 +133,12 @@ VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 &
```
**What cleanup.sh preserves:**
- `ctrl-1/certs/` — TLS cert and key
- `ctrl-1/conf/cluster.bin.*` + `conf/current` — primary config (standbys re-receive via push)
- `ctrl-{2,3}/certs/` — TLS cert and key
- `$TOP/certs/` — shared TLS cert and key
- `controller/ctrl-1/conf/cluster.bin.*` + `conf/current` — primary config (standbys re-receive via push)
**What cleanup.sh removes:**
- All `data/` dirs (WAL, snapshots, `ha-term.bin`) — forces fresh election next start
- `ctrl-{2,3}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush`
- All `controller/ctrl-{1..N}/data/` dirs (WAL, snapshots, `ha-term.bin`) — forces fresh election next start
- All `controller/ctrl-{2..N}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush`
- All `logs/*.log` — clean log slate
If you need a full redeploy (different peers or new binary), run steps 3–5 instead.
@ -149,7 +148,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
| Symptom | Cause | Fix |
|---|---|---|
| `cluster_open_listener: no port` | Peers missing from `--peers` | Ensure `name=host:port` format in generated start.sh |
| `ctrl-2/3/conf` empty | `push_config_to_standbys` bug or `OPCode_replConfigPush` not handled | Verify binary is latest build |
| `controller/ctrl-2/3/conf` empty | `push_config_to_standbys` bug or `OPCode_replConfigPush` not handled | Verify binary is latest build |
| `repl_ack_decode_and_dispatch: unexpected opcode` | Heartbeat NOP on promoted ep | Fixed in current build; update binary |
| `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports |
| `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above |

View file

@ -1 +0,0 @@
{"detected":false,"exe":null,"args":null,"extensions":[]}