finalize configuration replication
This commit is contained in:
parent
e1517bbf28
commit
fa66d1ba9c
30 changed files with 8867 additions and 1560 deletions
|
|
@ -58,9 +58,9 @@ All tests use the `/xce-deploy-ha` skill which automates deployment:
|
|||
```
|
||||
|
||||
This deploys to **`/tmp/xce-deploy/`** with:
|
||||
- `ctrl-{1,2,3}/certs/` — TLS cert and key (copied from `ha/data/`)
|
||||
- `ctrl-1/conf/` — initial config (primary only; standbys receive via push)
|
||||
- `ctrl-{1,2,3}/data/` — WAL, snapshots, `ha-term.bin`
|
||||
- `certs/` — shared TLS cert and key (copied from `ha/data/`)
|
||||
- `controller/ctrl-1/conf/` — initial config (primary only; standbys receive via push)
|
||||
- `controller/ctrl-{1,2,3}/data/` — WAL, snapshots, `ha-term.bin`
|
||||
- `logs/ctrl-{1,2,3}.log` — per-node log files
|
||||
- `scripts/start.sh` and `scripts/cleanup.sh` — lifecycle helpers
|
||||
|
||||
|
|
@ -78,7 +78,7 @@ VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
|
|||
|
||||
# Wait 15s for election + config replication, then verify
|
||||
grep -i "CTRL_PRIMARY\|raft.*leader\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
|
||||
find /tmp/xce-deploy/ctrl-2/conf /tmp/xce-deploy/ctrl-3/conf -type f | sort
|
||||
find /tmp/xce-deploy/controller/ctrl-2/conf /tmp/xce-deploy/controller/ctrl-3/conf -type f | sort
|
||||
```
|
||||
|
||||
### Clean restart (preserves config, wipes data)
|
||||
|
|
@ -89,8 +89,8 @@ bash /tmp/xce-deploy/scripts/cleanup.sh
|
|||
# Then restart nodes as above
|
||||
```
|
||||
|
||||
**What cleanup.sh preserves:** `ctrl-1/conf/*`, all `certs/`
|
||||
**What cleanup.sh removes:** all `data/` dirs, `ctrl-{2,3}/conf/*`, all `logs/*.log`
|
||||
**What cleanup.sh preserves:** `controller/ctrl-1/conf/*`, `$TOP/certs/`
|
||||
**What cleanup.sh removes:** all `controller/ctrl-{1..N}/data/` dirs, `controller/ctrl-{2..N}/conf/*`, all `logs/*.log`
|
||||
|
||||
### Manual restart with config preserved (for failover testing)
|
||||
|
||||
|
|
@ -98,8 +98,8 @@ To restart without wiping standby configs (so standbys retain `config_tag > 0`):
|
|||
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
rm -rf /tmp/xce-deploy/ctrl-{1,2,3}/data /tmp/xce-deploy/logs/*.log
|
||||
mkdir -p /tmp/xce-deploy/ctrl-{1,2,3}/data
|
||||
rm -rf /tmp/xce-deploy/controller/ctrl-{1,2,3}/data /tmp/xce-deploy/logs/*.log
|
||||
mkdir -p /tmp/xce-deploy/controller/ctrl-{1,2,3}/data
|
||||
# Then restart nodes as above
|
||||
```
|
||||
|
||||
|
|
@ -124,8 +124,8 @@ mkdir -p /tmp/xce-deploy/ctrl-{1,2,3}/data
|
|||
- ctrl-2/ctrl-3: `handle_repl_config_push: config saved and loaded (tag=<hex>)`
|
||||
- Config files appear in both standby workdirs:
|
||||
```bash
|
||||
ls /tmp/xce-deploy/ctrl-2/conf/
|
||||
ls /tmp/xce-deploy/ctrl-3/conf/
|
||||
ls /tmp/xce-deploy/controller/ctrl-2/conf/
|
||||
ls /tmp/xce-deploy/controller/ctrl-3/conf/
|
||||
# Expected: cluster.bin.<hex_tag> and current
|
||||
```
|
||||
|
||||
|
|
@ -133,8 +133,8 @@ mkdir -p /tmp/xce-deploy/ctrl-{1,2,3}/data
|
|||
- [ ] Exactly ctrl-1 becomes `CTRL_PRIMARY`
|
||||
- [ ] ctrl-2 and ctrl-3 become `CTRL_STANDBY`
|
||||
- [ ] No crash or ASAN error
|
||||
- [ ] Config replicated to ctrl-2: `cluster.bin.*` exists in `/tmp/xce-deploy/ctrl-2/conf/`
|
||||
- [ ] Config replicated to ctrl-3: `cluster.bin.*` exists in `/tmp/xce-deploy/ctrl-3/conf/`
|
||||
- [ ] Config replicated to ctrl-2: `cluster.bin.*` exists in `/tmp/xce-deploy/controller/ctrl-2/conf/`
|
||||
- [ ] Config replicated to ctrl-3: `cluster.bin.*` exists in `/tmp/xce-deploy/controller/ctrl-3/conf/`
|
||||
- [ ] Config tag on standbys matches primary's tag
|
||||
|
||||
---
|
||||
|
|
@ -207,7 +207,7 @@ CTRL_STANDBY
|
|||
**Scenario:** ctrl-3 starts with no local cluster config (config_tag=0); ctrl-1 and ctrl-2 already hold a valid config from Test 11.1.
|
||||
|
||||
1. Start ctrl-1 and ctrl-2 first (let one become primary).
|
||||
2. Ensure ctrl-3 has no conf: `rm -rf /tmp/xce-deploy/ctrl-3/conf/* 2>/dev/null` (or rely on fresh deploy where only ctrl-1 has conf).
|
||||
2. Ensure ctrl-3 has no conf: `rm -rf /tmp/xce-deploy/controller/ctrl-3/conf/* 2>/dev/null` (or rely on fresh deploy where only ctrl-1 has conf).
|
||||
3. Start ctrl-3.
|
||||
|
||||
**Expected in ctrl-3 log:**
|
||||
|
|
@ -217,14 +217,14 @@ myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
|||
handle_repl_config_push: config saved and loaded (tag=...)
|
||||
```
|
||||
|
||||
**Expected in `/tmp/xce-deploy/ctrl-3/conf/`:** a `cluster.bin.<tag>` and `current` pointer file appear after primary push.
|
||||
**Expected in `/tmp/xce-deploy/controller/ctrl-3/conf/`:** a `cluster.bin.<tag>` and `current` pointer file appear after primary push.
|
||||
|
||||
**Checklist:**
|
||||
- [ ] ctrl-3 detects config_tag=0, participates as voter only
|
||||
- [ ] ctrl-3 yields election to the existing primary (fresher config)
|
||||
- [ ] ctrl-3 receives config via `handle_repl_config_push` from primary
|
||||
- [ ] ctrl-3 enters CTRL_STANDBY with correct config
|
||||
- [ ] Config files appear in `/tmp/xce-deploy/ctrl-3/conf/` after push
|
||||
- [ ] Config files appear in `/tmp/xce-deploy/controller/ctrl-3/conf/` after push
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -237,7 +237,7 @@ handle_repl_config_push: config saved and loaded (tag=...)
|
|||
1. Run 3-node cluster from Test 11.1, wait for election to complete.
|
||||
2. Check that `ha-term.bin` was written on the primary:
|
||||
```bash
|
||||
xxd /tmp/xce-deploy/ctrl-1/data/ha-term.bin | head -4
|
||||
xxd /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin | head -4
|
||||
# Expected: first 8 bytes = 0xBEBAFECAEFBEADDE (magic), next 8 bytes = term > 0
|
||||
```
|
||||
3. Kill ctrl-1.
|
||||
|
|
@ -246,15 +246,15 @@ xxd /tmp/xce-deploy/ctrl-1/data/ha-term.bin | head -4
|
|||
|
||||
**Expected in startup log:**
|
||||
```
|
||||
HA: restored term=N from /tmp/xce-deploy/ctrl-1/data/ha-term.bin
|
||||
HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
```
|
||||
(See `cluster_init_phase1` log line)
|
||||
|
||||
**Checklist:**
|
||||
- [ ] `ha-term.bin` file exists in `/tmp/xce-deploy/ctrl-1/data/` after election
|
||||
- [ ] `ha-term.bin` file exists in `/tmp/xce-deploy/controller/ctrl-1/data/` after election
|
||||
- [ ] Magic bytes are `0xDEADBEEFCAFEBABE` (appears as `beba feca efbe adde` in `xxd` little-endian)
|
||||
- [ ] Term field (bytes 8–15) is > 0 after a vote was granted
|
||||
- [ ] On restart: log shows `HA: restored term=N from /tmp/xce-deploy/ctrl-1/data/ha-term.bin`
|
||||
- [ ] On restart: log shows `HA: restored term=N from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin`
|
||||
- [ ] Node re-joins correctly without double-voting
|
||||
|
||||
---
|
||||
|
|
@ -312,8 +312,8 @@ Build: /xce-build controller completed successfully? YES
|
|||
- Time to election: ~16s (first boot: term=1 standby → phase2 → term=2 leader → term=3 PRIMARY)
|
||||
- Two standbys confirmed: YES
|
||||
- Config replication result: PASS
|
||||
- cluster.bin exists in /tmp/xce-deploy/ctrl-2/conf/: YES (cluster.bin.4986624fd309)
|
||||
- cluster.bin exists in /tmp/xce-deploy/ctrl-3/conf/: YES (cluster.bin.4986624fd309)
|
||||
- cluster.bin exists in /tmp/xce-deploy/controller/ctrl-2/conf/: YES (cluster.bin.4986624fd309)
|
||||
- cluster.bin exists in /tmp/xce-deploy/controller/ctrl-3/conf/: YES (cluster.bin.4986624fd309)
|
||||
- Config tag on standbys matches primary: YES (4986624fd309142c)
|
||||
- Log excerpt:
|
||||
ctrl-1: `raft_cb_become_leader → CTRL_PRIMARY_COMMITTING (term=2)`
|
||||
|
|
@ -339,8 +339,8 @@ Build: /xce-build controller completed successfully? YES
|
|||
|
||||
### 11.3 Unconfigured Node Bootstrap (ctrl-2/ctrl-3 join ctrl-1+cluster)
|
||||
- Result: PASS
|
||||
- cluster.bin created in /tmp/xce-deploy/ctrl-2/conf/? YES
|
||||
- cluster.bin created in /tmp/xce-deploy/ctrl-3/conf/? YES
|
||||
- cluster.bin created in /tmp/xce-deploy/controller/ctrl-2/conf/? YES
|
||||
- cluster.bin created in /tmp/xce-deploy/controller/ctrl-3/conf/? YES
|
||||
- Log excerpt:
|
||||
ctrl-2: `ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only`
|
||||
ctrl-2: `myapp_run: CTRL_STANDBY — waiting for primary to connect and push config`
|
||||
|
|
@ -356,7 +356,7 @@ Build: /xce-build controller completed successfully? YES
|
|||
Term: 2 ✓
|
||||
voted_for: "ctrl-1" ✓
|
||||
- Restart log (term restore line):
|
||||
`cluster_init_phase1: HA: restored term=2 from /tmp/xce-deploy/ctrl-1/data/ha-term.bin`
|
||||
`cluster_init_phase1: HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin`
|
||||
- Result: PASS
|
||||
|
||||
### 11.5 ASAN Build
|
||||
|
|
|
|||
|
|
@ -71,6 +71,19 @@ No new code needed for either spec.
|
|||
- **Recommendation**: Run the 5 integration tests from `manual-tests.md`
|
||||
against a 3-node cluster, then mark the spec `implemented`.
|
||||
|
||||
---
|
||||
|
||||
# Cross-Spec Gap Audit (2026-06-08)
|
||||
|
||||
> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.
|
||||
|
||||
## Analysis Summary
|
||||
|
||||
- **ha-config-bootstrap** is **~95% complete**. All 10 requirements have verified implementation assets. All 22 leaf subtasks (1.1–10.2) are done.
|
||||
- **Remaining work**: Tasks 11.1–11.5 (manual integration tests documented in `manual-tests.md`).
|
||||
- **Cross-spec verification**: `election-iomgr-refactor` superseded the blocking-pthread election path while preserving all ha-config-bootstrap requirements. The two specs are consistent. `ha-repl-hardening` extends Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence — no conflicts.
|
||||
- **Cleanup debt**: `am_i_primary()` (ha.c:869), `ha_wait_for_heartbeat_or_timeout`, and `start_heartbeat_listener()`/`stop_heartbeat_listener()` stubs are dead code — safe to delete in a dedicated cleanup commit (tracked in election-iomgr-refactor/research.md).
|
||||
|
||||
## Requirement-to-Asset Map — Final State
|
||||
|
||||
| Requirement | Verified Key Assets | Status |
|
||||
|
|
@ -446,7 +459,7 @@ Add all new stubs with `-ENOSYS` first, then wire each one. Matches the existing
|
|||
|
||||
### Gap 1: Task 11.1 — Concurrent startup and config-fresh election
|
||||
- **What's needed**: 3-node cluster starts simultaneously; verify only one elects primary; config-fresh node wins.
|
||||
- **How to run**: See `manual-tests.md` — Test 1 (3-node clean start, verify ctrl-a primary).
|
||||
- **How to run**: See `manual-tests.md` — Test 1 (3-node clean start, verify ctrl-1 primary).
|
||||
- **Infrastructure**: `deploy-ha.sh` + `start.sh` scripts; log grep for `CTRL_PRIMARY`.
|
||||
- **Effort**: ~30 min.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue