forked from lei.chen/myxce
chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
This commit is contained in:
parent
084244941b
commit
06135954ff
88 changed files with 25611 additions and 5285 deletions
|
|
@ -5,10 +5,9 @@ C daemon managing cluster-level job and queue operations. Acts as the central co
|
|||
|
||||
## Build
|
||||
```bash
|
||||
# From workspace root
|
||||
nix develop
|
||||
meson build && ninja -C build
|
||||
/xce-build controller # debug + ASAN + clean build + runs tests
|
||||
```
|
||||
For full build system details, see `.kiro/steering/build-deployment.md`.
|
||||
|
||||
## Architecture
|
||||
Core source files handle distinct domains:
|
||||
|
|
@ -25,7 +24,9 @@ Links against `libxce` from `src/lib/`.
|
|||
- Component-specific headers in `include/`
|
||||
- Shared types defined in `src/include/xce/` (libxce)
|
||||
|
||||
## HA Protocol Verification
|
||||
## HA Protocol
|
||||
|
||||
See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
|
||||
|
||||
### Raft TLA+ Formal Specification
|
||||
|
||||
|
|
@ -91,94 +92,9 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
|
|||
|
||||
## HA Design Constraints
|
||||
|
||||
### Inbound vs. Outbound Connection Ownership (CRITICAL)
|
||||
|
||||
**Strict rule: inbound connections are for voting only. All persistent control channels are outbound from the primary.**
|
||||
|
||||
| Connection type | Direction | Purpose | Who closes |
|
||||
|---|---|---|---|
|
||||
| Vote RPC | Standby → Primary (outbound from standby) | `OPCode_requestVote` + reply only | Remote closes after reply |
|
||||
| Standby control channel | Primary → Standby (outbound from primary) | ctrlhello, config push, WAL replication, heartbeat | `vote_ep_err_handler` / `standby_ep_err_handler_primary` on disconnect |
|
||||
|
||||
Rules:
|
||||
- **Inbound vote connections** (accepted on the listener) are used **exclusively** for the vote exchange. After the reply is sent the primary does NOT reuse the connection for config/replication. The inbound ep is left open until the remote closes it.
|
||||
- **Primary never uses an inbound connection to push state.** All state push (ctrlhello, config, WAL, heartbeat nop) goes over **outbound connections** stored in `ha->standby_endpoints`.
|
||||
- During election: candidate opens outgoing vote connections (one per peer). After winning, `raft_cb_become_leader` promotes surviving vote connections to `standby_endpoints`.
|
||||
- Late joiners: the primary's heartbeat task (`primary_heartbeat_fn`) scans the peers list and **opens a fresh outbound connection** to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello and registers the ep.
|
||||
- `CTRL_STANDBY` must not call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary` after an election result — the primary will connect to it.
|
||||
|
||||
### Asynchronous, Non-Blocking I/O (CRITICAL)
|
||||
All inter-controller communication uses iomgr (epoll-based) and vtaskmgr:
|
||||
- `iomgr` dispatches inbound packets to vtask handlers — never block inside a vtask.
|
||||
- `vtaskmgr` runs handlers on a thread pool; each handler must return quickly.
|
||||
- Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — non-blocking.
|
||||
- Blocking I/O (e.g., `fetch_config_from_primary`, `ha_find_and_connect_to_primary`) is only permitted from `delaymgr` worker threads, **never** from iomgr/vtask context.
|
||||
|
||||
### Heartbeat Model
|
||||
|
||||
XCE uses a bidirectional heartbeat model aligned with Raft's AppendEntries:
|
||||
- The **primary** sends periodic heartbeats (`OPCode_nop`) to all standbys via `ha_start_primary_heartbeat` (delaymgr task, interval=3s). During `CTRL_PRIMARY_COMMITTING`, this also serves as the replication driver — the primary evaluates standby state from heartbeat replies and pushes config/WAL/snapshot as needed.
|
||||
- The **standby** replies to each heartbeat with its current `(applied_lsn, config_tag)` in `RequestReply.stateReport`. This gives the primary continuous visibility into standby convergence.
|
||||
- The **standby** also arms a liveness watchdog (`ha_arm_heartbeat_timeout_vtask`) when entering `CTRL_STANDBY`. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), it triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||||
- `raft.c` (the election library) is election-only. Heartbeat and replication logic belongs in the HA layer (`ha.c`, `config_repl.c`).
|
||||
|
||||
### Connection Ownership
|
||||
- Promoted vote connections (`ha->standby_endpoints`): owned by `vote_ep_err_handler`; freed when the standby disconnects.
|
||||
- Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`.
|
||||
- `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect.
|
||||
|
||||
## HA Cluster Deployment
|
||||
|
||||
Use the `/xce-deploy-ha` skill to deploy a local 3-node HA cluster for testing:
|
||||
|
||||
```
|
||||
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
||||
```
|
||||
|
||||
**Prerequisites:**
|
||||
- Build binary: `/xce-build controller`
|
||||
- TLS certs and `cluster.bin` at `ha/data/` (already in repo)
|
||||
|
||||
**Quick deploy (from repo root):**
|
||||
```bash
|
||||
# Deploy + start all 3 nodes with raft trace logging
|
||||
bash src/tools/deploy-ha.sh \
|
||||
src/xce.build/controller/xce_controller \
|
||||
ha/data \
|
||||
/tmp/xce-deploy \
|
||||
"localhost:9991,localhost:9992,localhost:9993"
|
||||
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
|
||||
```
|
||||
|
||||
**Verify health (after ~15s):**
|
||||
```bash
|
||||
# Primary elected and config pushed
|
||||
grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
|
||||
# Config files present on standbys
|
||||
find /tmp/xce-deploy/controller/ctrl-2/conf /tmp/xce-deploy/controller/ctrl-3/conf -type f
|
||||
```
|
||||
|
||||
**Clean restart** (keeps ctrl-1 config + all certs, wipes data/standby-conf/logs):
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
bash /tmp/xce-deploy/scripts/cleanup.sh
|
||||
# Then restart nodes (no redeploy needed)
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
|
||||
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
|
||||
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
|
||||
```
|
||||
|
||||
See `.claude/skills/xce-deploy-ha/SKILL.md` for full troubleshooting reference.
|
||||
Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.
|
||||
|
||||
## Key Files
|
||||
- `src/controller/` — Controller source files
|
||||
- `include/` — Controller-specific headers
|
||||
- `src/tools/deploy-ha.sh` — HA cluster deploy script
|
||||
|
||||
## See Also
|
||||
- Shared library: `src/lib/CLAUDE.md`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue