Rongsong Shen
40f510fb14
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
141 lines
5.7 KiB
Markdown
141 lines
5.7 KiB
Markdown
# Build & Deployment
|
|
|
|
## Philosophy
|
|
|
|
A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Every build includes debug symbols + ASAN. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.
|
|
|
|
## Build Commands
|
|
|
|
All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
|
|
|
```bash
|
|
/xce-build # all components + all tests
|
|
/xce-build lib # libxce + lib tests
|
|
/xce-build controller # lib + controller + controller tests + xce_test driver
|
|
/xce-build scheduler # lib + scheduler + scheduler tests
|
|
/xce-build worker # lib + worker (daemon only, no unit tests)
|
|
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
|
/xce-build xcecli # xcecli + Go tests
|
|
```
|
|
|
|
`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
|
|
|
|
Each invocation:
|
|
1. `nix develop` into the hermetic shell
|
|
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
|
|
3. `ninja -C <dir>` (compile all + test executables)
|
|
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
|
|
|
|
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
|
|
|
|
## ASAN Integration
|
|
|
|
ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.
|
|
|
|
## Meson Structure
|
|
|
|
```
|
|
src/
|
|
├── meson.build # top-level: subdir() order, project()
|
|
├── lib/meson.build # libxce shared library + libxce_test
|
|
├── controller/meson.build # xce_controller daemon + controller_unit_test
|
|
├── worker/meson.build # xcecd daemon
|
|
├── scheduler/meson.build # scheduler daemon + test
|
|
├── autoscaler/meson.build # autoscaler daemon
|
|
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
|
|
└── frontend/xcecli/meson.build # Go xcecli via tools/go_build
|
|
```
|
|
|
|
**Dependency flow**: `lib → {controller, worker, scheduler, apiserver}`
|
|
|
|
**Test registration**:
|
|
```python
|
|
# controller/meson.build
|
|
test('controller',
|
|
controller_unit_test,
|
|
suite: 'controller',
|
|
timeout: 300,
|
|
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
|
```
|
|
|
|
**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions).
|
|
|
|
## Nix Flake
|
|
|
|
Located at `src/flake.nix`. Provides:
|
|
|
|
```bash
|
|
nix develop # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
|
|
nix build .#xcecore # C daemons + apiserver (release)
|
|
nix build .#xcecore_debug # default package: debug + ASAN
|
|
nix build .#xcecli # CLI only
|
|
```
|
|
|
|
**AppImage**: The flake produces an AppImage wrapping `xce` (shell entry point) with bundled daemons, haproxy, glow, and less. The entry script reads from `/tools/appimage/entry.sh`.
|
|
|
|
## Local HA Cluster Deployment
|
|
|
|
`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`):
|
|
|
|
```bash
|
|
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
|
```
|
|
|
|
This creates under `/tmp/xce-deploy/`:
|
|
```
|
|
certs/ # TLS certs for all peers
|
|
controller/
|
|
ctrl-1/
|
|
conf/{current, cluster.bin} # primary's config
|
|
data/ # WAL + checkpoint storage
|
|
ctrl-2/
|
|
conf/ # receives config via OPCode_replConfigPush
|
|
data/
|
|
ctrl-3/ (same as ctrl-2)
|
|
logs/ # per-node log files
|
|
scripts/
|
|
start.sh ctrl-N # launch a single node
|
|
cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs
|
|
```
|
|
|
|
**Startup** (from deploy dir):
|
|
```bash
|
|
VERBOSE=-vvv scripts/start.sh ctrl-1 & # primary first
|
|
sleep 0.5
|
|
VERBOSE=-vvv scripts/start.sh ctrl-2 & # standby (0.5s stagger for election ordering)
|
|
sleep 0.5
|
|
VERBOSE=-vvv scripts/start.sh ctrl-3 &
|
|
```
|
|
|
|
**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
|
|
```bash
|
|
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
|
bash scripts/cleanup.sh
|
|
# Then re-start all nodes
|
|
```
|
|
|
|
## Go Frontend Build
|
|
|
|
Go builds use `src/frontend/Makefile` for development, Meson for production:
|
|
|
|
```bash
|
|
make build-server # → /tmp/apiserver
|
|
make build-cli # → /tmp/xce_static (CGO_ENABLED=0)
|
|
make test # unit tests (CGO_ENABLED=0)
|
|
make test-backend # CGO-backed backend tests (needs nix dev shell)
|
|
make test-integration # full regression suite (needs XCE_BIN + SRV_URL)
|
|
```
|
|
|
|
Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.
|
|
|
|
## Production Artifacts
|
|
|
|
| Artifact | Build command | Output |
|
|
|----------|-------------|--------|
|
|
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
|
|
| Worker daemon | same | `xcecd` binary |
|
|
| Scheduler daemon | same | `xce_scheduler` binary |
|
|
| REST API server | same | `apiserver` binary |
|
|
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
|
|
| AppImage | `nix build .#xcecore` | `xce-x86_64.AppImage` |
|
|
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |
|