lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: update design doc status; add config-replication-impl-status.md with full implementation summary

Browse source
This commit is contained in:
Rongsong Shen 2026-05-14 03:01:38 +00:00
parent 40c7a93951
commit e4c1bfb17f
2 changed files with 109 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

6
src/docs/config-replication-design.md
View file

@ -1,9 +1,11 @@
# XCE Controller: Configuration Replication Design
> **Status**: Draft
> **Last Updated**: 2026-05-12
> **Status**: Implemented (Phase 1 active; Phase 2 gated)
> **Last Updated**: 2026-05-13
> **Target**: `src/controller/` (C daemon)
> **Depends on**: `persistence-and-ha-design.md` Part 5 (HA Failover)
> **Branch**: `persistent_ha`
> **Implementation commits**: `683c092`–`40c7a93` (20 commits)
---

105
src/docs/config-replication-impl-status.md Normal file
View file

@ -0,0 +1,105 @@
# Config Replication — Implementation Status
> **Branch**: `persistent_ha`
> **Date**: 2026-05-13
> **Design doc**: `config-replication-design.md`
---
## Phase 1 (Pull-only replication) — ACTIVE
Phase 1 is fully implemented and active on merge. Standbys fetch `ClusterConfig`
from the primary on every startup, eliminating manual per-node provisioning.
| Component | Status | Commit | Notes |
|---|---|---|---|
| Cap'n Proto schema additions | ✅ Done | `683c092` | 3 opcodes, 6 structs, `configGeneration`, `HeartbeatMsg` |
| Struct fields (`myconf_t`, `cluster_t`, `ha_state_t`, `persist_t`) | ✅ Done | `2c375cf` | `config_version_tag`, `config_sha256`, `conf_keep_n`, `conf_stale_timeout`, `restart_requested`, `standby_endpoints`, `wal_record_count`, `checkpoint_in_progress` |
| `byteorder.h` portability header | ✅ Done | `2c375cf` | `be64toh()` for glibc + musl + macOS |
| HEAD-file config storage (`config_repl.c` Phase 1) | ✅ Done | `491c255` | `conf_save_config`, `conf_read_current_filename`, `conf_read_raw`, `conf_get_saved_tag`, `conf_check_stale`, `conf_trim_old_versions` |
| `main.c` — null `conffile`, `--conf-keep-n`, `--conf-stale-timeout` | ✅ Done | `e74fd2a` | Stack buffer removed; conffile heap-allocated at runtime |
| `load_primary_config()` + `myconf_fini()` | ✅ Done | `cd02cf4` | FATAL on missing config; SHA-256 cached; reserved-name check |
| Real `am_i_primary()` — election probing | ✅ Done | `fe3cc57` | Non-blocking connect + poll; priority step + xorshift64 jitter; `ha_is_primary()`, `ha_check_priority_stepdown()` |
| SSL double-call fix + TOCTOU fix | ✅ Done | `ec3cab4` | `SSL_get_error` called once; `ha->lock` held in stepdown |
| `cluster_init` phase split + `myapp_run()` two-phase flow | ✅ Done | `6b06aa0` | Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; `am_i_primary` gates config load |
| `ha->role` set before Phase 2 | ✅ Done | `93b8616` | Election result stored via `atomic_store` before `cluster_init_phase2` |
| `handle_config_request()` (primary handler) | ✅ Done | `8f4db30` | Tag compare, SHA-256 double-check, sends config bytes |
| `fetch_config_from_primary()` (standby fetch) | ✅ Done | `8f4db30` | 20-attempt exponential backoff; falls back to saved config; FATAL if none |
| `handle_controller_join()` | ✅ Done | `8f4db30` | Registers endpoint in `standby_endpoints` under lock |
| WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records |
| `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired |
---
## Phase 2 (Remote config push) — COMPILED, GATED
Phase 2 infrastructure ships compiled but inert. The `CONF_ENABLE_UPGRADE_CONFIG`
compile-time gate (default: 0) prevents `OP_upgradeConfig` from being reachable
in production builds until both prerequisites are met (see §5.3 and §2 of the
design doc).
| Component | Status | Notes |
|---|---|---|
| `handle_upgrade_config()` | ✅ Compiled, gated | SHA-256 transit check, Cap'n Proto decode, generation increment, disk save, restart signal. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` |
| `push_config_to_standbys()` | ⚠ Stub (returns 0) | Wire protocol for `OP_replConfigPush` deferred — standbys catch up via pull on reconnect |
| `create_lost_and_found()` | ⚠ Stub (no-op) | Synthetic queue/host for WAL recovery with unknown IDs; requires kvmgr integration |
**Prerequisites before enabling** (`CONF_ENABLE_UPGRADE_CONFIG=1`):
1. `validate_host()` implemented with real CN allowlist check (see `FUTURE_WORK.md §6`)
2. Admin config tool exists and tested end-to-end (see `FUTURE_WORK.md §1`)
---
## Test Coverage
| Suite | Tests | Covers |
|---|---|---|
| `test_config_repl` | 18 | Storage functions, SHA-256, trim, stale, sanitise_cn, path resolution |
| `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire |
| `test_ha_role` | (existing) | Role transitions, watchdog, redirect |
| `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default |
| `test_ha_cluster` | 5 | Cluster HA integration |
| `test_p1_integration` | 5 | Phase 1 init structure |
| All other suites | 60 total | 0 failures |
---
## Known Limitations / Deferred Work
| Item | Severity | Tracked |
|---|---|---|
| `push_config_to_standbys()` is a no-op — standbys get new config on reconnect pull, not pushed immediately | LOW | `FUTURE_WORK.md` |
| `create_lost_and_found()` is a no-op — WAL recovery routes unknown IDs silently without synthetic queue | LOW | `FUTURE_WORK.md` |
| `do_one_config_fetch()` stub burns ~47s on standby startup (all 20 retry attempts hit -ENOSYS) until fully wired | MEDIUM | Fix: wire `send_repl_config_request` over the pre-Phase-2 SSL connection |
| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority step only | LOW | `FUTURE_WORK.md §7` |
| Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` |
---
## Files Changed Summary
| File | Change |
|---|---|
| `src/schema/controller.capnp` | +3 opcodes, +6 structs, `configGeneration`, `HeartbeatMsg` |
| `src/include/myconf.h` | +6 fields to `myconf_t`; `load_primary_config`, `fetch_config_from_primary`, `myconf_fini` declared |
| `src/include/xce/cluster_def.h` | +4 fields to `cluster_t` |
| `src/controller/ha.h` | +`probe_ctx_t`; +3 fields to `ha_state_t`; +2 fields to `ha_config_t`; `xorshift64_next` inline |
| `src/include/persist.h` | +`max_wal_records`; +`wal_record_count`, `checkpoint_in_progress` atomics |
| `src/include/byteorder.h` | **New** — portable `be64toh()` |
| `src/include/config.h` | +`DEFAULT_MAX_WAL_RECORDS 10000` |
| `src/include/typemap.h` | +7 union arms for new message types |
| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations |
| `src/controller/config_repl.c` | **New** — ~600 LOC: storage, SHA-256, fetch, handlers |
| `src/controller/ha.c` | Real `am_i_primary()`, `probe_thread()`, `ha_is_primary()`, `ha_check_priority_stepdown()`, `ha_seed_jitter()` |
| `src/controller/myconf.c` | `load_primary_config()`, `myconf_fini()`, static `cf` global removed |
| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; `standby_endpoints` pruning; `restart_requested` check |
| `src/controller/myapp.c` | Two-phase startup; `am_i_primary` gates config load; `myconf_fini` on all exits |
| `src/controller/join.c` | `handle_controller_join` implemented (was `-ENOSYS`) |
| `src/controller/persist.c` | `schedule_checkpoint()`, WAL record-count trigger, `persist_init` defaults |
| `src/controller/main.c` | `conffile=NULL`; `--conf-keep-n`; `--conf-stale-timeout`; peers validation |
| `src/controller/meson.build` | `config_repl.c` added; `ssl_dep` for unit tests |
| `src/controller/include/cluster.h` | `cluster_init_phase1/2` declared |
| `src/controller/testsuite/test_config_repl.c` | **New** — 18 unit tests |
| `src/controller/testsuite/test_ha_role.c` | +3 election probing tests |
| `src/controller/testsuite/test_persist_wal.c` | +3 WAL bound tests |
| `src/controller/testsuite/controller_test.c` | New suites registered |