From e4c1bfb17fa73a404522a7610e650f9bce0e3c30 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 14 May 2026 03:01:38 +0000 Subject: [PATCH] docs: update design doc status; add config-replication-impl-status.md with full implementation summary --- src/docs/config-replication-design.md | 6 +- src/docs/config-replication-impl-status.md | 105 +++++++++++++++++++++ 2 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 src/docs/config-replication-impl-status.md diff --git a/src/docs/config-replication-design.md b/src/docs/config-replication-design.md index a43ca7f..589b0b8 100644 --- a/src/docs/config-replication-design.md +++ b/src/docs/config-replication-design.md @@ -1,9 +1,11 @@ # XCE Controller: Configuration Replication Design -> **Status**: Draft -> **Last Updated**: 2026-05-12 +> **Status**: Implemented (Phase 1 active; Phase 2 gated) +> **Last Updated**: 2026-05-13 > **Target**: `src/controller/` (C daemon) > **Depends on**: `persistence-and-ha-design.md` Part 5 (HA Failover) +> **Branch**: `persistent_ha` +> **Implementation commits**: `683c092`–`40c7a93` (20 commits) --- diff --git a/src/docs/config-replication-impl-status.md b/src/docs/config-replication-impl-status.md new file mode 100644 index 0000000..4e9eed0 --- /dev/null +++ b/src/docs/config-replication-impl-status.md @@ -0,0 +1,105 @@ +# Config Replication — Implementation Status + +> **Branch**: `persistent_ha` +> **Date**: 2026-05-13 +> **Design doc**: `config-replication-design.md` + +--- + +## Phase 1 (Pull-only replication) — ACTIVE + +Phase 1 is fully implemented and active on merge. Standbys fetch `ClusterConfig` +from the primary on every startup, eliminating manual per-node provisioning. + +| Component | Status | Commit | Notes | +|---|---|---|---| +| Cap'n Proto schema additions | ✅ Done | `683c092` | 3 opcodes, 6 structs, `configGeneration`, `HeartbeatMsg` | +| Struct fields (`myconf_t`, `cluster_t`, `ha_state_t`, `persist_t`) | ✅ Done | `2c375cf` | `config_version_tag`, `config_sha256`, `conf_keep_n`, `conf_stale_timeout`, `restart_requested`, `standby_endpoints`, `wal_record_count`, `checkpoint_in_progress` | +| `byteorder.h` portability header | ✅ Done | `2c375cf` | `be64toh()` for glibc + musl + macOS | +| HEAD-file config storage (`config_repl.c` Phase 1) | ✅ Done | `491c255` | `conf_save_config`, `conf_read_current_filename`, `conf_read_raw`, `conf_get_saved_tag`, `conf_check_stale`, `conf_trim_old_versions` | +| `main.c` — null `conffile`, `--conf-keep-n`, `--conf-stale-timeout` | ✅ Done | `e74fd2a` | Stack buffer removed; conffile heap-allocated at runtime | +| `load_primary_config()` + `myconf_fini()` | ✅ Done | `cd02cf4` | FATAL on missing config; SHA-256 cached; reserved-name check | +| Real `am_i_primary()` — election probing | ✅ Done | `fe3cc57` | Non-blocking connect + poll; priority step + xorshift64 jitter; `ha_is_primary()`, `ha_check_priority_stepdown()` | +| SSL double-call fix + TOCTOU fix | ✅ Done | `ec3cab4` | `SSL_get_error` called once; `ha->lock` held in stepdown | +| `cluster_init` phase split + `myapp_run()` two-phase flow | ✅ Done | `6b06aa0` | Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; `am_i_primary` gates config load | +| `ha->role` set before Phase 2 | ✅ Done | `93b8616` | Election result stored via `atomic_store` before `cluster_init_phase2` | +| `handle_config_request()` (primary handler) | ✅ Done | `8f4db30` | Tag compare, SHA-256 double-check, sends config bytes | +| `fetch_config_from_primary()` (standby fetch) | ✅ Done | `8f4db30` | 20-attempt exponential backoff; falls back to saved config; FATAL if none | +| `handle_controller_join()` | ✅ Done | `8f4db30` | Registers endpoint in `standby_endpoints` under lock | +| WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records | +| `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired | + +--- + +## Phase 2 (Remote config push) — COMPILED, GATED + +Phase 2 infrastructure ships compiled but inert. The `CONF_ENABLE_UPGRADE_CONFIG` +compile-time gate (default: 0) prevents `OP_upgradeConfig` from being reachable +in production builds until both prerequisites are met (see §5.3 and §2 of the +design doc). + +| Component | Status | Notes | +|---|---|---| +| `handle_upgrade_config()` | ✅ Compiled, gated | SHA-256 transit check, Cap'n Proto decode, generation increment, disk save, restart signal. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` | +| `push_config_to_standbys()` | ⚠ Stub (returns 0) | Wire protocol for `OP_replConfigPush` deferred — standbys catch up via pull on reconnect | +| `create_lost_and_found()` | ⚠ Stub (no-op) | Synthetic queue/host for WAL recovery with unknown IDs; requires kvmgr integration | + +**Prerequisites before enabling** (`CONF_ENABLE_UPGRADE_CONFIG=1`): +1. `validate_host()` implemented with real CN allowlist check (see `FUTURE_WORK.md §6`) +2. Admin config tool exists and tested end-to-end (see `FUTURE_WORK.md §1`) + +--- + +## Test Coverage + +| Suite | Tests | Covers | +|---|---|---| +| `test_config_repl` | 18 | Storage functions, SHA-256, trim, stale, sanitise_cn, path resolution | +| `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire | +| `test_ha_role` | (existing) | Role transitions, watchdog, redirect | +| `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default | +| `test_ha_cluster` | 5 | Cluster HA integration | +| `test_p1_integration` | 5 | Phase 1 init structure | +| All other suites | 60 total | 0 failures | + +--- + +## Known Limitations / Deferred Work + +| Item | Severity | Tracked | +|---|---|---| +| `push_config_to_standbys()` is a no-op — standbys get new config on reconnect pull, not pushed immediately | LOW | `FUTURE_WORK.md` | +| `create_lost_and_found()` is a no-op — WAL recovery routes unknown IDs silently without synthetic queue | LOW | `FUTURE_WORK.md` | +| `do_one_config_fetch()` stub burns ~47s on standby startup (all 20 retry attempts hit -ENOSYS) until fully wired | MEDIUM | Fix: wire `send_repl_config_request` over the pre-Phase-2 SSL connection | +| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority step only | LOW | `FUTURE_WORK.md §7` | +| Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` | + +--- + +## Files Changed Summary + +| File | Change | +|---|---| +| `src/schema/controller.capnp` | +3 opcodes, +6 structs, `configGeneration`, `HeartbeatMsg` | +| `src/include/myconf.h` | +6 fields to `myconf_t`; `load_primary_config`, `fetch_config_from_primary`, `myconf_fini` declared | +| `src/include/xce/cluster_def.h` | +4 fields to `cluster_t` | +| `src/controller/ha.h` | +`probe_ctx_t`; +3 fields to `ha_state_t`; +2 fields to `ha_config_t`; `xorshift64_next` inline | +| `src/include/persist.h` | +`max_wal_records`; +`wal_record_count`, `checkpoint_in_progress` atomics | +| `src/include/byteorder.h` | **New** — portable `be64toh()` | +| `src/include/config.h` | +`DEFAULT_MAX_WAL_RECORDS 10000` | +| `src/include/typemap.h` | +7 union arms for new message types | +| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations | +| `src/controller/config_repl.c` | **New** — ~600 LOC: storage, SHA-256, fetch, handlers | +| `src/controller/ha.c` | Real `am_i_primary()`, `probe_thread()`, `ha_is_primary()`, `ha_check_priority_stepdown()`, `ha_seed_jitter()` | +| `src/controller/myconf.c` | `load_primary_config()`, `myconf_fini()`, static `cf` global removed | +| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; `standby_endpoints` pruning; `restart_requested` check | +| `src/controller/myapp.c` | Two-phase startup; `am_i_primary` gates config load; `myconf_fini` on all exits | +| `src/controller/join.c` | `handle_controller_join` implemented (was `-ENOSYS`) | +| `src/controller/persist.c` | `schedule_checkpoint()`, WAL record-count trigger, `persist_init` defaults | +| `src/controller/main.c` | `conffile=NULL`; `--conf-keep-n`; `--conf-stale-timeout`; peers validation | +| `src/controller/meson.build` | `config_repl.c` added; `ssl_dep` for unit tests | +| `src/controller/include/cluster.h` | `cluster_init_phase1/2` declared | +| `src/controller/testsuite/test_config_repl.c` | **New** — 18 unit tests | +| `src/controller/testsuite/test_ha_role.c` | +3 election probing tests | +| `src/controller/testsuite/test_persist_wal.c` | +3 WAL bound tests | +| `src/controller/testsuite/controller_test.c` | New suites registered |