lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(config-repl): update impl-status — Phase 1 wire protocol complete

Browse source 
- Mark Phase 1 as WIRE PROTOCOL COMPLETE (was: ACTIVE with stubs)
- Add commits 2ed7e34-5053c8d to component table
- Remove resolved limitations (do_one_config_fetch stub, push no-op)
- Update test count: 18 → 24 tests, total 84 passing
- Update Phase 2 table: handle_upgrade_config wired, push timeout enforced
- Update files changed to include request.c shims and new test additions
This commit is contained in:
Rongsong Shen 2026-05-22 09:20:29 +00:00
parent 5053c8d42f
commit 7a922cf184
1 changed files with 26 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

44
src/docs/config-replication-impl-status.md
View file

@ -1,15 +1,17 @@
# Config Replication — Implementation Status
> **Branch**: `persistent_ha`
> **Date**: 2026-05-13 (updated)
> **Date**: 2026-05-22 (updated)
> **Design doc**: `config-replication-design.md`
---
## Phase 1 (Pull-only replication) — ACTIVE
## Phase 1 (Pull-only replication) — WIRE PROTOCOL COMPLETE
Phase 1 is fully implemented and active on merge. Standbys fetch `ClusterConfig`
from the primary on every startup, eliminating manual per-node provisioning.
Phase 1 is fully implemented. Standbys fetch `ClusterConfig` from the primary on
every startup over a real SSL connection, eliminating manual per-node provisioning.
The wire protocol (capnp encode/decode, SHA-256 verify, disk save) is complete as
of 2026-05-22 (commits `2ed7e34`–`5053c8d`).
| Component | Status | Commit | Notes |
|---|---|---|---|
@ -17,14 +19,16 @@ from the primary on every startup, eliminating manual per-node provisioning.
| Struct fields (`myconf_t`, `cluster_t`, `ha_state_t`, `persist_t`) | ✅ Done | `2c375cf` | `config_version_tag`, `config_sha256`, `conf_keep_n`, `conf_stale_timeout`, `restart_requested`, `standby_endpoints`, `wal_record_count`, `checkpoint_in_progress` |
| `byteorder.h` portability header | ✅ Done | `2c375cf` | `be64toh()` for glibc + musl + macOS |
| HEAD-file config storage (`config_repl.c` Phase 1) | ✅ Done | `491c255` | `conf_save_config`, `conf_read_current_filename`, `conf_read_raw`, `conf_get_saved_tag`, `conf_check_stale`, `conf_trim_old_versions` |
| `conf_save_config()` path fix | ✅ Done | `2ed7e34` | Was writing to `workdir/conf`; fixed to `workdir/name/conf` to match `load_primary_config()` |
| `main.c` — null `conffile`, `--conf-keep-n`, `--conf-stale-timeout` | ✅ Done | `e74fd2a` | Stack buffer removed; conffile heap-allocated at runtime |
| `load_primary_config()` + `myconf_fini()` | ✅ Done | `cd02cf4` | FATAL on missing config; SHA-256 cached; reserved-name check |
| Real `am_i_primary()` — election probing | ✅ Done | `fe3cc57` | Non-blocking connect + poll; priority step + xorshift64 jitter; `ha_is_primary()`, `ha_check_priority_stepdown()` |
| SSL double-call fix + TOCTOU fix | ✅ Done | `ec3cab4` | `SSL_get_error` called once; `ha->lock` held in stepdown |
| `cluster_init` phase split + `myapp_run()` two-phase flow | ✅ Done | `6b06aa0` | Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; `am_i_primary` gates config load |
| `ha->role` set before Phase 2 | ✅ Done | `93b8616` | Election result stored via `atomic_store` before `cluster_init_phase2` |
| `handle_config_request()` (primary handler) | ✅ Done | `8f4db30` | Tag compare, SHA-256 double-check, sends config bytes |
| `fetch_config_from_primary()` (standby fetch) | ✅ Done | `8f4db30` | 20-attempt exponential backoff; falls back to saved config; FATAL if none |
| Raw capnp shims (`request.c`) | ✅ Done | `26393e9` | `xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` — bypass `xce_data_t` for raw bytes |
| `handle_config_request()` (primary handler) | ✅ Done | `4ef2089` | Tag compare, SHA-256 double-check; sends real bytes via `xce_replconfig_reply_raw()` |
| `fetch_config_from_primary()` + `do_one_config_fetch()` | ✅ Done | `2aae45f` | Full SSL roundtrip: non-blocking connect+poll, `xce_replconfig_req_ssl()`, SHA-256 verify, `conf_save_config()`, reload; 20-attempt exponential backoff |
| `handle_controller_join()` | ✅ Done | `8f4db30` | Registers endpoint in `standby_endpoints` under lock |
| WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records |
| `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired |
@ -41,8 +45,8 @@ design doc).
| Component | Status | Notes |
|---|---|---|
| `handle_upgrade_config()` | ✅ Compiled, gated | SHA-256 transit check, Cap'n Proto decode, generation increment, disk save, restart signal. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` |
| `push_config_to_standbys()` | ⚠ Stub (returns 0) | Wire protocol for `OP_replConfigPush` deferred — standbys catch up via pull on reconnect |
| `handle_upgrade_config()` | ✅ Wired, gated | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` |
| `push_config_to_standbys()` | ✅ Timeout enforced | `f3babec` — `wait_all_with_timeout()` with `CONF_PUSH_TIMEOUT_MS` (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect |
| `create_lost_and_found()` | ✅ Done | `91e66b8` | Inserts synthetic queue via `create_queue()` and host via `makeWorker()` before `persist_recover()`; gated with `#ifndef XCE_TESTING` for unit test binary |
**Prerequisites before enabling** (`CONF_ENABLE_UPGRADE_CONFIG=1`):
@ -55,13 +59,14 @@ design doc).
| Suite | Tests | Covers |
|---|---|---|
| `test_config_repl` | 18 | Storage functions, SHA-256, trim, stale, sanitise_cn, path resolution |
| `test_config_repl` | 24 | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips |
| `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire |
| `test_ha_role` | (existing) | Role transitions, watchdog, redirect |
| `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default |
| `test_ha_cluster` | 5 | Cluster HA integration |
| `test_p1_integration` | 5 | Phase 1 init structure |
| All other suites | 60 total | 0 failures |
| All other suites | ~41 | 0 failures |
| **Total** | **84** | **0 failures, 0 ignored** |
---
@ -69,10 +74,10 @@ design doc).
| Item | Severity | Tracked |
|---|---|---|
| `push_config_to_standbys()` is a no-op — standbys get new config on reconnect pull, not pushed immediately | LOW | `FUTURE_WORK.md` |
| `do_one_config_fetch()` stub burns ~47s on standby startup (all 20 retry attempts hit -ENOSYS) until fully wired | MEDIUM | Fix: wire `send_repl_config_request` over the pre-Phase-2 SSL connection |
| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority step only | LOW | `FUTURE_WORK.md §7` |
| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority-step delay only | LOW | `FUTURE_WORK.md §7` |
| Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` |
| `OP_replConfigPush` standby-side handler not implemented — pushed config accepted only via pull on reconnect | LOW | `FUTURE_WORK.md` |
| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` | LOW | `FUTURE_WORK.md` |
---
@ -88,21 +93,24 @@ design doc).
| `src/include/byteorder.h` | **New** — portable `be64toh()` |
| `src/include/config.h` | +`DEFAULT_MAX_WAL_RECORDS 10000` |
| `src/include/typemap.h` | +7 union arms for new message types |
| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations |
| `src/controller/config_repl.c` | **New** — ~600 LOC: storage, SHA-256, fetch, handlers |
| `src/include/request.h` | +4 raw shim declarations; `SSL*` parameters |
| `src/lib/request.c` | +`xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` |
| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations; `handle_upgrade_config` raw-pkt params |
| `src/controller/config_repl.c` | **New** — ~900 LOC: storage, SHA-256, full SSL fetch, handlers, push timeout, upgrade wiring |
| `src/controller/ha.c` | Real `am_i_primary()`, `probe_thread()`, `ha_is_primary()`, `ha_check_priority_stepdown()`, `ha_seed_jitter()` |
| `src/controller/myconf.c` | `load_primary_config()`, `myconf_fini()`, static `cf` global removed |
| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; `standby_endpoints` pruning; `restart_requested` check |
| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; pre-decode peek for `OP_upgradeConfig`; `standby_endpoints` pruning; `restart_requested` check |
| `src/controller/myapp.c` | Two-phase startup; `am_i_primary` gates config load; `myconf_fini` on all exits |
| `src/controller/join.c` | `handle_controller_join` implemented (was `-ENOSYS`) |
| `src/controller/persist.c` | `schedule_checkpoint()`, WAL record-count trigger, `persist_init` defaults |
| `src/controller/main.c` | `conffile=NULL`; `--conf-keep-n`; `--conf-stale-timeout`; peers validation |
| `src/controller/meson.build` | `config_repl.c` added; `ssl_dep` for unit tests |
| `src/controller/include/cluster.h` | `cluster_init_phase1/2` declared |
| `src/controller/testsuite/test_config_repl.c` | **New** — 18 unit tests |
| `src/controller/testsuite/test_config_repl.c` | 24 unit tests (was 18; +path consistency, +push timeout, +capnp roundtrips) |
| `src/controller/testsuite/test_ha_role.c` | +3 election probing tests |
| `src/controller/testsuite/test_persist_wal.c` | +3 WAL bound tests |
| `src/controller/testsuite/controller_test.c` | New suites registered |
| `src/frontend/apiserver/controllers/jobs.go` | +reserved-name guard in `preCheckJobRequest()` |
| `src/frontend/apiserver/controllers/jobs_test.go` | **New** — 3 tests: lost_and_found rejected, normal queue passes, empty queue passes |
| `src/docs/config-replication-impl-status.md` | **New** — this file |
| `src/docs/config-replication-design.md` | §3.2 `effective_dir` definition; §6.3 timeout implementation note |
| `src/docs/config-replication-impl-status.md` | This file |