From 7a922cf184d2b36eeea3585b9630960ebc17e4e4 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Fri, 22 May 2026 09:20:29 +0000 Subject: [PATCH] =?UTF-8?q?docs(config-repl):=20update=20impl-status=20?= =?UTF-8?q?=E2=80=94=20Phase=201=20wire=20protocol=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Mark Phase 1 as WIRE PROTOCOL COMPLETE (was: ACTIVE with stubs) - Add commits 2ed7e34-5053c8d to component table - Remove resolved limitations (do_one_config_fetch stub, push no-op) - Update test count: 18 → 24 tests, total 84 passing - Update Phase 2 table: handle_upgrade_config wired, push timeout enforced - Update files changed to include request.c shims and new test additions --- src/docs/config-replication-impl-status.md | 44 +++++++++++++--------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/src/docs/config-replication-impl-status.md b/src/docs/config-replication-impl-status.md index 30f544c..ee89f12 100644 --- a/src/docs/config-replication-impl-status.md +++ b/src/docs/config-replication-impl-status.md @@ -1,15 +1,17 @@ # Config Replication — Implementation Status > **Branch**: `persistent_ha` -> **Date**: 2026-05-13 (updated) +> **Date**: 2026-05-22 (updated) > **Design doc**: `config-replication-design.md` --- -## Phase 1 (Pull-only replication) — ACTIVE +## Phase 1 (Pull-only replication) — WIRE PROTOCOL COMPLETE -Phase 1 is fully implemented and active on merge. Standbys fetch `ClusterConfig` -from the primary on every startup, eliminating manual per-node provisioning. +Phase 1 is fully implemented. Standbys fetch `ClusterConfig` from the primary on +every startup over a real SSL connection, eliminating manual per-node provisioning. +The wire protocol (capnp encode/decode, SHA-256 verify, disk save) is complete as +of 2026-05-22 (commits `2ed7e34`–`5053c8d`). | Component | Status | Commit | Notes | |---|---|---|---| @@ -17,14 +19,16 @@ from the primary on every startup, eliminating manual per-node provisioning. | Struct fields (`myconf_t`, `cluster_t`, `ha_state_t`, `persist_t`) | ✅ Done | `2c375cf` | `config_version_tag`, `config_sha256`, `conf_keep_n`, `conf_stale_timeout`, `restart_requested`, `standby_endpoints`, `wal_record_count`, `checkpoint_in_progress` | | `byteorder.h` portability header | ✅ Done | `2c375cf` | `be64toh()` for glibc + musl + macOS | | HEAD-file config storage (`config_repl.c` Phase 1) | ✅ Done | `491c255` | `conf_save_config`, `conf_read_current_filename`, `conf_read_raw`, `conf_get_saved_tag`, `conf_check_stale`, `conf_trim_old_versions` | +| `conf_save_config()` path fix | ✅ Done | `2ed7e34` | Was writing to `workdir/conf`; fixed to `workdir/name/conf` to match `load_primary_config()` | | `main.c` — null `conffile`, `--conf-keep-n`, `--conf-stale-timeout` | ✅ Done | `e74fd2a` | Stack buffer removed; conffile heap-allocated at runtime | | `load_primary_config()` + `myconf_fini()` | ✅ Done | `cd02cf4` | FATAL on missing config; SHA-256 cached; reserved-name check | | Real `am_i_primary()` — election probing | ✅ Done | `fe3cc57` | Non-blocking connect + poll; priority step + xorshift64 jitter; `ha_is_primary()`, `ha_check_priority_stepdown()` | | SSL double-call fix + TOCTOU fix | ✅ Done | `ec3cab4` | `SSL_get_error` called once; `ha->lock` held in stepdown | | `cluster_init` phase split + `myapp_run()` two-phase flow | ✅ Done | `6b06aa0` | Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; `am_i_primary` gates config load | | `ha->role` set before Phase 2 | ✅ Done | `93b8616` | Election result stored via `atomic_store` before `cluster_init_phase2` | -| `handle_config_request()` (primary handler) | ✅ Done | `8f4db30` | Tag compare, SHA-256 double-check, sends config bytes | -| `fetch_config_from_primary()` (standby fetch) | ✅ Done | `8f4db30` | 20-attempt exponential backoff; falls back to saved config; FATAL if none | +| Raw capnp shims (`request.c`) | ✅ Done | `26393e9` | `xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` — bypass `xce_data_t` for raw bytes | +| `handle_config_request()` (primary handler) | ✅ Done | `4ef2089` | Tag compare, SHA-256 double-check; sends real bytes via `xce_replconfig_reply_raw()` | +| `fetch_config_from_primary()` + `do_one_config_fetch()` | ✅ Done | `2aae45f` | Full SSL roundtrip: non-blocking connect+poll, `xce_replconfig_req_ssl()`, SHA-256 verify, `conf_save_config()`, reload; 20-attempt exponential backoff | | `handle_controller_join()` | ✅ Done | `8f4db30` | Registers endpoint in `standby_endpoints` under lock | | WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records | | `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired | @@ -41,8 +45,8 @@ design doc). | Component | Status | Notes | |---|---|---| -| `handle_upgrade_config()` | ✅ Compiled, gated | SHA-256 transit check, Cap'n Proto decode, generation increment, disk save, restart signal. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` | -| `push_config_to_standbys()` | ⚠ Stub (returns 0) | Wire protocol for `OP_replConfigPush` deferred — standbys catch up via pull on reconnect | +| `handle_upgrade_config()` | ✅ Wired, gated | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` | +| `push_config_to_standbys()` | ✅ Timeout enforced | `f3babec` — `wait_all_with_timeout()` with `CONF_PUSH_TIMEOUT_MS` (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect | | `create_lost_and_found()` | ✅ Done | `91e66b8` | Inserts synthetic queue via `create_queue()` and host via `makeWorker()` before `persist_recover()`; gated with `#ifndef XCE_TESTING` for unit test binary | **Prerequisites before enabling** (`CONF_ENABLE_UPGRADE_CONFIG=1`): @@ -55,13 +59,14 @@ design doc). | Suite | Tests | Covers | |---|---|---| -| `test_config_repl` | 18 | Storage functions, SHA-256, trim, stale, sanitise_cn, path resolution | +| `test_config_repl` | 24 | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips | | `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire | | `test_ha_role` | (existing) | Role transitions, watchdog, redirect | | `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default | | `test_ha_cluster` | 5 | Cluster HA integration | | `test_p1_integration` | 5 | Phase 1 init structure | -| All other suites | 60 total | 0 failures | +| All other suites | ~41 | 0 failures | +| **Total** | **84** | **0 failures, 0 ignored** | --- @@ -69,10 +74,10 @@ design doc). | Item | Severity | Tracked | |---|---|---| -| `push_config_to_standbys()` is a no-op — standbys get new config on reconnect pull, not pushed immediately | LOW | `FUTURE_WORK.md` | -| `do_one_config_fetch()` stub burns ~47s on standby startup (all 20 retry attempts hit -ENOSYS) until fully wired | MEDIUM | Fix: wire `send_repl_config_request` over the pre-Phase-2 SSL connection | -| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority step only | LOW | `FUTURE_WORK.md §7` | +| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority-step delay only | LOW | `FUTURE_WORK.md §7` | | Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` | +| `OP_replConfigPush` standby-side handler not implemented — pushed config accepted only via pull on reconnect | LOW | `FUTURE_WORK.md` | +| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` | LOW | `FUTURE_WORK.md` | --- @@ -88,21 +93,24 @@ design doc). | `src/include/byteorder.h` | **New** — portable `be64toh()` | | `src/include/config.h` | +`DEFAULT_MAX_WAL_RECORDS 10000` | | `src/include/typemap.h` | +7 union arms for new message types | -| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations | -| `src/controller/config_repl.c` | **New** — ~600 LOC: storage, SHA-256, fetch, handlers | +| `src/include/request.h` | +4 raw shim declarations; `SSL*` parameters | +| `src/lib/request.c` | +`xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` | +| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations; `handle_upgrade_config` raw-pkt params | +| `src/controller/config_repl.c` | **New** — ~900 LOC: storage, SHA-256, full SSL fetch, handlers, push timeout, upgrade wiring | | `src/controller/ha.c` | Real `am_i_primary()`, `probe_thread()`, `ha_is_primary()`, `ha_check_priority_stepdown()`, `ha_seed_jitter()` | | `src/controller/myconf.c` | `load_primary_config()`, `myconf_fini()`, static `cf` global removed | -| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; `standby_endpoints` pruning; `restart_requested` check | +| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; pre-decode peek for `OP_upgradeConfig`; `standby_endpoints` pruning; `restart_requested` check | | `src/controller/myapp.c` | Two-phase startup; `am_i_primary` gates config load; `myconf_fini` on all exits | | `src/controller/join.c` | `handle_controller_join` implemented (was `-ENOSYS`) | | `src/controller/persist.c` | `schedule_checkpoint()`, WAL record-count trigger, `persist_init` defaults | | `src/controller/main.c` | `conffile=NULL`; `--conf-keep-n`; `--conf-stale-timeout`; peers validation | | `src/controller/meson.build` | `config_repl.c` added; `ssl_dep` for unit tests | | `src/controller/include/cluster.h` | `cluster_init_phase1/2` declared | -| `src/controller/testsuite/test_config_repl.c` | **New** — 18 unit tests | +| `src/controller/testsuite/test_config_repl.c` | 24 unit tests (was 18; +path consistency, +push timeout, +capnp roundtrips) | | `src/controller/testsuite/test_ha_role.c` | +3 election probing tests | | `src/controller/testsuite/test_persist_wal.c` | +3 WAL bound tests | | `src/controller/testsuite/controller_test.c` | New suites registered | | `src/frontend/apiserver/controllers/jobs.go` | +reserved-name guard in `preCheckJobRequest()` | | `src/frontend/apiserver/controllers/jobs_test.go` | **New** — 3 tests: lost_and_found rejected, normal queue passes, empty queue passes | -| `src/docs/config-replication-impl-status.md` | **New** — this file | +| `src/docs/config-replication-design.md` | §3.2 `effective_dir` definition; §6.3 timeout implementation note | +| `src/docs/config-replication-impl-status.md` | This file |