Rongsong Shen
7a922cf184
- Mark Phase 1 as WIRE PROTOCOL COMPLETE (was: ACTIVE with stubs) - Add commits 2ed7e34-5053c8d to component table - Remove resolved limitations (do_one_config_fetch stub, push no-op) - Update test count: 18 → 24 tests, total 84 passing - Update Phase 2 table: handle_upgrade_config wired, push timeout enforced - Update files changed to include request.c shims and new test additions
8.9 KiB
8.9 KiB
Config Replication — Implementation Status
Branch:
persistent_haDate: 2026-05-22 (updated) Design doc:config-replication-design.md
Phase 1 (Pull-only replication) — WIRE PROTOCOL COMPLETE
Phase 1 is fully implemented. Standbys fetch ClusterConfig from the primary on
every startup over a real SSL connection, eliminating manual per-node provisioning.
The wire protocol (capnp encode/decode, SHA-256 verify, disk save) is complete as
of 2026-05-22 (commits 2ed7e34–5053c8d).
| Component | Status | Commit | Notes |
|---|---|---|---|
| Cap'n Proto schema additions | ✅ Done | 683c092 |
3 opcodes, 6 structs, configGeneration, HeartbeatMsg |
Struct fields (myconf_t, cluster_t, ha_state_t, persist_t) |
✅ Done | 2c375cf |
config_version_tag, config_sha256, conf_keep_n, conf_stale_timeout, restart_requested, standby_endpoints, wal_record_count, checkpoint_in_progress |
byteorder.h portability header |
✅ Done | 2c375cf |
be64toh() for glibc + musl + macOS |
HEAD-file config storage (config_repl.c Phase 1) |
✅ Done | 491c255 |
conf_save_config, conf_read_current_filename, conf_read_raw, conf_get_saved_tag, conf_check_stale, conf_trim_old_versions |
conf_save_config() path fix |
✅ Done | 2ed7e34 |
Was writing to workdir/conf; fixed to workdir/name/conf to match load_primary_config() |
main.c — null conffile, --conf-keep-n, --conf-stale-timeout |
✅ Done | e74fd2a |
Stack buffer removed; conffile heap-allocated at runtime |
load_primary_config() + myconf_fini() |
✅ Done | cd02cf4 |
FATAL on missing config; SHA-256 cached; reserved-name check |
Real am_i_primary() — election probing |
✅ Done | fe3cc57 |
Non-blocking connect + poll; priority step + xorshift64 jitter; ha_is_primary(), ha_check_priority_stepdown() |
| SSL double-call fix + TOCTOU fix | ✅ Done | ec3cab4 |
SSL_get_error called once; ha->lock held in stepdown |
cluster_init phase split + myapp_run() two-phase flow |
✅ Done | 6b06aa0 |
Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; am_i_primary gates config load |
ha->role set before Phase 2 |
✅ Done | 93b8616 |
Election result stored via atomic_store before cluster_init_phase2 |
Raw capnp shims (request.c) |
✅ Done | 26393e9 |
xce_replconfig_reply_raw(), xce_replconfig_req_ssl(), xce_replconfig_reply_ssl_decode(), xce_upgrade_config_req_extract_raw() — bypass xce_data_t for raw bytes |
handle_config_request() (primary handler) |
✅ Done | 4ef2089 |
Tag compare, SHA-256 double-check; sends real bytes via xce_replconfig_reply_raw() |
fetch_config_from_primary() + do_one_config_fetch() |
✅ Done | 2aae45f |
Full SSL roundtrip: non-blocking connect+poll, xce_replconfig_req_ssl(), SHA-256 verify, conf_save_config(), reload; 20-attempt exponential backoff |
handle_controller_join() |
✅ Done | 8f4db30 |
Registers endpoint in standby_endpoints under lock |
WAL record-count schedule_checkpoint() |
✅ Done | a95476b |
Idempotent CAS; coalesced via delaymgr 1s delay; bounds persist_recover to ≤10k records |
typemap.h union arms + capnp shims |
✅ Done | 491c255 |
replconfigreplymsg, upgradeconfigreplymsg, etc. wired |
API server rejects lost_and_found job submissions |
✅ Done | bc92166 |
preCheckJobRequest() returns "internal recovery queue" error before store lookup; 3 tests in jobs_test.go |
Phase 2 (Remote config push) — COMPILED, GATED
Phase 2 infrastructure ships compiled but inert. The CONF_ENABLE_UPGRADE_CONFIG
compile-time gate (default: 0) prevents OP_upgradeConfig from being reachable
in production builds until both prerequisites are met (see §5.3 and §2 of the
design doc).
| Component | Status | Notes |
|---|---|---|
handle_upgrade_config() |
✅ Wired, gated | 5053c8d — pre-decode opcode peek in myreq_handler() captures raw packet bytes; xce_upgrade_config_req_extract_raw() extracts sha256+bytes; delegates to do_upgrade_config_with_bytes(). Active only when CONF_ENABLE_UPGRADE_CONFIG=1 |
push_config_to_standbys() |
✅ Timeout enforced | f3babec — wait_all_with_timeout() with CONF_PUSH_TIMEOUT_MS (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect |
create_lost_and_found() |
✅ Done | 91e66b8 |
Prerequisites before enabling (CONF_ENABLE_UPGRADE_CONFIG=1):
validate_host()implemented with real CN allowlist check (seeFUTURE_WORK.md §6)- Admin config tool exists and tested end-to-end (see
FUTURE_WORK.md §1)
Test Coverage
| Suite | Tests | Covers |
|---|---|---|
test_config_repl |
24 | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips |
test_am_i_primary |
3 | Priority-0 immediate, priority-0 with peers, deadline-expire |
test_ha_role |
(existing) | Role transitions, watchdog, redirect |
test_persist_wal |
6 (+3 new) | WAL append, CAS idempotency, max_wal_records default |
test_ha_cluster |
5 | Cluster HA integration |
test_p1_integration |
5 | Phase 1 init structure |
| All other suites | ~41 | 0 failures |
| Total | 84 | 0 failures, 0 ignored |
Known Limitations / Deferred Work
| Item | Severity | Tracked |
|---|---|---|
Heartbeat listener (start_heartbeat_listener) is a no-op stub — simultaneous-boot symmetry relies on priority-step delay only |
LOW | FUTURE_WORK.md §7 |
Peers cross-validation hash in OP_heartbeat not implemented |
LOW | FUTURE_WORK.md |
OP_replConfigPush standby-side handler not implemented — pushed config accepted only via pull on reconnect |
LOW | FUTURE_WORK.md |
WAL routing helpers (resolve_queue_or_lost_and_found etc.) not wired in apply_wal_entry() |
LOW | FUTURE_WORK.md |
Files Changed Summary
| File | Change |
|---|---|
src/schema/controller.capnp |
+3 opcodes, +6 structs, configGeneration, HeartbeatMsg |
src/include/myconf.h |
+6 fields to myconf_t; load_primary_config, fetch_config_from_primary, myconf_fini declared |
src/include/xce/cluster_def.h |
+4 fields to cluster_t |
src/controller/ha.h |
+probe_ctx_t; +3 fields to ha_state_t; +2 fields to ha_config_t; xorshift64_next inline |
src/include/persist.h |
+max_wal_records; +wal_record_count, checkpoint_in_progress atomics |
src/include/byteorder.h |
New — portable be64toh() |
src/include/config.h |
+DEFAULT_MAX_WAL_RECORDS 10000 |
src/include/typemap.h |
+7 union arms for new message types |
src/include/request.h |
+4 raw shim declarations; SSL* parameters |
src/lib/request.c |
+xce_replconfig_reply_raw(), xce_replconfig_req_ssl(), xce_replconfig_reply_ssl_decode(), xce_upgrade_config_req_extract_raw() |
src/controller/config_repl.h |
New — Phase 1+2 API declarations; handle_upgrade_config raw-pkt params |
src/controller/config_repl.c |
New — ~900 LOC: storage, SHA-256, full SSL fetch, handlers, push timeout, upgrade wiring |
src/controller/ha.c |
Real am_i_primary(), probe_thread(), ha_is_primary(), ha_check_priority_stepdown(), ha_seed_jitter() |
src/controller/myconf.c |
load_primary_config(), myconf_fini(), static cf global removed |
src/controller/cluster.c |
cluster_init_phase1/2 split; opcode dispatch; pre-decode peek for OP_upgradeConfig; standby_endpoints pruning; restart_requested check |
src/controller/myapp.c |
Two-phase startup; am_i_primary gates config load; myconf_fini on all exits |
src/controller/join.c |
handle_controller_join implemented (was -ENOSYS) |
src/controller/persist.c |
schedule_checkpoint(), WAL record-count trigger, persist_init defaults |
src/controller/main.c |
conffile=NULL; --conf-keep-n; --conf-stale-timeout; peers validation |
src/controller/meson.build |
config_repl.c added; ssl_dep for unit tests |
src/controller/include/cluster.h |
cluster_init_phase1/2 declared |
src/controller/testsuite/test_config_repl.c |
24 unit tests (was 18; +path consistency, +push timeout, +capnp roundtrips) |
src/controller/testsuite/test_ha_role.c |
+3 election probing tests |
src/controller/testsuite/test_persist_wal.c |
+3 WAL bound tests |
src/controller/testsuite/controller_test.c |
New suites registered |
src/frontend/apiserver/controllers/jobs.go |
+reserved-name guard in preCheckJobRequest() |
src/frontend/apiserver/controllers/jobs_test.go |
New — 3 tests: lost_and_found rejected, normal queue passes, empty queue passes |
src/docs/config-replication-design.md |
§3.2 effective_dir definition; §6.3 timeout implementation note |
src/docs/config-replication-impl-status.md |
This file |