12 KiB
12 KiB
Config Replication — Implementation Status
Branch:
persistent_haDate: 2026-05-22 (updated) Design doc:config-replication-design.md
Phase 1 (Pull-only replication) — WIRE PROTOCOL COMPLETE
Phase 1 is fully implemented. Standbys fetch ClusterConfig from the primary on
every startup over a real SSL connection, eliminating manual per-node provisioning.
The wire protocol (capnp encode/decode, SHA-256 verify, disk save) is complete as
of 2026-05-22 (commits 2ed7e34–5053c8d).
| Component | Status | Commit | Notes |
|---|---|---|---|
| Cap'n Proto schema additions | ✅ Done | 683c092 |
3 opcodes, 6 structs, configGeneration, HeartbeatMsg |
Struct fields (myconf_t, cluster_t, ha_state_t, persist_t) |
✅ Done | 2c375cf |
config_version_tag, config_sha256, conf_keep_n, conf_stale_timeout, restart_requested, standby_endpoints, wal_record_count, checkpoint_in_progress |
byteorder.h portability header |
✅ Done | 2c375cf |
be64toh() for glibc + musl + macOS |
HEAD-file config storage (config_repl.c Phase 1) |
✅ Done | 491c255 |
conf_save_config, conf_read_current_filename, conf_read_raw, conf_get_saved_tag, conf_check_stale, conf_trim_old_versions |
conf_save_config() path fix |
✅ Done | 2ed7e34 |
Was writing to workdir/conf; fixed to workdir/name/conf to match load_primary_config() |
main.c — null conffile, --conf-keep-n, --conf-stale-timeout |
✅ Done | e74fd2a |
Stack buffer removed; conffile heap-allocated at runtime |
load_primary_config() + myconf_fini() |
✅ Done | cd02cf4 |
FATAL on missing config; SHA-256 cached; reserved-name check |
Real am_i_primary() — election probing |
✅ Done | fe3cc57 |
Non-blocking connect + poll; priority step + xorshift64 jitter; ha_is_primary(), ha_check_priority_stepdown() |
| SSL double-call fix + TOCTOU fix | ✅ Done | ec3cab4 |
SSL_get_error called once; ha->lock held in stepdown |
cluster_init phase split + myapp_run() two-phase flow |
✅ Done | 6b06aa0 |
Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; am_i_primary gates config load |
ha->role set before Phase 2 |
✅ Done | 93b8616 |
Election result stored via atomic_store before cluster_init_phase2 |
Raw capnp shims (request.c) |
✅ Done | 26393e9 |
xce_replconfig_reply_raw(), xce_replconfig_req_ssl(), xce_replconfig_reply_ssl_decode(), xce_upgrade_config_req_extract_raw() — bypass xce_data_t for raw bytes |
handle_config_request() (primary handler) |
✅ Done | 4ef2089 |
Tag compare, SHA-256 double-check; sends real bytes via xce_replconfig_reply_raw() |
fetch_config_from_primary() + do_one_config_fetch() |
✅ Done | 2aae45f |
Full SSL roundtrip: non-blocking connect+poll, xce_replconfig_req_ssl(), SHA-256 verify, conf_save_config(), reload; 20-attempt exponential backoff |
handle_controller_join() |
⚠️ Reversed | 8f4db30 → 40f510fb |
Originally registered the endpoint in standby_endpoints under lock, but this was later identified as a live ownership-model violation (controller-controller peering is primary-initiated outbound only, per ha-async-io.md's Connection Ownership). 40f510fb (post-dates this doc) rewrote it to unconditionally reject inbound role=HostRole_controller joins with -E_XCE_BAD_ROLE; it no longer touches standby_endpoints |
WAL record-count schedule_checkpoint() |
✅ Done | a95476b |
Idempotent CAS; coalesced via delaymgr 1s delay; bounds persist_recover to ≤10k records |
typemap.h union arms + capnp shims |
✅ Done | 491c255 |
replconfigreplymsg, upgradeconfigreplymsg, etc. wired |
API server rejects lost_and_found job submissions |
✅ Done | bc92166 |
preCheckJobRequest() returns "internal recovery queue" error before store lookup; 3 tests in jobs_test.go |
Phase 2 (Remote config push) — COMPILED, ENABLED BY DEFAULT
⚠ Doc drift note (2026-08-08): the paragraph and prerequisites list below describe Phase 2 as gated off pending two prerequisites. Neither prerequisite is met, but
CONF_ENABLE_UPGRADE_CONFIGhas defaulted to 1 (enabled) since commitfd93997f(2026-05-31) — 9 days after this doc's own "updated" date — and has never been reverted.OP_upgradeConfigis a live, reachable path in current production builds. Seesrc/controller/config_repl.h:27-29and dispatch atsrc/controller/cluster.c:772-807. This is a real, outstanding risk, not a future gate — track via.kiro/steering/roadmap.md.
Phase 2 infrastructure ships compiled and enabled. The
CONF_ENABLE_UPGRADE_CONFIG compile-time gate (default: 1 as of
fd93997f, 2026-05-31; originally 0) was intended to keep OP_upgradeConfig
unreachable in production builds until both prerequisites below are met (see
§5.3 and §2 of the design doc), but the gate now defaults to enabled with
neither prerequisite satisfied.
| Component | Status | Notes |
|---|---|---|
handle_upgrade_config() |
✅ Wired, enabled by default | 5053c8d — pre-decode opcode peek in myreq_handler() captures raw packet bytes; xce_upgrade_config_req_extract_raw() extracts sha256+bytes; delegates to do_upgrade_config_with_bytes(). Reachable by default since CONF_ENABLE_UPGRADE_CONFIG defaults to 1 (fd93997f, 2026-05-31) |
push_config_to_standbys() |
✅ Timeout enforced | f3babec — wait_all_with_timeout() with CONF_PUSH_TIMEOUT_MS (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect |
create_lost_and_found() |
✅ Done | 91e66b8 |
Prerequisites — NOT met, despite the gate defaulting to enabled
(CONF_ENABLE_UPGRADE_CONFIG=1 in src/controller/config_repl.h:27-29):
- ❌
validate_host()still an unconditional-accept stub —src/controller/join.c:13-16returns1for every host/key pair, i.e. no real CN allowlist check (seeFUTURE_WORK.md §6, confirmed still open per.kiro/steering/roadmap.md:82-83) - Admin config tool exists and tested end-to-end (see
FUTURE_WORK.md §1)
Test Coverage
| Suite | Tests | Covers |
|---|---|---|
test_config_repl |
101 (was 24 as of this doc's date) | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips — grown substantially since 2026-05-22 by ha-repl-hardening, ha-term-safety, raft-adapter-conformance, config-upgrade-snapshot, and assorted bug-fix coverage (e.g. bug-1475, bug-1476) |
test_am_i_primary |
3 | Priority-0 immediate, priority-0 with peers, deadline-expire |
test_ha_role |
(existing) | Role transitions, watchdog, redirect |
test_persist_wal |
6 (+3 new) | WAL append, CAS idempotency, max_wal_records default |
test_ha_cluster |
5 | Cluster HA integration |
test_p1_integration |
5 | Phase 1 init structure |
| All other suites | ~41 | 0 failures |
| Total | 84 at doc date; stale — test_config_repl alone now exceeds this |
0 failures, 0 ignored (as of 2026-05-22) |
Note (2026-08-08): this table was accurate as of the doc's 2026-05-22 date but has not been kept in sync with ~2.5 months of subsequent test growth.
test_config_replalone now has 101RUN_TEST(...)registrations (verified viasrc/controller/testsuite/test_config_repl.c), so the suite total is understated. Re-run and re-tally before relying on this table for current coverage figures.
Known Limitations / Deferred Work
| Item | Severity | Tracked |
|---|---|---|
start_heartbeat_listener) is a no-op stubstart_heartbeat_listener()/stop_heartbeat_listener() have been deleted from the codebase entirely (never called, per election-iomgr-refactor; see src/controller/ha.h:798, src/controller/ha.c:2916). The priority-step-delay election model this limitation described has been superseded by real Raft-based election/prevote machinery (raft-library, ha-prevote, election-iomgr-refactor specs, all CLOSED) |
— | superseded |
Peers cross-validation hash in OP_heartbeat not implemented |
LOW | FUTURE_WORK.md |
OP_replConfigPush standby-side handler not implementedhandle_repl_config_push() (src/controller/config_repl.c:1308) is fully implemented — validates the request, checks SHA-256 (line 1341), calls conf_save_config() (line 1357), activates the config, and sends an explicit ReplConfigPushReply. Dispatched from OPCode_replConfigPush in src/controller/cluster.c:754-757. Added by commit 3d076ccd ("fix(ha): fix config replication to standbys; add replConfigPush handler"), which post-dates this doc |
— | resolved |
WAL routing helpers (resolve_queue_or_lost_and_found etc.) not wired in apply_wal_entry() — unverifiable as written: resolve_queue_or_lost_and_found no longer appears anywhere in src/controller/*.c/*.h. apply_wal_entry() (src/controller/persist.c:3281) and the WAL_CONFIG_MARKER apply path (persist.c:3851-3866, added by config-upgrade-snapshot task 5) now treat lost-and-found entity re-homing as a deliberately separate, later step from apply/activation — suggesting this routing design was reworked or renamed rather than simply left unfinished. Needs re-investigation, not a straight carry-forward |
LOW | FUTURE_WORK.md |
Files Changed Summary
| File | Change |
|---|---|
src/schema/controller.capnp |
+3 opcodes, +6 structs, configGeneration, HeartbeatMsg |
src/include/myconf.h |
+6 fields to myconf_t; load_primary_config, fetch_config_from_primary, myconf_fini declared |
src/include/xce/cluster_def.h |
+4 fields to cluster_t |
src/controller/ha.h |
+probe_ctx_t; +3 fields to ha_state_t; +2 fields to ha_config_t; xorshift64_next inline |
src/include/persist.h |
+max_wal_records; +wal_record_count, checkpoint_in_progress atomics |
src/include/byteorder.h |
New — portable be64toh() |
src/include/config.h |
+DEFAULT_MAX_WAL_RECORDS 10000 |
src/include/typemap.h |
+7 union arms for new message types |
src/include/request.h |
+4 raw shim declarations; SSL* parameters |
src/lib/request.c |
+xce_replconfig_reply_raw(), xce_replconfig_req_ssl(), xce_replconfig_reply_ssl_decode(), xce_upgrade_config_req_extract_raw() |
src/controller/config_repl.h |
New — Phase 1+2 API declarations; handle_upgrade_config raw-pkt params |
src/controller/config_repl.c |
New — ~900 LOC: storage, SHA-256, full SSL fetch, handlers, push timeout, upgrade wiring |
src/controller/ha.c |
Real am_i_primary(), probe_thread(), ha_is_primary(), ha_check_priority_stepdown(), ha_seed_jitter() |
src/controller/myconf.c |
load_primary_config(), myconf_fini(), static cf global removed |
src/controller/cluster.c |
cluster_init_phase1/2 split; opcode dispatch; pre-decode peek for OP_upgradeConfig; standby_endpoints pruning; restart_requested check |
src/controller/myapp.c |
Two-phase startup; am_i_primary gates config load; myconf_fini on all exits |
src/controller/join.c |
handle_controller_join implemented (was -ENOSYS) |
src/controller/persist.c |
schedule_checkpoint(), WAL record-count trigger, persist_init defaults |
src/controller/main.c |
conffile=NULL; --conf-keep-n; --conf-stale-timeout; peers validation |
src/controller/meson.build |
config_repl.c added; ssl_dep for unit tests |
src/controller/include/cluster.h |
cluster_init_phase1/2 declared |
src/controller/testsuite/test_config_repl.c |
24 unit tests (was 18; +path consistency, +push timeout, +capnp roundtrips) |
src/controller/testsuite/test_ha_role.c |
+3 election probing tests |
src/controller/testsuite/test_persist_wal.c |
+3 WAL bound tests |
src/controller/testsuite/controller_test.c |
New suites registered |
src/frontend/apiserver/controllers/jobs.go |
+reserved-name guard in preCheckJobRequest() |
src/frontend/apiserver/controllers/jobs_test.go |
New — 3 tests: lost_and_found rejected, normal queue passes, empty queue passes |
src/docs/config-replication-design.md |
§3.2 effective_dir definition; §6.3 timeout implementation note |
src/docs/config-replication-impl-status.md |
This file |