# Config Replication — Implementation Status

> **Branch**: `persistent_ha`
> **Date**: 2026-05-22 (updated)
> **Design doc**: `config-replication-design.md`

---

## Phase 1 (Pull-only replication) — WIRE PROTOCOL COMPLETE

Phase 1 is fully implemented. Standbys fetch `ClusterConfig` from the primary on
every startup over a real SSL connection, eliminating manual per-node provisioning.
The wire protocol (capnp encode/decode, SHA-256 verify, disk save) is complete as
of 2026-05-22 (commits `2ed7e34`–`5053c8d`).

| Component | Status | Commit | Notes |
|---|---|---|---|
| Cap'n Proto schema additions | ✅ Done | `683c092` | 3 opcodes, 6 structs, `configGeneration`, `HeartbeatMsg` |
| Struct fields (`myconf_t`, `cluster_t`, `ha_state_t`, `persist_t`) | ✅ Done | `2c375cf` | `config_version_tag`, `config_sha256`, `conf_keep_n`, `conf_stale_timeout`, `restart_requested`, `standby_endpoints`, `wal_record_count`, `checkpoint_in_progress` |
| `byteorder.h` portability header | ✅ Done | `2c375cf` | `be64toh()` for glibc + musl + macOS |
| HEAD-file config storage (`config_repl.c` Phase 1) | ✅ Done | `491c255` | `conf_save_config`, `conf_read_current_filename`, `conf_read_raw`, `conf_get_saved_tag`, `conf_check_stale`, `conf_trim_old_versions` |
| `conf_save_config()` path fix | ✅ Done | `2ed7e34` | Was writing to `workdir/conf`; fixed to `workdir/name/conf` to match `load_primary_config()` |
| `main.c` — null `conffile`, `--conf-keep-n`, `--conf-stale-timeout` | ✅ Done | `e74fd2a` | Stack buffer removed; conffile heap-allocated at runtime |
| `load_primary_config()` + `myconf_fini()` | ✅ Done | `cd02cf4` | FATAL on missing config; SHA-256 cached; reserved-name check |
| Real `am_i_primary()` — election probing | ✅ Done | `fe3cc57` | Non-blocking connect + poll; priority step + xorshift64 jitter; `ha_is_primary()`, `ha_check_priority_stepdown()` |
| SSL double-call fix + TOCTOU fix | ✅ Done | `ec3cab4` | `SSL_get_error` called once; `ha->lock` held in stepdown |
| `cluster_init` phase split + `myapp_run()` two-phase flow | ✅ Done | `6b06aa0` | Phase 1: hooks/HA/cbuf; Phase 2: iomgr/kvmgr/persist; `am_i_primary` gates config load |
| `ha->role` set before Phase 2 | ✅ Done | `93b8616` | Election result stored via `atomic_store` before `cluster_init_phase2` |
| Raw capnp shims (`request.c`) | ✅ Done | `26393e9` | `xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` — bypass `xce_data_t` for raw bytes |
| `handle_config_request()` (primary handler) | ✅ Done | `4ef2089` | Tag compare, SHA-256 double-check; sends real bytes via `xce_replconfig_reply_raw()` |
| `fetch_config_from_primary()` + `do_one_config_fetch()` | ✅ Done | `2aae45f` | Full SSL roundtrip: non-blocking connect+poll, `xce_replconfig_req_ssl()`, SHA-256 verify, `conf_save_config()`, reload; 20-attempt exponential backoff |
| `handle_controller_join()` | ⚠️ Reversed | `8f4db30` → `40f510fb` | Originally registered the endpoint in `standby_endpoints` under lock, but this was later identified as a live ownership-model violation (controller-controller peering is primary-initiated outbound only, per `ha-async-io.md`'s Connection Ownership). `40f510fb` (post-dates this doc) rewrote it to unconditionally reject inbound `role=HostRole_controller` joins with `-E_XCE_BAD_ROLE`; it no longer touches `standby_endpoints` |
| WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records |
| `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired |
| API server rejects `lost_and_found` job submissions | ✅ Done | `bc92166` | `preCheckJobRequest()` returns "internal recovery queue" error before store lookup; 3 tests in `jobs_test.go` |

---

## Phase 2 (Remote config push) — COMPILED, ENABLED BY DEFAULT

> **⚠ Doc drift note (2026-08-08):** the paragraph and prerequisites list below
> describe Phase 2 as gated off pending two prerequisites. Neither prerequisite
> is met, but `CONF_ENABLE_UPGRADE_CONFIG` has defaulted to **1** (enabled)
> since commit `fd93997f` (2026-05-31) — 9 days after this doc's own "updated"
> date — and has never been reverted. `OP_upgradeConfig` is a **live, reachable
> path in current production builds**. See `src/controller/config_repl.h:27-29`
> and dispatch at `src/controller/cluster.c:772-807`. This is a real,
> outstanding risk, not a future gate — track via `.kiro/steering/roadmap.md`.

Phase 2 infrastructure ships compiled and **enabled**. The
`CONF_ENABLE_UPGRADE_CONFIG` compile-time gate (default: **1** as of
`fd93997f`, 2026-05-31; originally 0) was intended to keep `OP_upgradeConfig`
unreachable in production builds until both prerequisites below are met (see
§5.3 and §2 of the design doc), but the gate now defaults to enabled with
neither prerequisite satisfied.

| Component | Status | Notes |
|---|---|---|
| `handle_upgrade_config()` | ✅ Wired, enabled by default | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Reachable by default since `CONF_ENABLE_UPGRADE_CONFIG` defaults to 1 (`fd93997f`, 2026-05-31) |
| `push_config_to_standbys()` | ✅ Timeout enforced | `f3babec` — `wait_all_with_timeout()` with `CONF_PUSH_TIMEOUT_MS` (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect |
| `create_lost_and_found()` | ✅ Done | `91e66b8` | Inserts synthetic queue via `create_queue()` and host via `makeWorker()` before `persist_recover()`; gated with `#ifndef XCE_TESTING` for unit test binary |

**Prerequisites — NOT met, despite the gate defaulting to enabled**
(`CONF_ENABLE_UPGRADE_CONFIG=1` in `src/controller/config_repl.h:27-29`):
1. ❌ `validate_host()` still an unconditional-accept stub — `src/controller/join.c:13-16` returns `1` for every host/key pair, i.e. no real CN allowlist check (see `FUTURE_WORK.md §6`, confirmed still open per `.kiro/steering/roadmap.md:82-83`)
2. Admin config tool exists and tested end-to-end (see `FUTURE_WORK.md §1`)

---

## Test Coverage

| Suite | Tests | Covers |
|---|---|---|
| `test_config_repl` | 101 (was 24 as of this doc's date) | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips — grown substantially since 2026-05-22 by `ha-repl-hardening`, `ha-term-safety`, `raft-adapter-conformance`, `config-upgrade-snapshot`, and assorted bug-fix coverage (e.g. bug-1475, bug-1476) |
| `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire |
| `test_ha_role` | (existing) | Role transitions, watchdog, redirect |
| `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default |
| `test_ha_cluster` | 5 | Cluster HA integration |
| `test_p1_integration` | 5 | Phase 1 init structure |
| All other suites | ~41 | 0 failures |
| **Total** | **84 at doc date; stale — `test_config_repl` alone now exceeds this** | **0 failures, 0 ignored (as of 2026-05-22)** |

> **Note (2026-08-08):** this table was accurate as of the doc's 2026-05-22
> date but has not been kept in sync with ~2.5 months of subsequent test
> growth. `test_config_repl` alone now has 101 `RUN_TEST(...)` registrations
> (verified via `src/controller/testsuite/test_config_repl.c`), so the suite
> total is understated. Re-run and re-tally before relying on this table for
> current coverage figures.

---

## Known Limitations / Deferred Work

| Item | Severity | Tracked |
|---|---|---|
| ~~Heartbeat listener (`start_heartbeat_listener`) is a no-op stub~~ — **RESOLVED/OBSOLETE**: `start_heartbeat_listener()`/`stop_heartbeat_listener()` have been deleted from the codebase entirely (never called, per `election-iomgr-refactor`; see `src/controller/ha.h:798`, `src/controller/ha.c:2916`). The priority-step-delay election model this limitation described has been superseded by real Raft-based election/prevote machinery (`raft-library`, `ha-prevote`, `election-iomgr-refactor` specs, all CLOSED) | — | superseded |
| Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` |
| ~~`OP_replConfigPush` standby-side handler not implemented~~ — **FALSE, now implemented**: `handle_repl_config_push()` (`src/controller/config_repl.c:1308`) is fully implemented — validates the request, checks SHA-256 (line 1341), calls `conf_save_config()` (line 1357), activates the config, and sends an explicit `ReplConfigPushReply`. Dispatched from `OPCode_replConfigPush` in `src/controller/cluster.c:754-757`. Added by commit `3d076ccd` ("fix(ha): fix config replication to standbys; add replConfigPush handler"), which post-dates this doc | — | resolved |
| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` — **unverifiable as written**: `resolve_queue_or_lost_and_found` no longer appears anywhere in `src/controller/*.c`/`*.h`. `apply_wal_entry()` (`src/controller/persist.c:3281`) and the `WAL_CONFIG_MARKER` apply path (`persist.c:3851-3866`, added by `config-upgrade-snapshot` task 5) now treat lost-and-found entity re-homing as a deliberately separate, later step from apply/activation — suggesting this routing design was reworked or renamed rather than simply left unfinished. Needs re-investigation, not a straight carry-forward | LOW | `FUTURE_WORK.md` |

---

## Files Changed Summary

| File | Change |
|---|---|
| `src/schema/controller.capnp` | +3 opcodes, +6 structs, `configGeneration`, `HeartbeatMsg` |
| `src/include/myconf.h` | +6 fields to `myconf_t`; `load_primary_config`, `fetch_config_from_primary`, `myconf_fini` declared |
| `src/include/xce/cluster_def.h` | +4 fields to `cluster_t` |
| `src/controller/ha.h` | +`probe_ctx_t`; +3 fields to `ha_state_t`; +2 fields to `ha_config_t`; `xorshift64_next` inline |
| `src/include/persist.h` | +`max_wal_records`; +`wal_record_count`, `checkpoint_in_progress` atomics |
| `src/include/byteorder.h` | **New** — portable `be64toh()` |
| `src/include/config.h` | +`DEFAULT_MAX_WAL_RECORDS 10000` |
| `src/include/typemap.h` | +7 union arms for new message types |
| `src/include/request.h` | +4 raw shim declarations; `SSL*` parameters |
| `src/lib/request.c` | +`xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` |
| `src/controller/config_repl.h` | **New** — Phase 1+2 API declarations; `handle_upgrade_config` raw-pkt params |
| `src/controller/config_repl.c` | **New** — ~900 LOC: storage, SHA-256, full SSL fetch, handlers, push timeout, upgrade wiring |
| `src/controller/ha.c` | Real `am_i_primary()`, `probe_thread()`, `ha_is_primary()`, `ha_check_priority_stepdown()`, `ha_seed_jitter()` |
| `src/controller/myconf.c` | `load_primary_config()`, `myconf_fini()`, static `cf` global removed |
| `src/controller/cluster.c` | `cluster_init_phase1/2` split; opcode dispatch; pre-decode peek for `OP_upgradeConfig`; `standby_endpoints` pruning; `restart_requested` check |
| `src/controller/myapp.c` | Two-phase startup; `am_i_primary` gates config load; `myconf_fini` on all exits |
| `src/controller/join.c` | `handle_controller_join` implemented (was `-ENOSYS`) |
| `src/controller/persist.c` | `schedule_checkpoint()`, WAL record-count trigger, `persist_init` defaults |
| `src/controller/main.c` | `conffile=NULL`; `--conf-keep-n`; `--conf-stale-timeout`; peers validation |
| `src/controller/meson.build` | `config_repl.c` added; `ssl_dep` for unit tests |
| `src/controller/include/cluster.h` | `cluster_init_phase1/2` declared |
| `src/controller/testsuite/test_config_repl.c` | 24 unit tests (was 18; +path consistency, +push timeout, +capnp roundtrips) |
| `src/controller/testsuite/test_ha_role.c` | +3 election probing tests |
| `src/controller/testsuite/test_persist_wal.c` | +3 WAL bound tests |
| `src/controller/testsuite/controller_test.c` | New suites registered |
| `src/frontend/apiserver/controllers/jobs.go` | +reserved-name guard in `preCheckJobRequest()` |
| `src/frontend/apiserver/controllers/jobs_test.go` | **New** — 3 tests: lost_and_found rejected, normal queue passes, empty queue passes |
| `src/docs/config-replication-design.md` | §3.2 `effective_dir` definition; §6.3 timeout implementation note |
| `src/docs/config-replication-impl-status.md` | This file |