finalize configuration replication

2026-06-09 16:40:13 +08:00 · 2026-06-09 16:40:13 +08:00 · fa66d1ba9c
commit fa66d1ba9c
parent e1517bbf28
30 changed files with 8867 additions and 1560 deletions
--- a/src/docs/cluster-config-split-analysis.md
+++ b/src/docs/cluster-config-split-analysis.md
@ -0,0 +1,272 @@
+# ClusterConfig Schema Split Analysis
+
+> Analysis date: 2026-06-08. Branch: persistent_ha.
+> Source schema: `src/schema/controller.capnp:680`, struct `ClusterConfig`.
+
+## Summary
+
+The `ClusterConfig` struct (20 fields) mixes three distinct configuration concerns:
+cluster-wide business config, HA peer membership, and per-controller performance
+tunables. These should be split into separate objects so each can evolve
+independently without forcing a re-push of the full config blob to all standbys.
+
+## Current State Discovery
+
+### Peer list is already structurally separate
+
+The `--peers` CLI flag (`main.c:178-210`) parses into `myconf.peers_cfg`
+(`xce_cluster_config_t`), which is a **separate struct** from the config blob's
+`cluster_config_t`. At startup, `cluster_init_phase1` uses `peers_cfg.candidates`
+to determine `my_name`, `my_port`, `my_priority`, and the peer list — all before
+the config blob is loaded (`cluster.c:865-880`). The config blob's `candidates`
+field is **not consulted** for HA peer discovery; it is carried for backward
+compatibility in the blob but superseded by `--peers`.
+
+### HA tunables are hardcoded at phase1
+
+`cluster_init_phase1` uses compile-time defaults for all HA timing parameters
+(`cluster.c:854-863`):
+```c
+ha_config_t ha_cfg = {
+    .heartbeat_interval_ms     = HA_DEFAULT_HEARTBEAT_INTERVAL_MS,
+    .heartbeat_timeout_ms      = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS,
+    .election_timeout_ms       = HA_DEFAULT_ELECTION_TIMEOUT_MS,
+    .committing_timeout_ms     = HA_DEFAULT_COMMITTING_TIMEOUT_MS,
+    ...
+};
+```
+
+The `heartbeatInterval` and `inactiveHeartbeatCount` fields in
+`ClusterConfig` are consumed later (e.g., `cluster.c:581-585` writes them into
+a config key-value store for runtime access), but the HA init path does not
+read them. The heartbeat timeout validation in `startup_validation.c:144` works
+against the hardcoded defaults, not the config blob values.
+
+### vcpus is already deprecated
+
+Field `@7 vcpus` is marked deprecated in the schema comment: "use --vcpus CLI
+flag; kept for config blob backward compatibility." `cluster_init_phase2`
+reads `c->myconf->vcpus` (set from CLI), not the blob's `vcpus` field
+(`cluster.c:1045`).
+
+## Proposed Split: Three Configuration Objects
+
+```
+                         ┌──────────────────────────────┐
+                         │    Current ClusterConfig      │
+                         │         (20 fields)           │
+                         └──────────────┬───────────────┘
+                                        │
+                    ┌───────────────────┼───────────────────┐
+                    ▼                   ▼                   ▼
+        ┌───────────────────┐ ┌─────────────────┐ ┌──────────────────────┐
+        │  ClusterConfig     │ │  PeersConfig     │ │  ControllerConfig    │
+        │  (cluster-wide)   │ │  (HA membership) │ │  (per-controller)    │
+        └───────────────────┘ └─────────────────┘ └──────────────────────┘
+```
+
+### Group 1: Cluster-Wide Business Config (keep in `ClusterConfig`)
+
+These fields define the cluster's behavior, resources, and policy. They **must
+be identical** across all controllers. When modified via `OP_upgradeConfig`,
+the new blob is pushed to all standbys and committed on majority.
+
+| Field | Ord | Type | Rationale |
+|-------|-----|------|-----------|
+| `clusterName` | @0 | Text | Cluster identity — must be identical cluster-wide |
+| `resources` | @8 | List(Resource) | Resource catalog — shared state across all nodes |
+| `queues` | @9 | List(QueueConf) | Queue definitions — cluster scheduling policy |
+| `ugroups` | @10 | List(GroupConfig) | User group membership — access control |
+| `hgroups` | @11 | List(GroupConfig) | Host group membership — scheduling constraints |
+| `hostConfigs` | @12 | List(HostConfig) | Per-host overrides — cluster-wide host knowledge |
+| `hproviders` | @18 | List(ScalerConfig) | Autoscaler provider config — cluster infrastructure |
+| `limits` | @19 | List(LimitConfig) | Resource limits — cluster policy enforcement |
+| `configGeneration` | @20 | UInt64 | Monotonic version counter — replication consistency |
+
+**How consumed**: `cluster_init_phase2` (after election and config load)
+iterates these to populate `c->resmgr`, `c->jobmgr`, queue tables, host
+groups, etc.
+
+### Group 2: HA Peer Membership (extract to `PeersConfig`)
+
+These fields define **who** is in the cluster. They change independently when
+controllers are added, removed, or re-IP'd — without any change to the
+business config.
+
+| Field | Ord | Type | Rationale |
+|-------|-----|------|-----------|
+| `candidates` | @2 | List(MasterCandidate) | HA peer list — changes independently of business config |
+| `port` | @1 | UInt32 | Fallback listener port — derived from candidates; per-controller in practice |
+
+**Already structurally separate**: The `--peers` CLI flag populates
+`myconf.peers_cfg` (`xce_cluster_config_t` in `src/include/clusterconfig.h`),
+a struct distinct from the config blob's `cluster_config_t`. The candidates
+list in the blob is carried for backward compatibility but is not read for HA
+peer discovery at startup. Making the separation formal means:
+- `PeersConfig` travels via its own capnp message type and replication path
+- Cluster config upgrades via `OP_upgradeConfig` no longer need to carry the candidate list
+- Peer list changes (add/remove controllers) are a dedicated operation with its own quorum semantics
+
+**Candidate list has special quorum requirements**: Changing the peer set is a
+Raft cluster membership change. It must not use the same
+`repl_wait_majority_ack` path as business config because the majority
+denominator itself is changing. A separate message type enables a two-phase
+membership change protocol (joint consensus) in the future.
+
+**Impact of removing `candidates` from `ClusterConfig` (verified against codebase):**
+
+`cluster_build_config()` (`cluster.c:575–609`) is the **only** place that reads
+`cluster_config->candidates` and `cluster_config->ncandidates` — it copies them
+into the runtime KV store as `ncandidates` and `candidate_N_{name,host,port}`.
+No other component (scheduler, apiserver, opget, sched) reads those KV keys.
+The live peer list used by `ha.c:3266–3272` and `cluster_open_listener` comes
+exclusively from `peers_cfg.candidates[]` (parsed by `--peers`). Removing
+candidates from the blob is therefore a **zero-blast-radius** change.
+
+| File | Location | Change |
+|------|----------|--------|
+| `src/controller/cluster.c` | 587–606 | Remove `ncandidates` + `candidate_N_*` KV loop from `cluster_build_config()` |
+| `src/include/clusterconfig.h` | 34–35 | Remove `int ncandidates` and `master_candidate_t** candidates` from `cluster_config_t` |
+| `src/schema/controller.capnp` | 683 | Remove `candidates @2: List(MasterCandidate)` from `ClusterConfig` |
+
+**`MasterCandidate` fields**:
+```
+struct MasterCandidate {
+  name  @0: Text;    // controller identity (e.g., "ctrl-1")
+  host  @1: Text;    // hostname or IP
+  port  @2: UInt32;  // peer listener port
+}
+```
+
+### Group 3: Per-Controller Performance Tunables (extract to `ControllerConfig`)
+
+These fields configure **how** an individual controller process behaves. They
+can legitimately differ across controllers (e.g., a lower-memory node with
+fewer vcpus, or a WAN-connected standby with a longer heartbeat timeout).
+
+| Field | Ord | Type | Rationale |
+|-------|-----|------|-----------|
+| `heartbeatInterval` | @3 | UInt32 | Primary heartbeat send interval — can be tuned per primary |
+| `inactiveHeartbeatCount` | @4 | UInt32 | Missed-heartbeat tolerance multiplier |
+| `fetchJobInterval` | @5 | UInt32 | Job fetch poll interval — local scheduler interaction |
+| `netioTimeout` | @6 | UInt32 | Network I/O timeout — environment-specific |
+| `vcpus` | @7 | UInt32 | **Already deprecated** — use `--vcpus` CLI flag; remove from schema |
+| `jobmgr` | @13 | HashZArrayConfig | Job manager hash table sizing (nHashSlots, delta) |
+| `workermgr` | @14 | HashZArrayConfig | Worker manager hash table sizing |
+| `qmgr` | @15 | HashZArrayConfig | Queue manager hash table sizing |
+| `resmgr` | @16 | ResmgrConfig | Resource manager hash + instance table sizing |
+| `xstrings` | @17 | HashZArrayConfig | String interning table sizing |
+
+**How consumed**: 
+- `jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings` are read once at
+  `cluster_init_phase2` to construct the internal key-value stores
+  (`cluster.c:803-808`). After construction, the config values are not referenced again.
+- `heartbeatInterval`/`inactiveHeartbeatCount` are read at runtime by the
+  heartbeat and liveness tasks (`cluster.c:1180, 1195`).
+- `fetchJobInterval` drives the scheduler polling loop.
+- `netioTimeout` is used by the iomgr layer for TCP timeouts.
+
+**Why these can differ per controller**: A controller on a high-memory machine
+can be configured with larger hash tables. A controller on a high-latency WAN
+link can have a longer netioTimeout. These differences do not affect cluster
+correctness — they only affect local process behavior.
+
+**`vcpus` removal**: Already deprecated. The `--vcpus` CLI flag
+(`main.c:1045`) is the canonical source. Remove field @7 in the next
+schema revision.
+
+## Fields With Dual Usage (require care)
+
+### `port` (@1)
+
+**Decision: Remove. No fallback bind port.**
+
+`cluster_config->port` currently serves as a fallback in `cluster_open_listener`
+(`cluster.c:953–975`) when `--peers` does not specify a port for the matching
+candidate. This fallback is removed: the `--peers` port is mandatory.
+
+**Affected code (all changes mechanical):**
+
+| File | Location | Change |
+|------|----------|--------|
+| `src/controller/cluster.c` | 953–975 | Remove the `if (my_port == 0) { cluster_config->port }` branch; return `EINVAL` immediately if no port found in `peers_cfg` |
+| `src/controller/myconf.c` | 74 | Remove `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` |
+| `src/controller/config_repl.c` | 55 | Remove `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` |
+| `src/include/clusterconfig.h` | 33 | Remove `int port` from `cluster_config_t` |
+| `src/schema/controller.capnp` | 682 | Remove `port @1: UInt32` from `ClusterConfig` |
+
+Note: `"port"` in `testsuite/test.c:124,266` is the scheduler/worker
+connect-to-primary port (via `get_setting("port")`), not the listener bind
+port. It is unaffected.
+
+## Implementation Approach
+
+### Phase 1: Remove `port` and `candidates` from `ClusterConfig` (immediate)
+
+No backward-compat transition needed for these two fields — the live code
+already ignores both (peer discovery uses `peers_cfg`; port fallback is being
+deleted). All changes are mechanical.
+
+1. `src/schema/controller.capnp` — Remove `port @1` and `candidates @2` from
+   `ClusterConfig`; mark or remove `MasterCandidate` struct if no longer
+   referenced elsewhere
+2. `src/include/clusterconfig.h` — Remove `int port`, `int ncandidates`,
+   `master_candidate_t** candidates` from `cluster_config_t`
+3. `src/controller/cluster.c` — Remove `if (my_port == 0)` fallback block in
+   `cluster_open_listener` (lines 967–975); remove `ncandidates` + `candidate_N_*`
+   KV loop in `cluster_build_config` (lines 587–606)
+4. `src/controller/myconf.c` and `src/controller/config_repl.c` — Remove
+   `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` from `fill_default_cluster_config`
+5. Regenerate capnp bindings
+
+### Phase 2: Schema split (wire-compatible)
+
+1. Define `PeersConfig` and `ControllerConfig` structs in `controller.capnp`
+2. Add them as new fields in the top-level `ConfigData` union alongside
+   the existing `clusterconfig @2`
+3. Regenerate capnp bindings
+4. **Keep the remaining `ClusterConfig` fields intact** during transition —
+   new fields are additive
+
+### Phase 3: Controller reads from new structs
+
+1. `cluster_init_phase2`: read `ControllerConfig` for tunables
+2. `cluster_config_t` retains business config fields only
+3. Fall back to old `ClusterConfig` fields when new structs are absent
+   (backward compatibility for existing `cluster.bin` files)
+
+### Phase 4: Remove superseded tunables from `ClusterConfig`
+
+1. Remove `heartbeatInterval`, `inactiveHeartbeatCount`, `fetchJobInterval`,
+   `netioTimeout`, `vcpus` from `ClusterConfig`
+2. Remove `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings` from
+   `ClusterConfig`
+
+### Phase 5 (future): Peer list replication
+
+1. Define `OPCode_peersUpdate` / `OPCode_peersUpdateAck` for peer list
+   replication with joint-consensus semantics
+2. Cluster membership changes become a dedicated operation with its own
+   quorum protocol
+
+## Effort and Risk
+
+| Phase | Effort | Risk | Justification |
+|-------|--------|------|---------------|
+| 1 — Remove port + candidates | XS (½d) | Low | No transition needed; dead code removal + schema field removal |
+| 2 — Schema split | S (1d) | Low | Additive capnp change; old fields preserved |
+| 3 — Read from new structs | M (2-3d) | Medium | Touches cluster_init paths; backward compat needed |
+| 4 — Remove superseded tunables | S (1d) | Low | Mechanical removal after Phase 3 is validated |
+| 5 — Peer replication | L (1-2wk) | High | New protocol; joint consensus; extensive testing |
+
+## References
+
+- `src/schema/controller.capnp:680-700` — `ClusterConfig` definition
+- `src/schema/controller.capnp:253` — `ConfigData.clusterconfig @2`
+- `src/controller/main.c:178-213` — `--peers` CLI parsing into `peers_cfg`
+- `src/controller/cluster.c:854-880` — `ha_config_t` hardcoded defaults + priority derivation
+- `src/controller/cluster.c:945-977` — port derivation from candidates vs config blob
+- `src/controller/cluster.c:803-808` — internal manager construction
+- `src/controller/ha.h:134` — `ha_config_t.heartbeat_interval_ms` default via `ha_init`
+- `src/controller/startup_validation.c:133-163` — heartbeat config validation
+- `src/include/clusterconfig.h` — `xce_cluster_config_t` (separate peers struct)