wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity
myxce/src/docs/cluster-config-split-analysis.md

14 KiB
Raw Blame History

ClusterConfig Schema Split Analysis

Analysis date: 2026-06-08. Branch: persistent_ha. Source schema: src/schema/controller.capnp:680, struct ClusterConfig.

Summary

The ClusterConfig struct (20 fields) mixes three distinct configuration concerns: cluster-wide business config, HA peer membership, and per-controller performance tunables. These should be split into separate objects so each can evolve independently without forcing a re-push of the full config blob to all standbys.

Current State Discovery

Peer list is already structurally separate

The --peers CLI flag (main.c:178-210) parses into myconf.peers_cfg (xce_cluster_config_t), which is a separate struct from the config blob's cluster_config_t. At startup, cluster_init_phase1 uses peers_cfg.candidates to determine my_name, my_port, my_priority, and the peer list — all before the config blob is loaded (cluster.c:865-880). The config blob's candidates field is not consulted for HA peer discovery; it is carried for backward compatibility in the blob but superseded by --peers.

HA tunables are hardcoded at phase1

cluster_init_phase1 uses compile-time defaults for all HA timing parameters (cluster.c:854-863):

ha_config_t ha_cfg = {
    .heartbeat_interval_ms     = HA_DEFAULT_HEARTBEAT_INTERVAL_MS,
    .heartbeat_timeout_ms      = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS,
    .election_timeout_ms       = HA_DEFAULT_ELECTION_TIMEOUT_MS,
    .committing_timeout_ms     = HA_DEFAULT_COMMITTING_TIMEOUT_MS,
    ...
};

The heartbeatInterval and inactiveHeartbeatCount fields in ClusterConfig are consumed later (e.g., cluster.c:581-585 writes them into a config key-value store for runtime access), but the HA init path does not read them. The heartbeat timeout validation in startup_validation.c:144 works against the hardcoded defaults, not the config blob values.

vcpus is already deprecated

Field @7 vcpus is marked deprecated in the schema comment: "use --vcpus CLI flag; kept for config blob backward compatibility." cluster_init_phase2 reads c->myconf->vcpus (set from CLI), not the blob's vcpus field (cluster.c:1045).

Proposed Split: Three Configuration Objects

                         ┌──────────────────────────────┐
                         │    Current ClusterConfig      │
                         │         (20 fields)           │
                         └──────────────┬───────────────┘
                                        │
                    ┌───────────────────┼───────────────────┐
                    ▼                   ▼                   ▼
        ┌───────────────────┐ ┌─────────────────┐ ┌──────────────────────┐
        │  ClusterConfig     │ │  PeersConfig     │ │  ControllerConfig    │
        │  (cluster-wide)   │ │  (HA membership) │ │  (per-controller)    │
        └───────────────────┘ └─────────────────┘ └──────────────────────┘

Group 1: Cluster-Wide Business Config (keep in ClusterConfig)

These fields define the cluster's behavior, resources, and policy. They must be identical across all controllers. When modified via OP_upgradeConfig, the new blob is pushed to all standbys and committed on majority.

Field Ord Type Rationale
clusterName @0 Text Cluster identity — must be identical cluster-wide
resources @8 List(Resource) Resource catalog — shared state across all nodes
queues @9 List(QueueConf) Queue definitions — cluster scheduling policy
ugroups @10 List(GroupConfig) User group membership — access control
hgroups @11 List(GroupConfig) Host group membership — scheduling constraints
hostConfigs @12 List(HostConfig) Per-host overrides — cluster-wide host knowledge
hproviders @18 List(ScalerConfig) Autoscaler provider config — cluster infrastructure
limits @19 List(LimitConfig) Resource limits — cluster policy enforcement
configGeneration @20 UInt64 Monotonic version counter — replication consistency

How consumed: cluster_init_phase2 (after election and config load) iterates these to populate c->resmgr, c->jobmgr, queue tables, host groups, etc.

Group 2: HA Peer Membership (extract to PeersConfig)

These fields define who is in the cluster. They change independently when controllers are added, removed, or re-IP'd — without any change to the business config.

Field Ord Type Rationale
candidates @2 List(MasterCandidate) HA peer list — changes independently of business config
port @1 UInt32 Fallback listener port — derived from candidates; per-controller in practice

Already structurally separate: The --peers CLI flag populates myconf.peers_cfg (xce_cluster_config_t in src/include/clusterconfig.h), a struct distinct from the config blob's cluster_config_t. The candidates list in the blob is carried for backward compatibility but is not read for HA peer discovery at startup. Making the separation formal means:

  • PeersConfig travels via its own capnp message type and replication path
  • Cluster config upgrades via OP_upgradeConfig no longer need to carry the candidate list
  • Peer list changes (add/remove controllers) are a dedicated operation with its own quorum semantics

Candidate list has special quorum requirements: Changing the peer set is a Raft cluster membership change. It must not use the same repl_wait_majority_ack path as business config because the majority denominator itself is changing. A separate message type enables a two-phase membership change protocol (joint consensus) in the future.

Impact of removing candidates from ClusterConfig (verified against codebase):

cluster_build_config() (cluster.c:575–609) is the only place that reads cluster_config->candidates and cluster_config->ncandidates — it copies them into the runtime KV store as ncandidates and candidate_N_{name,host,port}. No other component (scheduler, apiserver, opget, sched) reads those KV keys. The live peer list used by ha.c:3266–3272 and cluster_open_listener comes exclusively from peers_cfg.candidates[] (parsed by --peers). Removing candidates from the blob is therefore a zero-blast-radius change.

File Location Change
src/controller/cluster.c 587–606 Remove ncandidates + candidate_N_* KV loop from cluster_build_config()
src/include/clusterconfig.h 34–35 Remove int ncandidates and master_candidate_t** candidates from cluster_config_t
src/schema/controller.capnp 683 Remove candidates @2: List(MasterCandidate) from ClusterConfig

MasterCandidate fields:

struct MasterCandidate {
  name  @0: Text;    // controller identity (e.g., "ctrl-1")
  host  @1: Text;    // hostname or IP
  port  @2: UInt32;  // peer listener port
}

Group 3: Per-Controller Performance Tunables (extract to ControllerConfig)

These fields configure how an individual controller process behaves. They can legitimately differ across controllers (e.g., a lower-memory node with fewer vcpus, or a WAN-connected standby with a longer heartbeat timeout).

Field Ord Type Rationale
heartbeatInterval @3 UInt32 Primary heartbeat send interval — can be tuned per primary
inactiveHeartbeatCount @4 UInt32 Missed-heartbeat tolerance multiplier
fetchJobInterval @5 UInt32 Job fetch poll interval — local scheduler interaction
netioTimeout @6 UInt32 Network I/O timeout — environment-specific
vcpus @7 UInt32 Already deprecated — use --vcpus CLI flag; remove from schema
jobmgr @13 HashZArrayConfig Job manager hash table sizing (nHashSlots, delta)
workermgr @14 HashZArrayConfig Worker manager hash table sizing
qmgr @15 HashZArrayConfig Queue manager hash table sizing
resmgr @16 ResmgrConfig Resource manager hash + instance table sizing
xstrings @17 HashZArrayConfig String interning table sizing

How consumed:

  • jobmgr/workermgr/qmgr/resmgr/xstrings are read once at cluster_init_phase2 to construct the internal key-value stores (cluster.c:803-808). After construction, the config values are not referenced again.
  • heartbeatInterval/inactiveHeartbeatCount are read at runtime by the heartbeat and liveness tasks (cluster.c:1180, 1195).
  • fetchJobInterval drives the scheduler polling loop.
  • netioTimeout is used by the iomgr layer for TCP timeouts.

Why these can differ per controller: A controller on a high-memory machine can be configured with larger hash tables. A controller on a high-latency WAN link can have a longer netioTimeout. These differences do not affect cluster correctness — they only affect local process behavior.

vcpus removal: Already deprecated. The --vcpus CLI flag (main.c:1045) is the canonical source. Remove field @7 in the next schema revision.

Fields With Dual Usage (require care)

port (@1)

Decision: Remove. No fallback bind port.

cluster_config->port currently serves as a fallback in cluster_open_listener (cluster.c:953–975) when --peers does not specify a port for the matching candidate. This fallback is removed: the --peers port is mandatory.

Affected code (all changes mechanical):

File Location Change
src/controller/cluster.c 953–975 Remove the if (my_port == 0) { cluster_config->port } branch; return EINVAL immediately if no port found in peers_cfg
src/controller/myconf.c 74 Remove SET_DEFAULT_IF_UNSET(c->port, 0, 9990)
src/controller/config_repl.c 55 Remove SET_DEFAULT_IF_UNSET(c->port, 0, 9990)
src/include/clusterconfig.h 33 Remove int port from cluster_config_t
src/schema/controller.capnp 682 Remove port @1: UInt32 from ClusterConfig

Note: "port" in testsuite/test.c:124,266 is the scheduler/worker connect-to-primary port (via get_setting("port")), not the listener bind port. It is unaffected.

Implementation Approach

Phase 1: Remove port and candidates from ClusterConfig (immediate)

No backward-compat transition needed for these two fields — the live code already ignores both (peer discovery uses peers_cfg; port fallback is being deleted). All changes are mechanical.

  1. src/schema/controller.capnp — Remove port @1 and candidates @2 from ClusterConfig; mark or remove MasterCandidate struct if no longer referenced elsewhere
  2. src/include/clusterconfig.h — Remove int port, int ncandidates, master_candidate_t** candidates from cluster_config_t
  3. src/controller/cluster.c — Remove if (my_port == 0) fallback block in cluster_open_listener (lines 967–975); remove ncandidates + candidate_N_* KV loop in cluster_build_config (lines 587–606)
  4. src/controller/myconf.c and src/controller/config_repl.c — Remove SET_DEFAULT_IF_UNSET(c->port, 0, 9990) from fill_default_cluster_config
  5. Regenerate capnp bindings

Phase 2: Schema split (wire-compatible)

  1. Define PeersConfig and ControllerConfig structs in controller.capnp
  2. Add them as new fields in the top-level ConfigData union alongside the existing clusterconfig @2
  3. Regenerate capnp bindings
  4. Keep the remaining ClusterConfig fields intact during transition — new fields are additive

Phase 3: Controller reads from new structs

  1. cluster_init_phase2: read ControllerConfig for tunables
  2. cluster_config_t retains business config fields only
  3. Fall back to old ClusterConfig fields when new structs are absent (backward compatibility for existing cluster.bin files)

Phase 4: Remove superseded tunables from ClusterConfig

  1. Remove heartbeatInterval, inactiveHeartbeatCount, fetchJobInterval, netioTimeout, vcpus from ClusterConfig
  2. Remove jobmgr, workermgr, qmgr, resmgr, xstrings from ClusterConfig

Phase 5 (future): Peer list replication

  1. Define OPCode_peersUpdate / OPCode_peersUpdateAck for peer list replication with joint-consensus semantics
  2. Cluster membership changes become a dedicated operation with its own quorum protocol

Effort and Risk

Phase Effort Risk Justification
1 — Remove port + candidates XS (½d) Low No transition needed; dead code removal + schema field removal
2 — Schema split S (1d) Low Additive capnp change; old fields preserved
3 — Read from new structs M (2-3d) Medium Touches cluster_init paths; backward compat needed
4 — Remove superseded tunables S (1d) Low Mechanical removal after Phase 3 is validated
5 — Peer replication L (1-2wk) High New protocol; joint consensus; extensive testing

References

  • src/schema/controller.capnp:680-700 — ClusterConfig definition
  • src/schema/controller.capnp:253 — ConfigData.clusterconfig @2
  • src/controller/main.c:178-213 — --peers CLI parsing into peers_cfg
  • src/controller/cluster.c:854-880 — ha_config_t hardcoded defaults + priority derivation
  • src/controller/cluster.c:945-977 — port derivation from candidates vs config blob
  • src/controller/cluster.c:803-808 — internal manager construction
  • src/controller/ha.h:134 — ha_config_t.heartbeat_interval_ms default via ha_init
  • src/controller/startup_validation.c:133-163 — heartbeat config validation
  • src/include/clusterconfig.h — xce_cluster_config_t (separate peers struct)