# ClusterConfig Schema Split Analysis

> Analysis date: 2026-06-08. Branch: persistent_ha.
> Source schema: `src/schema/controller.capnp:680`, struct `ClusterConfig`.

## Summary

The `ClusterConfig` struct (20 fields) mixes three distinct configuration concerns:
cluster-wide business config, HA peer membership, and per-controller performance
tunables. These should be split into separate objects so each can evolve
independently without forcing a re-push of the full config blob to all standbys.

## Current State Discovery

### Peer list is already structurally separate

The `--peers` CLI flag (`main.c:178-210`) parses into `myconf.peers_cfg`
(`xce_cluster_config_t`), which is a **separate struct** from the config blob's
`cluster_config_t`. At startup, `cluster_init_phase1` uses `peers_cfg.candidates`
to determine `my_name`, `my_port`, `my_priority`, and the peer list — all before
the config blob is loaded (`cluster.c:865-880`). The config blob's `candidates`
field is **not consulted** for HA peer discovery; it is carried for backward
compatibility in the blob but superseded by `--peers`.

### HA tunables are hardcoded at phase1

`cluster_init_phase1` uses compile-time defaults for all HA timing parameters
(`cluster.c:854-863`):
```c
ha_config_t ha_cfg = {
    .heartbeat_interval_ms     = HA_DEFAULT_HEARTBEAT_INTERVAL_MS,
    .heartbeat_timeout_ms      = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS,
    .election_timeout_ms       = HA_DEFAULT_ELECTION_TIMEOUT_MS,
    .committing_timeout_ms     = HA_DEFAULT_COMMITTING_TIMEOUT_MS,
    ...
};
```

The `heartbeatInterval` and `inactiveHeartbeatCount` fields in
`ClusterConfig` are consumed later (e.g., `cluster.c:581-585` writes them into
a config key-value store for runtime access), but the HA init path does not
read them. The heartbeat timeout validation in `startup_validation.c:144` works
against the hardcoded defaults, not the config blob values.

### vcpus is already deprecated

Field `@7 vcpus` is marked deprecated in the schema comment: "use --vcpus CLI
flag; kept for config blob backward compatibility." `cluster_init_phase2`
reads `c->myconf->vcpus` (set from CLI), not the blob's `vcpus` field
(`cluster.c:1045`).

## Proposed Split: Three Configuration Objects

```
                         ┌──────────────────────────────┐
                         │    Current ClusterConfig      │
                         │         (20 fields)           │
                         └──────────────┬───────────────┘
                                        │
                    ┌───────────────────┼───────────────────┐
                    ▼                   ▼                   ▼
        ┌───────────────────┐ ┌─────────────────┐ ┌──────────────────────┐
        │  ClusterConfig     │ │  PeersConfig     │ │  ControllerConfig    │
        │  (cluster-wide)   │ │  (HA membership) │ │  (per-controller)    │
        └───────────────────┘ └─────────────────┘ └──────────────────────┘
```

### Group 1: Cluster-Wide Business Config (keep in `ClusterConfig`)

These fields define the cluster's behavior, resources, and policy. They **must
be identical** across all controllers. When modified via `OP_upgradeConfig`,
the new blob is pushed to all standbys and committed on majority.

| Field | Ord | Type | Rationale |
|-------|-----|------|-----------|
| `clusterName` | @0 | Text | Cluster identity — must be identical cluster-wide |
| `resources` | @8 | List(Resource) | Resource catalog — shared state across all nodes |
| `queues` | @9 | List(QueueConf) | Queue definitions — cluster scheduling policy |
| `ugroups` | @10 | List(GroupConfig) | User group membership — access control |
| `hgroups` | @11 | List(GroupConfig) | Host group membership — scheduling constraints |
| `hostConfigs` | @12 | List(HostConfig) | Per-host overrides — cluster-wide host knowledge |
| `hproviders` | @18 | List(ScalerConfig) | Autoscaler provider config — cluster infrastructure |
| `limits` | @19 | List(LimitConfig) | Resource limits — cluster policy enforcement |
| `configGeneration` | @20 | UInt64 | Monotonic version counter — replication consistency |

**How consumed**: `cluster_init_phase2` (after election and config load)
iterates these to populate `c->resmgr`, `c->jobmgr`, queue tables, host
groups, etc.

### Group 2: HA Peer Membership (extract to `PeersConfig`)

These fields define **who** is in the cluster. They change independently when
controllers are added, removed, or re-IP'd — without any change to the
business config.

| Field | Ord | Type | Rationale |
|-------|-----|------|-----------|
| `candidates` | @2 | List(MasterCandidate) | HA peer list — changes independently of business config |
| `port` | @1 | UInt32 | Fallback listener port — derived from candidates; per-controller in practice |

**Already structurally separate**: The `--peers` CLI flag populates
`myconf.peers_cfg` (`xce_cluster_config_t` in `src/include/clusterconfig.h`),
a struct distinct from the config blob's `cluster_config_t`. The candidates
list in the blob is carried for backward compatibility but is not read for HA
peer discovery at startup. Making the separation formal means:
- `PeersConfig` travels via its own capnp message type and replication path
- Cluster config upgrades via `OP_upgradeConfig` no longer need to carry the candidate list
- Peer list changes (add/remove controllers) are a dedicated operation with its own quorum semantics

**Candidate list has special quorum requirements**: Changing the peer set is a
Raft cluster membership change. It must not use the same
`repl_wait_majority_ack` path as business config because the majority
denominator itself is changing. A separate message type enables a two-phase
membership change protocol (joint consensus) in the future.

**Impact of removing `candidates` from `ClusterConfig` (verified against codebase):**

`cluster_build_config()` (`cluster.c:575–609`) is the **only** place that reads
`cluster_config->candidates` and `cluster_config->ncandidates` — it copies them
into the runtime KV store as `ncandidates` and `candidate_N_{name,host,port}`.
No other component (scheduler, apiserver, opget, sched) reads those KV keys.
The live peer list used by `ha.c:3266–3272` and `cluster_open_listener` comes
exclusively from `peers_cfg.candidates[]` (parsed by `--peers`). Removing
candidates from the blob is therefore a **zero-blast-radius** change.

| File | Location | Change |
|------|----------|--------|
| `src/controller/cluster.c` | 587–606 | Remove `ncandidates` + `candidate_N_*` KV loop from `cluster_build_config()` |
| `src/include/clusterconfig.h` | 34–35 | Remove `int ncandidates` and `master_candidate_t** candidates` from `cluster_config_t` |
| `src/schema/controller.capnp` | 683 | Remove `candidates @2: List(MasterCandidate)` from `ClusterConfig` |

**`MasterCandidate` fields**:
```
struct MasterCandidate {
  name  @0: Text;    // controller identity (e.g., "ctrl-1")
  host  @1: Text;    // hostname or IP
  port  @2: UInt32;  // peer listener port
}
```

### Group 3: Per-Controller Performance Tunables (extract to `ControllerConfig`)

These fields configure **how** an individual controller process behaves. They
can legitimately differ across controllers (e.g., a lower-memory node with
fewer vcpus, or a WAN-connected standby with a longer heartbeat timeout).

| Field | Ord | Type | Rationale |
|-------|-----|------|-----------|
| `heartbeatInterval` | @3 | UInt32 | Primary heartbeat send interval — can be tuned per primary |
| `inactiveHeartbeatCount` | @4 | UInt32 | Missed-heartbeat tolerance multiplier |
| `fetchJobInterval` | @5 | UInt32 | Job fetch poll interval — local scheduler interaction |
| `netioTimeout` | @6 | UInt32 | Network I/O timeout — environment-specific |
| `vcpus` | @7 | UInt32 | **Already deprecated** — use `--vcpus` CLI flag; remove from schema |
| `jobmgr` | @13 | HashZArrayConfig | Job manager hash table sizing (nHashSlots, delta) |
| `workermgr` | @14 | HashZArrayConfig | Worker manager hash table sizing |
| `qmgr` | @15 | HashZArrayConfig | Queue manager hash table sizing |
| `resmgr` | @16 | ResmgrConfig | Resource manager hash + instance table sizing |
| `xstrings` | @17 | HashZArrayConfig | String interning table sizing |

**How consumed**: 
- `jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings` are read once at
  `cluster_init_phase2` to construct the internal key-value stores
  (`cluster.c:803-808`). After construction, the config values are not referenced again.
- `heartbeatInterval`/`inactiveHeartbeatCount` are read at runtime by the
  heartbeat and liveness tasks (`cluster.c:1180, 1195`).
- `fetchJobInterval` drives the scheduler polling loop.
- `netioTimeout` is used by the iomgr layer for TCP timeouts.

**Why these can differ per controller**: A controller on a high-memory machine
can be configured with larger hash tables. A controller on a high-latency WAN
link can have a longer netioTimeout. These differences do not affect cluster
correctness — they only affect local process behavior.

**`vcpus` removal**: Already deprecated. The `--vcpus` CLI flag
(`main.c:1045`) is the canonical source. Remove field @7 in the next
schema revision.

## Fields With Dual Usage (require care)

### `port` (@1)

**Decision: Remove. No fallback bind port.**

`cluster_config->port` currently serves as a fallback in `cluster_open_listener`
(`cluster.c:953–975`) when `--peers` does not specify a port for the matching
candidate. This fallback is removed: the `--peers` port is mandatory.

**Affected code (all changes mechanical):**

| File | Location | Change |
|------|----------|--------|
| `src/controller/cluster.c` | 953–975 | Remove the `if (my_port == 0) { cluster_config->port }` branch; return `EINVAL` immediately if no port found in `peers_cfg` |
| `src/controller/myconf.c` | 74 | Remove `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` |
| `src/controller/config_repl.c` | 55 | Remove `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` |
| `src/include/clusterconfig.h` | 33 | Remove `int port` from `cluster_config_t` |
| `src/schema/controller.capnp` | 682 | Remove `port @1: UInt32` from `ClusterConfig` |

Note: `"port"` in `testsuite/test.c:124,266` is the scheduler/worker
connect-to-primary port (via `get_setting("port")`), not the listener bind
port. It is unaffected.

## Implementation Approach

### Phase 1: Remove `port` and `candidates` from `ClusterConfig` (immediate)

No backward-compat transition needed for these two fields — the live code
already ignores both (peer discovery uses `peers_cfg`; port fallback is being
deleted). All changes are mechanical.

1. `src/schema/controller.capnp` — Remove `port @1` and `candidates @2` from
   `ClusterConfig`; mark or remove `MasterCandidate` struct if no longer
   referenced elsewhere
2. `src/include/clusterconfig.h` — Remove `int port`, `int ncandidates`,
   `master_candidate_t** candidates` from `cluster_config_t`
3. `src/controller/cluster.c` — Remove `if (my_port == 0)` fallback block in
   `cluster_open_listener` (lines 967–975); remove `ncandidates` + `candidate_N_*`
   KV loop in `cluster_build_config` (lines 587–606)
4. `src/controller/myconf.c` and `src/controller/config_repl.c` — Remove
   `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` from `fill_default_cluster_config`
5. Regenerate capnp bindings

### Phase 2: Schema split (wire-compatible)

1. Define `PeersConfig` and `ControllerConfig` structs in `controller.capnp`
2. Add them as new fields in the top-level `ConfigData` union alongside
   the existing `clusterconfig @2`
3. Regenerate capnp bindings
4. **Keep the remaining `ClusterConfig` fields intact** during transition —
   new fields are additive

### Phase 3: Controller reads from new structs

1. `cluster_init_phase2`: read `ControllerConfig` for tunables
2. `cluster_config_t` retains business config fields only
3. Fall back to old `ClusterConfig` fields when new structs are absent
   (backward compatibility for existing `cluster.bin` files)

### Phase 4: Remove superseded tunables from `ClusterConfig`

1. Remove `heartbeatInterval`, `inactiveHeartbeatCount`, `fetchJobInterval`,
   `netioTimeout`, `vcpus` from `ClusterConfig`
2. Remove `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings` from
   `ClusterConfig`

### Phase 5 (future): Peer list replication

1. Define `OPCode_peersUpdate` / `OPCode_peersUpdateAck` for peer list
   replication with joint-consensus semantics
2. Cluster membership changes become a dedicated operation with its own
   quorum protocol

## Effort and Risk

| Phase | Effort | Risk | Justification |
|-------|--------|------|---------------|
| 1 — Remove port + candidates | XS (½d) | Low | No transition needed; dead code removal + schema field removal |
| 2 — Schema split | S (1d) | Low | Additive capnp change; old fields preserved |
| 3 — Read from new structs | M (2-3d) | Medium | Touches cluster_init paths; backward compat needed |
| 4 — Remove superseded tunables | S (1d) | Low | Mechanical removal after Phase 3 is validated |
| 5 — Peer replication | L (1-2wk) | High | New protocol; joint consensus; extensive testing |

## References

- `src/schema/controller.capnp:680-700` — `ClusterConfig` definition
- `src/schema/controller.capnp:253` — `ConfigData.clusterconfig @2`
- `src/controller/main.c:178-213` — `--peers` CLI parsing into `peers_cfg`
- `src/controller/cluster.c:854-880` — `ha_config_t` hardcoded defaults + priority derivation
- `src/controller/cluster.c:945-977` — port derivation from candidates vs config blob
- `src/controller/cluster.c:803-808` — internal manager construction
- `src/controller/ha.h:134` — `ha_config_t.heartbeat_interval_ms` default via `ha_init`
- `src/controller/startup_validation.c:133-163` — heartbeat config validation
- `src/include/clusterconfig.h` — `xce_cluster_config_t` (separate peers struct)