8.1 KiB
Configuration Design: What Belongs in Shared Config
Governs any XCE distributed-config schema decision — ClusterConfig (controller.capnp), and any future worker/scheduler/apiserver shared-config struct. Applies whenever a new field is proposed for a config blob that gets pushed/replicated to multiple nodes (e.g. via OP_upgradeConfig or an equivalent future mechanism).
The Question
A parameter needs a home. Two choices:
- Shared config — part of the cluster-wide config blob, distributed to every node, changed via a single coordinated push (build → validate → replicate → majority ACK).
- Local/instance config — set per-node (CLI flag, local config file, env var), changed independently, no cluster-wide coordination.
Getting this wrong in either direction has a cost: putting a per-instance value in shared config forces every routine node-level change through a cluster-wide config push; putting a cluster-invariant value in local config lets nodes silently drift out of sync.
Rule 0: Bootstrap availability (checked first, overrides everything else)
If a parameter is needed to establish consensus, elect a coordinator, or otherwise receive the shared config in the first place, it CANNOT live inside that shared config.
This isn't a judgment call like rules 1–2 below — it's a structural/chicken-egg constraint. A node can't depend on config it doesn't have yet to obtain that same config. Example: an HA/Raft election heartbeat interval/timeout — a controller needs this value to participate in the election that determines who's primary and delivers the cluster config; it must be locally available (CLI flag, hardcoded default) before any config exists. This is independent of whether the field would otherwise pass or fail the independence/frequency rules below — even a field that's cluster-invariant and rarely-changing (which would normally mean "shared") is disqualified from shared config if it's needed to bootstrap access to that config.
If a candidate field passes Rule 0 (i.e., it isn't a bootstrap dependency), proceed to the two rules below.
The Two Rules
Apply both to every candidate field. Excluding on either rule is sufficient — a field only belongs in shared config if it fails both. Note: failing both rules means shared config is the default home, not automatic — see "Operational-uniformity override" below for when a field that would otherwise qualify for local config gets deliberately centralized anyway.
1. Independence rule
If a parameter can change on one node without requiring every other node to match, it does NOT belong in shared config.
Test: does divergence between nodes on this field cause a correctness problem, or is it just... fine? If fine, it's independent — keep it local.
2. Frequency rule
If a parameter is expected to change more often than the rest of the config, it does NOT belong in shared config.
Test: would this field's natural rate of change force config pushes far more often than everything else in the blob needs? If yes, bundling it in shared config makes routine operational changes as heavyweight as a schema change.
Default: both rules fail → include
A field that must be identical across every node AND changes rarely belongs in shared config. That's the actual reason shared config exists — to keep genuinely cluster-invariant values from drifting.
Operational-uniformity override
Passing the independence rule (no correctness requirement to match across nodes) means a field is eligible for local config — it does not mean it must go local. A field can still be deliberately kept in shared config when centralizing it has operational value beyond pure correctness: predictable behavior across the cluster, one less per-node setting for an operator to track, avoiding "why does this node behave differently" support questions. This is a judgment call, not automatic — when exercising it, document the reasoning inline (don't just leave a field in shared config by default without stating why it's an exception).
Worked example: ClusterConfig's heartbeatInterval/inactiveHeartbeatCount fields govern controller→worker liveness (when a controller marks one of its own workers unavailable) — not the HA/Raft controller-to-controller election heartbeat (see Rule 0 example above; those are two different "heartbeat" concepts sharing similar names, easy to conflate). Each controller only manages its own connected workers, so nothing breaks if this value differs node-to-node — it passes the independence rule cleanly. Decided (2026-07-20) to keep it in shared config anyway: operators want uniform, predictable worker-timeout behavior across the cluster rather than per-node drift, even though no correctness invariant demands it.
Worked Examples
| Field | Rule 0 (bootstrap) | Independence | Frequency | Verdict | Why |
|---|---|---|---|---|---|
| Cluster name | passes (not a bootstrap dep) | fails — every node must agree | fails — set once at bootstrap | Shared | Identity mismatch across nodes is a correctness bug |
| HA/Raft election heartbeat interval/timeout | fails Rule 0 — needed to run the election that delivers config in the first place | n/a — disqualified before reaching this rule | n/a | Local (CLI flag; currently hardcoded, see roadmap) | Can't depend on config you don't have yet to obtain that config |
| Controller→worker liveness heartbeat/timeout | passes (not a bootstrap dep) | passes — one controller's worker-timeout tolerance has no cross-node dependency | — | Shared (operational-uniformity override, not a rule outcome — see above) | Would qualify for local by the rules alone; kept shared for predictable cluster-wide behavior |
| Listen port | passes | passes — each node's own bind port has no cross-node dependency | — | Local | Purely a per-process network setting |
| Peer/candidate list (cluster membership) | passes | passes — adding a standby doesn't require the rest of the config to change | passes — routine operational action | Local (own mechanism) | Coupling membership changes to a full config push makes scaling the cluster as heavyweight as a schema migration |
| Internal hash-table sizes, buffer sizes (perf tuning) | passes | passes — effect is local to the process that sets them | — | Local | No node needs to know another node's internal tuning |
| Any monotonic version/generation counter | n/a — not operator input | n/a | n/a | Not config at all | Protocol state the coordinator (primary) assigns, not a value an operator sets |
The last row is a distinct fourth category, not a rule outcome: some fields aren't "config" in the operator-input sense — they're state the system itself maintains (e.g. a config generation counter incremented by the primary on every push). Don't force these into the shared-config-vs-local-config framing; they're neither.
Applying This
When a new field is proposed for a shared config blob:
- Check Rule 0 first — if the field is needed to bootstrap/obtain the config itself, it's local, full stop, regardless of what rules 1–2 would say.
- Otherwise, run it through the independence rule, then the frequency rule.
- If it fails either, its default home is local (CLI flag, per-node config file, a dedicated dynamic-membership mechanism, etc.) — unless the operational-uniformity override applies and is explicitly documented as a deliberate exception.
- If it passes both (fails neither exclusion), add it to shared config — and document why in the owning spec, so the next person doesn't have to re-derive it.
- If it's coordinator-assigned protocol state rather than operator input, keep it out of the operator-facing config schema entirely.
Origin
Derived 2026-07-20 while classifying ClusterConfig's 21 fields for xceadmin config build (.kiro/specs/admin-config-tool/) — 6 fields were being pulled into shared YAML config by default with no principled reason to include vs. exclude them. See brief.md's "Configuration Inclusion Rules" section in that spec for the full field-by-field classification of ClusterConfig specifically; this file holds the general rule for reuse beyond that one spec.