save status
This commit is contained in:
parent
519e11c757
commit
4ad31d774d
16 changed files with 19676 additions and 133 deletions
|
|
@ -55,7 +55,7 @@ configs/
|
|||
```
|
||||
|
||||
- **config build**: Scan directory → validate present YAML files against corresponding JSON schemas → map to `ClusterConfig` capnp struct → serialize to `cluster.bin` → output SHA-256. Missing files = empty section (not error). Schema directory must exist.
|
||||
- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → probe current `configGeneration` → if binary's generation ≤ current, increment to `current + 1` and re-encode (controller rejects non-increasing generations as no-op) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. Admin can force a specific generation with `--generation N`.
|
||||
- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. `configGeneration` is fully controller-owned: the primary assigns `new_gen = current_gen + 1` on receipt and returns it as `versionTag`; xceadmin never reads, sets, or overrides it — no `--generation` flag.
|
||||
- **Shared package**: `xcecommon` Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools
|
||||
|
||||
## Approach
|
||||
|
|
@ -126,11 +126,37 @@ config-upgrade-snapshot (controller: replicate → snapshot → restart → acti
|
|||
entity-bearing-snapshots (upstream blocker, tracked in roadmap)
|
||||
```
|
||||
|
||||
## Configuration Inclusion Rules
|
||||
|
||||
Decided 2026-07-20. Governs `ClusterConfig`/`cluster.bin` field classification now and for any new field added in the future — apply both rules to every candidate field; excluding on either is sufficient.
|
||||
|
||||
> Promoted to project-wide steering: `.kiro/steering/config-design.md` holds the general rule (reusable for worker/scheduler/apiserver shared-config decisions, not just `ClusterConfig`). This section keeps the `ClusterConfig`-specific field classification; see the steering doc for the rule statement and rationale.
|
||||
|
||||
1. **Independence rule**: if a parameter can change without needing to stay consistent with the rest of the cluster's config (i.e., changing it on one node doesn't require every other node to match), keep it OUT of `cluster.bin`. Example: `port` — each controller's own listen port is local, other nodes don't need to know or match it via config push.
|
||||
2. **Frequency rule**: if a parameter is expected to change often relative to the rest of cluster config, keep it OUT of `cluster.bin`. Example: `candidates` — adding/removing a standby is a routine operational action; requiring a config build+push for it would make membership changes as heavyweight as a schema change.
|
||||
3. **Default (both rules fail → include)**: a parameter that must be identical across every controller instance AND changes rarely belongs IN `cluster.bin`. Example: `clusterName` — divergence between nodes is a correctness bug, and it's set once at bootstrap.
|
||||
|
||||
**Correction (2026-07-20, after investigating `clusterconfig-schema-split`)**: the `heartbeatInterval`/`inactiveHeartbeatCount` fields actually govern controller→**worker** liveness (marking a worker unavailable), not the HA/Raft controller-to-controller election heartbeat as originally assumed here — that election heartbeat has no `ClusterConfig` field at all (100% hardcoded, and per a bootstrap-availability constraint it never can have one — see `.kiro/steering/config-design.md` Rule 0). The worker-liveness fields actually *pass* the independence rule (one controller's worker-timeout tolerance doesn't need to match another's) — they stay IN `cluster.bin` by deliberate operational-uniformity override, not because they fail the rules. Verdict unchanged; reasoning corrected.
|
||||
|
||||
**Applying the rules** (reference table for future field additions):
|
||||
|
||||
| Field | Independence rule | Frequency rule | Verdict |
|
||||
|---|---|---|---|
|
||||
| `clusterName` | fails (must match cluster-wide) | fails (rarely changes) | IN |
|
||||
| heartbeat/inactive-count (worker liveness) | **passes** (per-controller worker-timeout tolerance, no cross-node dependency) | — | IN (operational-uniformity override — see correction above) |
|
||||
| `port` | passes (per-node, no cross-node dependency) | — | OUT |
|
||||
| `candidates` | passes (adding a standby doesn't require other nodes' config to change) | passes (routine op) | OUT |
|
||||
| `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash size | passes (perf tuning, local effect only) | — | OUT |
|
||||
| `xstrings` hash size | passes (perf tuning, local effect only) | — | OUT |
|
||||
| `configGeneration` | N/A — not operator input; controller-assigned on every push | N/A | OUT (different reason: not a config value at all, it's protocol state) |
|
||||
|
||||
When a new field is proposed for `ClusterConfig` in the future, run it through rules 1–2 before adding it to xceadmin's YAML→capnp mapping; document the verdict in this table.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**:
|
||||
- `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration` defaults to 0 (offline-safe).
|
||||
- `xceadmin config push`: single controller address (default `localhost:9991`), probe current configGeneration from controller, auto-increment if needed, auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `--generation N` flag for explicit override.
|
||||
- `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration`, `port`, and the per-instance perf-tuning fields (manager hash-table sizes, `xstrings`) are left unset — not xceadmin's to set.
|
||||
- `xceadmin config push`: single controller address (default `localhost:9991`), auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `configGeneration` assignment is entirely controller-side (primary assigns `current + 1`, returns it as `versionTag`) — no client probe, increment, or override flag.
|
||||
- `xceadmin config diff`: (nice-to-have) decode two cluster.bin files, human-readable field-by-field diff. Pre-push safety: "what will this config change actually do?"
|
||||
- `xceadmin config push --dry-run`: (nice-to-have) validate controller will accept the config without committing it
|
||||
- `xcecommon` shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types
|
||||
|
|
@ -143,7 +169,7 @@ entity-bearing-snapshots (upstream blocker, tracked in roadmap)
|
|||
- **Out**:
|
||||
- Controller-side config replication/snapshot/restart logic (owned by `config-upgrade-snapshot`)
|
||||
- Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`, upstream blocker)
|
||||
- Performance tuning parameter extraction (separate direct work item)
|
||||
- Performance tuning / per-instance parameter extraction (separate direct work item) — `port`, manager hash-table-size fields, `xstrings`; these stay local to each controller instance, never in `cluster.bin`
|
||||
- xcecli Python rewrite (future, separate spec)
|
||||
- xce-test-ha config push validation round (deferred until `config-upgrade-snapshot` lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective)
|
||||
|
||||
|
|
@ -453,9 +479,9 @@ Issues identified during 2026-07-07 review — need resolution before or during
|
|||
|
||||
| # | Question | Why it matters | Candidates |
|
||||
|---|----------|---------------|------------|
|
||||
| 1 | **YAML→capnp mapping incomplete** | `ClusterConfig` has 21 fields; 4 YAML files cover ~10. Remaining 11 (clusterName, port, candidates, heartbeat/interval timeouts, jobmgr/workermgr/qmgr/resmgr hash configs, xstrings, configGeneration) have no YAML source | (a) xceadmin fills defaults from controller's `get_default_conf()` values, (b) add a `cluster.yaml` for system-level fields, (c) accept CLI flags `--cluster-name`, `--port`, `--peers` |
|
||||
| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: Controller uses it to detect new configs — same generation as current = no-op (nothing changed). xceadmin MUST read the current generation from the controller, increment it, and embed the incremented value in the binary before pushing. `xceadmin config build` sets it to 0 (offline-safe default). `xceadmin config push` auto-reads current generation, increments, rebuilds with new generation, then pushes. Admin can override with `--generation N`. Controller strictly rejects generations ≤ current (monotonic enforcement). | ✅ Decided: xceadmin-managed increment, controller-enforced monotonic |
|
||||
| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). Currently passed as CLI args to `deploy-ha.sh`, not in YAML. xceadmin needs peer info for config build | (a) `--peers localhost:9991,localhost:9992,localhost:9993` CLI flag, (b) read from `hosts.yaml` host list, (c) separate `peers.yaml` |
|
||||
| 1 | **YAML→capnp mapping incomplete** | `ClusterConfig` has 21 fields; 4 YAML files cover ~10. Of the remaining 11, all now classified per the Configuration Inclusion Rules (see below): **excluded** — `port`, `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash configs, `xstrings` (Req 1.8) and `candidates` (Req 1.10, see row 3) — never sourced by xceadmin, left unset in `cluster.bin`. `configGeneration` **excluded, different reason** — controller-assigned, not operator input (row 2). **Included** — `clusterName`, `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness — NOT the HA/Raft election heartbeat, which has no `ClusterConfig` field at all, see `config-design.md` Rule 0) (Req 1.11) — must stay consistent cluster-wide; still need a YAML source | For the 3 still-included-but-unsourced fields: (a) xceadmin fills defaults from controller's `get_default_conf()` values, (b) add a `cluster.yaml` for system-level fields, (c) accept CLI flags `--cluster-name` |
|
||||
| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: fully controller-owned. Controller auto-increments `new_gen = current_gen + 1` on every `OP_upgradeConfig` receipt (already implemented, `config_repl.c`) and returns it as `versionTag`. xceadmin never reads, sets, or overrides it — `config build` leaves the field unset, `config push` sends the binary as-is, no `--generation` flag. Same generation as current = no-op is a controller-internal detail, not something xceadmin needs to detect. | ✅ Decided: controller-only, no client involvement |
|
||||
| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). **RESOLVED — excluded from configuration** (Req 1.10): standby add/remove changes independently and more frequently than the rest of cluster config, so it fails both Configuration Inclusion Rules below. xceadmin never sources it from YAML/CLI, `cluster.bin` leaves it unset. Open sub-question: how peer membership IS sourced/updated instead (today: CLI args to `deploy-ha.sh`) — not a `cluster.bin` field going forward | (a) keep as `deploy-ha.sh` CLI args (status quo, out of xceadmin scope), (b) separate `xceadmin peer add/remove` command (future), (c) controller-side dynamic membership (future, own spec) |
|
||||
| 4 | **YAML key ≠ capnp field name** | YAML `groupname` → capnp `GroupConfig.name`, YAML `exclude_members` → capnp `exmembers`. Every field needs explicit mapping definition | Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible |
|
||||
| 5 | **Nested struct defaults** | `QueueConf` has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults | Use controller's `get_default_conf()` values from `myconf.c`; document which fields are operator-facing vs system-defaulted |
|
||||
|
||||
|
|
@ -464,7 +490,7 @@ Issues identified during 2026-07-07 review — need resolution before or during
|
|||
| # | Question | Why it matters | Candidates |
|
||||
|---|----------|---------------|------------|
|
||||
| 6 | **Wire framing format** | Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject | Document in design.md; mock controller test catches framing errors |
|
||||
| 7 | **TLS cert discovery** | xceadmin needs CA cert to verify controller TLS. Dev: `ha/data/xce.crt`. Production: configurable path | (a) `--ca-cert` flag, (b) `XCECERT` env var, (c) default `ha/data/xce.crt` for dev, require explicit for non-localhost |
|
||||
| 7 | **TLS cert discovery** | ✅ Decided: reuse server-side cert bundle. xceadmin loads `xce.crt`/`xce.key` from `ha/data/` by default (same files `deploy-ha.sh` copies for controller-to-controller TLS), overridable via `--cert-dir`. No separate CA flag, no client cert issuance — same trust model as `xce_test`. See Requirement 2.3/2.4. |
|
||||
| 8 | **capnp schema compilation** | `capnp compile -o python controller.capnp` → `controller_capnp.py`. When and where? | (a) build time in xce-build.sh (committed output), (b) runtime in xceadmin startup (needs capnp binary), (c) nix derivation that produces the .py file |
|
||||
| 9 | **Python package layout** | Where does code live? Affects pip install, import paths, xce-build.sh | (a) `src/xceadmin/` alongside C code, (b) `tools/xceadmin/` with deploy-ha.sh, (c) top-level `xceadmin/` |
|
||||
| 10 | **existing cluster.bin provenance** | `ha/data/cluster.bin` is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it | Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs |
|
||||
|
|
@ -484,7 +510,7 @@ git tag v1.2.0 → YAML at v1.2.0 → xceadmin config build → cluster.bi
|
|||
controller cluster
|
||||
```
|
||||
|
||||
**Deterministic builds**: same YAML + same generation value + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` defaults to 0 in offline builds (`xceadmin config build`) and is auto-incremented by `xceadmin config push` (read current from controller, +1). Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries.
|
||||
**Deterministic builds**: same YAML + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` is left unset by `xceadmin config build` and never touched by `xceadmin config push` — the primary controller alone assigns it on receipt (`current + 1`), so it plays no part in build determinism. Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries.
|
||||
|
||||
### What git gives us (free)
|
||||
|
||||
|
|
|
|||
|
|
@ -25,7 +25,9 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact
|
|||
- **Out of scope**:
|
||||
- Controller-side config replication, snapshot, restart logic (owned by `config-upgrade-snapshot`)
|
||||
- Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`)
|
||||
- Performance tuning parameter extraction (separate work item)
|
||||
- Performance tuning / per-instance parameter extraction (separate work item) — specifically `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, and `xstrings` (xstring hash size). These are not cluster-shared: each controller instance is expected to set them locally, so `xceadmin` never reads, accepts, or writes them (see Requirement 1.8).
|
||||
- `candidates` (peer/standby list) — changes independently and more frequently than the rest of cluster config (adding/removing a standby shouldn't require a `config build`/`config push` cycle); `xceadmin` never reads, accepts, or writes it (see Requirement 1.10). Peer membership is owned elsewhere (mechanism TBD — see brief.md Open Design Question #3).
|
||||
- `configGeneration` management — fully controller-owned (see Adjacent Expectations below); `xceadmin` never reads, increments, or overrides it.
|
||||
- xcecli Python rewrite (future, separate spec)
|
||||
- Controller config version history, rollback safety validation, config activation status (owned by `config-upgrade-snapshot`, see Future Roadmap in brief.md)
|
||||
|
||||
|
|
@ -33,11 +35,12 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact
|
|||
- **Controller `OP_upgradeConfig` handler** (`config_repl.c`) exists and is operational. On receiving `UpgradeConfigRequest{sha256, bytes}`:
|
||||
1. SHA-256 transit verification
|
||||
2. Cap'n Proto decode + reserved name check (`"lost_and_found"`)
|
||||
3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary
|
||||
3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary — the primary is the sole authority for `configGeneration`; whatever value `xceadmin` ships in `cluster.bin` is discarded and replaced
|
||||
4. Save to disk, replicate to standbys, await majority ACK
|
||||
5. Commit `WAL_CONFIG_MARKER`, signal controller restart
|
||||
6. Return `UpgradeConfigReply{versionTag, ackedCount, error}`
|
||||
- **Controller redirect**: Standby returns `redirectHost`/`redirectPort` in the `RequestReply` union — xceadmin follows the redirect to reach the primary.
|
||||
- **Per-instance field preservation** — RESOLVED structurally, not by controller behavior: `port`, `candidates`, the hash-table-size fields, and `xstrings` are being removed from the `ClusterConfig` capnp struct entirely, owned by the new `clusterconfig-schema-split` spec. Once that lands, there is no zero-default-clobber risk to preserve against — these fields simply won't exist in `cluster.bin`.
|
||||
- **ProbeConfigTag RPC** (`config_repl.c:1774`): Separate direct-capnp RPC (not in `Request` union) for reading current config tag. Used by `config push --dry-run` to validate before committing.
|
||||
- `controller.capnp` schema is the single source of truth for `ClusterConfig`, `Request`/`Reply` unions, and all OP codes.
|
||||
- `config-upgrade-snapshot` spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status.
|
||||
|
|
@ -45,6 +48,9 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact
|
|||
- Entity-bearing snapshot after config upgrade (blocked on `entity-bearing-snapshots` spec)
|
||||
- Config rollback safety (e.g., reject rollback while jobs reference deleted queues)
|
||||
- Config version history query (`xceadmin config history`)
|
||||
- **Dependency on `clusterconfig-schema-split`** — split by subcommand:
|
||||
- `config build`: no dependency. Req 1.8/1.10 already exclude these fields from YAML/CLI regardless of schema state — safe to build in parallel, ship anytime.
|
||||
- `config push`: **soft-blocked in practice**. Until `clusterconfig-schema-split` lands (fields actually removed from the schema) or someone independently verifies `config_repl.c`'s `do_upgrade_config_with_bytes` preserves per-instance fields on a zero-valued push, running `xceadmin config push` against a real cluster risks silently wiping `candidates` (peer list — HA/Raft membership), `port`, and perf-tuning on every node it touches. That's not a hypothetical: nobody has checked the controller does the right thing here today. Recommend `clusterconfig-schema-split` (or equivalent verification) land before `config push` is exercised against anything but a disposable test cluster; `config build` and the rest of the CLI can proceed in parallel.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -61,6 +67,12 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact
|
|||
5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors).
|
||||
6. Where the `schema/` subdirectory is missing from the configs directory, the system shall report an error and exit non-zero.
|
||||
7. Where the YAML content passes JSON Schema validation but maps to fields not present in the `ClusterConfig` capnp struct, the system shall warn the operator about unrecognized fields.
|
||||
8. The system shall not source `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, or `xstrings` (xstring hash size) from YAML or CLI input under any circumstance. These are per-controller-instance performance-tuning parameters, not cluster-shared configuration, and the built `cluster.bin` shall leave them unset (capnp default).
|
||||
9. The system shall not source, read, or write `configGeneration`. The built `cluster.bin` shall leave it unset (capnp default) — the primary controller is the sole authority for assigning it (see Requirement 2 note).
|
||||
10. The system shall not source `candidates` (controller peer/standby list) from YAML or CLI input. Standby membership changes independently of, and more frequently than, the rest of cluster configuration, so it shall not require a `config build`/`config push` cycle; the built `cluster.bin` shall leave `candidates` unset (capnp default). Peer membership sourcing mechanism is out of scope for this spec (open question — see brief.md Open Design Question #3).
|
||||
11. `clusterName` and the `heartbeatInterval`/`inactiveHeartbeatCount` fields (controller→worker liveness — not the HA/Raft election heartbeat, which has no `ClusterConfig` field, see `.kiro/steering/config-design.md` Rule 0) SHALL be sourced from YAML and included in `cluster.bin` — these must stay consistent across all controller instances (unlike Criteria 8–10, which are intentionally excluded). Source location (existing YAML file vs. new `cluster.yaml`) is a design-phase decision.
|
||||
|
||||
Criteria 8–11 apply the **Configuration Inclusion Rules** (brief.md): a field is excluded from `cluster.bin` if it can change independently of the rest of cluster config, OR if it's expected to change frequently; otherwise it's included. Any future `ClusterConfig` field proposed for xceadmin's YAML mapping must be classified against these two rules first.
|
||||
|
||||
### Requirement 2: Config Push
|
||||
|
||||
|
|
@ -70,9 +82,11 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact
|
|||
|
||||
1. When the operator runs `xceadmin config push <cluster.bin>`, the system shall connect to the controller via TLS, send the binary as a capnp `Request` with `OP_upgradeConfig` opcode, `upgradeconfigrequestmsg` body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result.
|
||||
2. When the operator specifies `--controller <host:port>`, the system shall connect to the given controller address instead of the default `localhost:9991`.
|
||||
3. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message.
|
||||
4. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0.
|
||||
5. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero.
|
||||
3. The system shall use the same TLS certificate bundle the controller cluster is deployed with (`xce.crt`/`xce.key`, default path `ha/data/`) to establish trust — no separate client CA flag or cert-issuance step. This matches the trust model `xce_test` and `deploy-ha.sh` already use for controller-to-controller TLS.
|
||||
4. When the operator specifies `--cert-dir <path>`, the system shall load `xce.crt`/`xce.key` from the given directory instead of the default `ha/data/`.
|
||||
5. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message.
|
||||
6. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0.
|
||||
7. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero.
|
||||
|
||||
### Requirement 3: Auto Primary Discovery
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"feature_name": "admin-config-tool",
|
||||
"created_at": "2026-07-06T01:30:00Z",
|
||||
"updated_at": "2026-07-07T08:57:00Z",
|
||||
"updated_at": "2026-07-20T00:19:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-generated",
|
||||
"approvals": {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue