diff --git a/.kiro/specs/admin-config-tool/brief.md b/.kiro/specs/admin-config-tool/brief.md index f58d18e1..55842ff7 100644 --- a/.kiro/specs/admin-config-tool/brief.md +++ b/.kiro/specs/admin-config-tool/brief.md @@ -55,7 +55,7 @@ configs/ ``` - **config build**: Scan directory → validate present YAML files against corresponding JSON schemas → map to `ClusterConfig` capnp struct → serialize to `cluster.bin` → output SHA-256. Missing files = empty section (not error). Schema directory must exist. -- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → probe current `configGeneration` → if binary's generation ≤ current, increment to `current + 1` and re-encode (controller rejects non-increasing generations as no-op) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. Admin can force a specific generation with `--generation N`. +- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. `configGeneration` is fully controller-owned: the primary assigns `new_gen = current_gen + 1` on receipt and returns it as `versionTag`; xceadmin never reads, sets, or overrides it — no `--generation` flag. - **Shared package**: `xcecommon` Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools ## Approach @@ -126,11 +126,37 @@ config-upgrade-snapshot (controller: replicate → snapshot → restart → acti entity-bearing-snapshots (upstream blocker, tracked in roadmap) ``` +## Configuration Inclusion Rules + +Decided 2026-07-20. Governs `ClusterConfig`/`cluster.bin` field classification now and for any new field added in the future — apply both rules to every candidate field; excluding on either is sufficient. + +> Promoted to project-wide steering: `.kiro/steering/config-design.md` holds the general rule (reusable for worker/scheduler/apiserver shared-config decisions, not just `ClusterConfig`). This section keeps the `ClusterConfig`-specific field classification; see the steering doc for the rule statement and rationale. + +1. **Independence rule**: if a parameter can change without needing to stay consistent with the rest of the cluster's config (i.e., changing it on one node doesn't require every other node to match), keep it OUT of `cluster.bin`. Example: `port` — each controller's own listen port is local, other nodes don't need to know or match it via config push. +2. **Frequency rule**: if a parameter is expected to change often relative to the rest of cluster config, keep it OUT of `cluster.bin`. Example: `candidates` — adding/removing a standby is a routine operational action; requiring a config build+push for it would make membership changes as heavyweight as a schema change. +3. **Default (both rules fail → include)**: a parameter that must be identical across every controller instance AND changes rarely belongs IN `cluster.bin`. Example: `clusterName` — divergence between nodes is a correctness bug, and it's set once at bootstrap. + +**Correction (2026-07-20, after investigating `clusterconfig-schema-split`)**: the `heartbeatInterval`/`inactiveHeartbeatCount` fields actually govern controller→**worker** liveness (marking a worker unavailable), not the HA/Raft controller-to-controller election heartbeat as originally assumed here — that election heartbeat has no `ClusterConfig` field at all (100% hardcoded, and per a bootstrap-availability constraint it never can have one — see `.kiro/steering/config-design.md` Rule 0). The worker-liveness fields actually *pass* the independence rule (one controller's worker-timeout tolerance doesn't need to match another's) — they stay IN `cluster.bin` by deliberate operational-uniformity override, not because they fail the rules. Verdict unchanged; reasoning corrected. + +**Applying the rules** (reference table for future field additions): + +| Field | Independence rule | Frequency rule | Verdict | +|---|---|---|---| +| `clusterName` | fails (must match cluster-wide) | fails (rarely changes) | IN | +| heartbeat/inactive-count (worker liveness) | **passes** (per-controller worker-timeout tolerance, no cross-node dependency) | — | IN (operational-uniformity override — see correction above) | +| `port` | passes (per-node, no cross-node dependency) | — | OUT | +| `candidates` | passes (adding a standby doesn't require other nodes' config to change) | passes (routine op) | OUT | +| `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash size | passes (perf tuning, local effect only) | — | OUT | +| `xstrings` hash size | passes (perf tuning, local effect only) | — | OUT | +| `configGeneration` | N/A — not operator input; controller-assigned on every push | N/A | OUT (different reason: not a config value at all, it's protocol state) | + +When a new field is proposed for `ClusterConfig` in the future, run it through rules 1–2 before adding it to xceadmin's YAML→capnp mapping; document the verdict in this table. + ## Scope - **In**: - - `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration` defaults to 0 (offline-safe). - - `xceadmin config push`: single controller address (default `localhost:9991`), probe current configGeneration from controller, auto-increment if needed, auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `--generation N` flag for explicit override. + - `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration`, `port`, and the per-instance perf-tuning fields (manager hash-table sizes, `xstrings`) are left unset — not xceadmin's to set. + - `xceadmin config push`: single controller address (default `localhost:9991`), auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `configGeneration` assignment is entirely controller-side (primary assigns `current + 1`, returns it as `versionTag`) — no client probe, increment, or override flag. - `xceadmin config diff`: (nice-to-have) decode two cluster.bin files, human-readable field-by-field diff. Pre-push safety: "what will this config change actually do?" - `xceadmin config push --dry-run`: (nice-to-have) validate controller will accept the config without committing it - `xcecommon` shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types @@ -143,7 +169,7 @@ entity-bearing-snapshots (upstream blocker, tracked in roadmap) - **Out**: - Controller-side config replication/snapshot/restart logic (owned by `config-upgrade-snapshot`) - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`, upstream blocker) - - Performance tuning parameter extraction (separate direct work item) + - Performance tuning / per-instance parameter extraction (separate direct work item) — `port`, manager hash-table-size fields, `xstrings`; these stay local to each controller instance, never in `cluster.bin` - xcecli Python rewrite (future, separate spec) - xce-test-ha config push validation round (deferred until `config-upgrade-snapshot` lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective) @@ -453,9 +479,9 @@ Issues identified during 2026-07-07 review — need resolution before or during | # | Question | Why it matters | Candidates | |---|----------|---------------|------------| -| 1 | **YAML→capnp mapping incomplete** | `ClusterConfig` has 21 fields; 4 YAML files cover ~10. Remaining 11 (clusterName, port, candidates, heartbeat/interval timeouts, jobmgr/workermgr/qmgr/resmgr hash configs, xstrings, configGeneration) have no YAML source | (a) xceadmin fills defaults from controller's `get_default_conf()` values, (b) add a `cluster.yaml` for system-level fields, (c) accept CLI flags `--cluster-name`, `--port`, `--peers` | -| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: Controller uses it to detect new configs — same generation as current = no-op (nothing changed). xceadmin MUST read the current generation from the controller, increment it, and embed the incremented value in the binary before pushing. `xceadmin config build` sets it to 0 (offline-safe default). `xceadmin config push` auto-reads current generation, increments, rebuilds with new generation, then pushes. Admin can override with `--generation N`. Controller strictly rejects generations ≤ current (monotonic enforcement). | ✅ Decided: xceadmin-managed increment, controller-enforced monotonic | -| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). Currently passed as CLI args to `deploy-ha.sh`, not in YAML. xceadmin needs peer info for config build | (a) `--peers localhost:9991,localhost:9992,localhost:9993` CLI flag, (b) read from `hosts.yaml` host list, (c) separate `peers.yaml` | +| 1 | **YAML→capnp mapping incomplete** | `ClusterConfig` has 21 fields; 4 YAML files cover ~10. Of the remaining 11, all now classified per the Configuration Inclusion Rules (see below): **excluded** — `port`, `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash configs, `xstrings` (Req 1.8) and `candidates` (Req 1.10, see row 3) — never sourced by xceadmin, left unset in `cluster.bin`. `configGeneration` **excluded, different reason** — controller-assigned, not operator input (row 2). **Included** — `clusterName`, `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness — NOT the HA/Raft election heartbeat, which has no `ClusterConfig` field at all, see `config-design.md` Rule 0) (Req 1.11) — must stay consistent cluster-wide; still need a YAML source | For the 3 still-included-but-unsourced fields: (a) xceadmin fills defaults from controller's `get_default_conf()` values, (b) add a `cluster.yaml` for system-level fields, (c) accept CLI flags `--cluster-name` | +| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: fully controller-owned. Controller auto-increments `new_gen = current_gen + 1` on every `OP_upgradeConfig` receipt (already implemented, `config_repl.c`) and returns it as `versionTag`. xceadmin never reads, sets, or overrides it — `config build` leaves the field unset, `config push` sends the binary as-is, no `--generation` flag. Same generation as current = no-op is a controller-internal detail, not something xceadmin needs to detect. | ✅ Decided: controller-only, no client involvement | +| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). **RESOLVED — excluded from configuration** (Req 1.10): standby add/remove changes independently and more frequently than the rest of cluster config, so it fails both Configuration Inclusion Rules below. xceadmin never sources it from YAML/CLI, `cluster.bin` leaves it unset. Open sub-question: how peer membership IS sourced/updated instead (today: CLI args to `deploy-ha.sh`) — not a `cluster.bin` field going forward | (a) keep as `deploy-ha.sh` CLI args (status quo, out of xceadmin scope), (b) separate `xceadmin peer add/remove` command (future), (c) controller-side dynamic membership (future, own spec) | | 4 | **YAML key ≠ capnp field name** | YAML `groupname` → capnp `GroupConfig.name`, YAML `exclude_members` → capnp `exmembers`. Every field needs explicit mapping definition | Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible | | 5 | **Nested struct defaults** | `QueueConf` has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults | Use controller's `get_default_conf()` values from `myconf.c`; document which fields are operator-facing vs system-defaulted | @@ -464,7 +490,7 @@ Issues identified during 2026-07-07 review — need resolution before or during | # | Question | Why it matters | Candidates | |---|----------|---------------|------------| | 6 | **Wire framing format** | Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject | Document in design.md; mock controller test catches framing errors | -| 7 | **TLS cert discovery** | xceadmin needs CA cert to verify controller TLS. Dev: `ha/data/xce.crt`. Production: configurable path | (a) `--ca-cert` flag, (b) `XCECERT` env var, (c) default `ha/data/xce.crt` for dev, require explicit for non-localhost | +| 7 | **TLS cert discovery** | ✅ Decided: reuse server-side cert bundle. xceadmin loads `xce.crt`/`xce.key` from `ha/data/` by default (same files `deploy-ha.sh` copies for controller-to-controller TLS), overridable via `--cert-dir`. No separate CA flag, no client cert issuance — same trust model as `xce_test`. See Requirement 2.3/2.4. | | 8 | **capnp schema compilation** | `capnp compile -o python controller.capnp` → `controller_capnp.py`. When and where? | (a) build time in xce-build.sh (committed output), (b) runtime in xceadmin startup (needs capnp binary), (c) nix derivation that produces the .py file | | 9 | **Python package layout** | Where does code live? Affects pip install, import paths, xce-build.sh | (a) `src/xceadmin/` alongside C code, (b) `tools/xceadmin/` with deploy-ha.sh, (c) top-level `xceadmin/` | | 10 | **existing cluster.bin provenance** | `ha/data/cluster.bin` is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it | Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs | @@ -484,7 +510,7 @@ git tag v1.2.0 → YAML at v1.2.0 → xceadmin config build → cluster.bi controller cluster ``` -**Deterministic builds**: same YAML + same generation value + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` defaults to 0 in offline builds (`xceadmin config build`) and is auto-incremented by `xceadmin config push` (read current from controller, +1). Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries. +**Deterministic builds**: same YAML + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` is left unset by `xceadmin config build` and never touched by `xceadmin config push` — the primary controller alone assigns it on receipt (`current + 1`), so it plays no part in build determinism. Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries. ### What git gives us (free) diff --git a/.kiro/specs/admin-config-tool/requirements.md b/.kiro/specs/admin-config-tool/requirements.md index 9af75a5f..0914dddc 100644 --- a/.kiro/specs/admin-config-tool/requirements.md +++ b/.kiro/specs/admin-config-tool/requirements.md @@ -25,7 +25,9 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact - **Out of scope**: - Controller-side config replication, snapshot, restart logic (owned by `config-upgrade-snapshot`) - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`) - - Performance tuning parameter extraction (separate work item) + - Performance tuning / per-instance parameter extraction (separate work item) — specifically `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, and `xstrings` (xstring hash size). These are not cluster-shared: each controller instance is expected to set them locally, so `xceadmin` never reads, accepts, or writes them (see Requirement 1.8). + - `candidates` (peer/standby list) — changes independently and more frequently than the rest of cluster config (adding/removing a standby shouldn't require a `config build`/`config push` cycle); `xceadmin` never reads, accepts, or writes it (see Requirement 1.10). Peer membership is owned elsewhere (mechanism TBD — see brief.md Open Design Question #3). + - `configGeneration` management — fully controller-owned (see Adjacent Expectations below); `xceadmin` never reads, increments, or overrides it. - xcecli Python rewrite (future, separate spec) - Controller config version history, rollback safety validation, config activation status (owned by `config-upgrade-snapshot`, see Future Roadmap in brief.md) @@ -33,11 +35,12 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact - **Controller `OP_upgradeConfig` handler** (`config_repl.c`) exists and is operational. On receiving `UpgradeConfigRequest{sha256, bytes}`: 1. SHA-256 transit verification 2. Cap'n Proto decode + reserved name check (`"lost_and_found"`) - 3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary + 3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary — the primary is the sole authority for `configGeneration`; whatever value `xceadmin` ships in `cluster.bin` is discarded and replaced 4. Save to disk, replicate to standbys, await majority ACK 5. Commit `WAL_CONFIG_MARKER`, signal controller restart 6. Return `UpgradeConfigReply{versionTag, ackedCount, error}` - **Controller redirect**: Standby returns `redirectHost`/`redirectPort` in the `RequestReply` union — xceadmin follows the redirect to reach the primary. + - **Per-instance field preservation** — RESOLVED structurally, not by controller behavior: `port`, `candidates`, the hash-table-size fields, and `xstrings` are being removed from the `ClusterConfig` capnp struct entirely, owned by the new `clusterconfig-schema-split` spec. Once that lands, there is no zero-default-clobber risk to preserve against — these fields simply won't exist in `cluster.bin`. - **ProbeConfigTag RPC** (`config_repl.c:1774`): Separate direct-capnp RPC (not in `Request` union) for reading current config tag. Used by `config push --dry-run` to validate before committing. - `controller.capnp` schema is the single source of truth for `ClusterConfig`, `Request`/`Reply` unions, and all OP codes. - `config-upgrade-snapshot` spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status. @@ -45,6 +48,9 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact - Entity-bearing snapshot after config upgrade (blocked on `entity-bearing-snapshots` spec) - Config rollback safety (e.g., reject rollback while jobs reference deleted queues) - Config version history query (`xceadmin config history`) + - **Dependency on `clusterconfig-schema-split`** — split by subcommand: + - `config build`: no dependency. Req 1.8/1.10 already exclude these fields from YAML/CLI regardless of schema state — safe to build in parallel, ship anytime. + - `config push`: **soft-blocked in practice**. Until `clusterconfig-schema-split` lands (fields actually removed from the schema) or someone independently verifies `config_repl.c`'s `do_upgrade_config_with_bytes` preserves per-instance fields on a zero-valued push, running `xceadmin config push` against a real cluster risks silently wiping `candidates` (peer list — HA/Raft membership), `port`, and perf-tuning on every node it touches. That's not a hypothetical: nobody has checked the controller does the right thing here today. Recommend `clusterconfig-schema-split` (or equivalent verification) land before `config push` is exercised against anything but a disposable test cluster; `config build` and the rest of the CLI can proceed in parallel. ## Requirements @@ -61,6 +67,12 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact 5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors). 6. Where the `schema/` subdirectory is missing from the configs directory, the system shall report an error and exit non-zero. 7. Where the YAML content passes JSON Schema validation but maps to fields not present in the `ClusterConfig` capnp struct, the system shall warn the operator about unrecognized fields. +8. The system shall not source `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, or `xstrings` (xstring hash size) from YAML or CLI input under any circumstance. These are per-controller-instance performance-tuning parameters, not cluster-shared configuration, and the built `cluster.bin` shall leave them unset (capnp default). +9. The system shall not source, read, or write `configGeneration`. The built `cluster.bin` shall leave it unset (capnp default) — the primary controller is the sole authority for assigning it (see Requirement 2 note). +10. The system shall not source `candidates` (controller peer/standby list) from YAML or CLI input. Standby membership changes independently of, and more frequently than, the rest of cluster configuration, so it shall not require a `config build`/`config push` cycle; the built `cluster.bin` shall leave `candidates` unset (capnp default). Peer membership sourcing mechanism is out of scope for this spec (open question — see brief.md Open Design Question #3). +11. `clusterName` and the `heartbeatInterval`/`inactiveHeartbeatCount` fields (controller→worker liveness — not the HA/Raft election heartbeat, which has no `ClusterConfig` field, see `.kiro/steering/config-design.md` Rule 0) SHALL be sourced from YAML and included in `cluster.bin` — these must stay consistent across all controller instances (unlike Criteria 8–10, which are intentionally excluded). Source location (existing YAML file vs. new `cluster.yaml`) is a design-phase decision. + +Criteria 8–11 apply the **Configuration Inclusion Rules** (brief.md): a field is excluded from `cluster.bin` if it can change independently of the rest of cluster config, OR if it's expected to change frequently; otherwise it's included. Any future `ClusterConfig` field proposed for xceadmin's YAML mapping must be classified against these two rules first. ### Requirement 2: Config Push @@ -70,9 +82,11 @@ Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact 1. When the operator runs `xceadmin config push `, the system shall connect to the controller via TLS, send the binary as a capnp `Request` with `OP_upgradeConfig` opcode, `upgradeconfigrequestmsg` body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result. 2. When the operator specifies `--controller `, the system shall connect to the given controller address instead of the default `localhost:9991`. -3. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message. -4. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0. -5. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero. +3. The system shall use the same TLS certificate bundle the controller cluster is deployed with (`xce.crt`/`xce.key`, default path `ha/data/`) to establish trust — no separate client CA flag or cert-issuance step. This matches the trust model `xce_test` and `deploy-ha.sh` already use for controller-to-controller TLS. +4. When the operator specifies `--cert-dir `, the system shall load `xce.crt`/`xce.key` from the given directory instead of the default `ha/data/`. +5. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message. +6. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0. +7. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero. ### Requirement 3: Auto Primary Discovery diff --git a/.kiro/specs/admin-config-tool/spec.json b/.kiro/specs/admin-config-tool/spec.json index 02d95d04..fb15669f 100644 --- a/.kiro/specs/admin-config-tool/spec.json +++ b/.kiro/specs/admin-config-tool/spec.json @@ -1,7 +1,7 @@ { "feature_name": "admin-config-tool", "created_at": "2026-07-06T01:30:00Z", - "updated_at": "2026-07-07T08:57:00Z", + "updated_at": "2026-07-20T00:19:00Z", "language": "en", "phase": "requirements-generated", "approvals": { diff --git a/.kiro/specs/component-failover-adoption/design.md b/.kiro/specs/component-failover-adoption/design.md new file mode 100644 index 00000000..3a64946f --- /dev/null +++ b/.kiro/specs/component-failover-adoption/design.md @@ -0,0 +1,505 @@ +# Design Document: component-failover-adoption + +## Overview + +Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile `libxce` already ships a complete, tested, transparent multi-master mechanism (`XCECLI_FAILOVER_CALL` and friends in `src/lib/client.c`) — the same one `xce_test`'s `masters` setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect. + +**Users**: HA cluster operators who configure multiple controller candidates (`--peers` for scheduler/worker, `backend.peers` YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation `xce_test` already meets. + +**Impact**: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in `controller_connector.c` and `scheduler.c`); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to `libxce`'s failover mechanism itself. + +### Goals +- Scheduler, worker, and apiserver each hand the full configured candidate list to `xce_client_config_t` on the very first connection attempt (Req 1). +- All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2). +- Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3). +- apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4). +- Every connect/reconnect attempt and outcome is logged per-candidate (Req 5). +- apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6). + +### Non-Goals +- Changing `xce_test` (it is already the reference-correct client). +- Completing or adopting the separate, unused `xce_connect_cluster()`/`xce_reconnect_*` primitive in `cluster.c` — superseded by the mechanism this design adopts; left untouched. +- Raising the `N_MAX_MASTERS` (3) cap or reconciling it with `MAX_HA_CANDIDATES` (4) — pre-existing constraint, replicated (clamp + WARN) exactly as `xce_test` already does. +- Wiring CA-cert/TLS trust into the real connect path — `xce_client_config_t` carries no such field today and this feature does not add one; TLS is established the same way it already is. +- Controller-side changes (redirect-reply population, peer authentication) — already shipped by `libxce-failover`. +- Recovering worker job-state lost during a disconnect window — owned by `worker-rejoin-resync`. + +## Boundary Commitments + +### This Spec Owns +- A new shared libxce helper that converts a validated `xce_cluster_config_t` (parsed candidate list) into a fully-populated `xce_client_config_t`. +- Worker's, scheduler's, and apiserver's integration code that calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` with the full candidate list and registers a rejoin callback. +- apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation). +- apiserver's client-facing fail-fast behavior while its backend connection is degraded. +- Connect/reconnect diagnostic logging in all three components. + +### Out of Boundary +- `xce_test` and its `masters` setting (unchanged, reference implementation). +- `libxce`'s failover mechanism itself (`client.c`, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified. +- `xce_connect_cluster()`/`xce_reconnect_*` (unused, untouched). +- Controller-side membership/redirect logic. +- Worker job-state resync content on rejoin (`worker-rejoin-resync`). +- `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` value changes. +- TLS/CA-cert plumbing into the real connect path. + +### Allowed Dependencies +- `src/lib/client.c` / `client_internal.h` / `client_failover.h` (shipped `libxce-failover` mechanism) — read-only dependency, no modification. +- `src/lib/cluster.c` / `xce/cluster.h` (`xce_parse_peers`, `xce_cluster_config_validate`) — existing, unmodified; this spec adds one new function alongside them. +- Existing per-component candidate-list CLI/YAML surfaces (`--peers`, `--ca-cert`, `backend.peers`) — unmodified externally; only how they're consumed changes. + +### Revalidation Triggers +- Any change to `xce_client_config_t`'s layout or `N_MAX_MASTERS`. +- Any change to `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL` retry/classification semantics (`libxce-failover`). +- Any change to `xce_parse_peers`/`xce_cluster_config_validate` syntax or validation rules (apiserver's CGO-based parser would need re-verification). +- `worker-rejoin-resync` landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that. + +## Architecture + +### Existing Architecture Analysis +Today each of the three components independently drives its own `xcecli_ctx_t` lifecycle with `nmasters` hardcoded to `1` per connection attempt, and layers its own outer candidate loop on top: +- **worker**: `controller_connector.c`'s `xcecd_join_cluster` loops candidates itself, but the very first connection (inside `xcecd_start()`) happens before `cc_ctx->cluster_cfg` is even populated (`main.c` assigns it after `xcecd_start()` returns) — the candidate list is silently ignored on startup. +- **scheduler**: `scheduler.c`'s `sched_init` loops candidates correctly at startup, but there is no reconnect logic anywhere afterward. +- **apiserver**: `clib.go`'s `RegisterToBackend` loops candidates once via a hand-written Go parser, only at process startup. + +None of the three ever set `nmasters > 1` on a single `xce_client_config_t`, so the library's own rotation/retry (`client.c`, wrapped by `XCECLI_FAILOVER_CALL`) never actually engages for them. + +### Architecture Pattern & Boundary Map + +```mermaid +graph TB + subgraph Daemons + Worker[worker xcecd] + Scheduler[xce_scheduler] + Apiserver[apiserver backend package] + end + + subgraph NewSharedCode + Convert[xce_cluster_config_to_client_config] + end + + subgraph ExistingLibxce + Parse[xce_parse_peers] + Validate[xce_cluster_config_validate] + CtxInit[xcecli_ctx_init xcecli_start xcecli_join] + Failover[XCECLI_FAILOVER_CALL rotate retry backoff] + Rejoin[rejoin callback xcecli_set_reconnect_handler] + end + + subgraph Cluster + Primary[Controller Primary] + Standby1[Controller Standby] + end + + Worker --> Parse + Scheduler --> Parse + Apiserver --> Parse + Parse --> Validate + Validate --> Convert + Convert --> CtxInit + CtxInit --> Failover + Failover --> Rejoin + Failover --> Primary + Failover --> Standby1 +``` + +**Architecture Integration**: +- Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem). +- Domain boundaries: `libxce` owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining). +- Existing patterns preserved: the `xce_parse_peers`/`xce_cluster_config_validate`/`xce_client_config_t` split; the CGO bridge pattern already used throughout `backend/`; worker's dedicated connector thread. +- New components: one C helper (candidate-list → client-config conversion, generalized out of `xce_test`'s inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need). +- Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered. + +### Rejoin callback constraint: avoiding self-deadlock on `ac->failover.lock` + +Verified directly against `client_failover.c:341-411`: `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` (a plain, non-recursive `pthread_mutex_t` — `mylock_init` calls `pthread_mutex_init(l, NULL)`, no `PTHREAD_MUTEX_RECURSIVE` attribute) for the **entire** retry loop, and `xcecli_invoke_rejoin()` — which calls the registered rejoin callback — is invoked **synchronously from inside that loop, while the lock is held** (`client_failover.c:397`). Any rejoin callback that itself calls a public `xcecli_*` wrapper on the *same* `xcecli_ctx_t` (e.g. `xcecli_join`, which is itself `XCECLI_FAILOVER_CALL`-wrapped and tries to re-lock the same mutex) **self-deadlocks** — the calling thread blocks forever on its own already-held lock. This was never caught because `xcecli_set_reconnect_handler` has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client. + +This directly constrains every component's rejoin implementation in this spec: +- The callback **registered with the library** (worker's `checking_connection`→`xcecli_hello` path, scheduler's `xcecli_jobdispatch` path, apiserver's write-call path) must do **no I/O and call no `xcecli_*` function** — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return `0` so the interrupted operation resumes normally. +- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks and clears that flag and issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then. +- apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic **after** `xcecli_heartbeat` has returned (`XCECLI_FF_CALL` also releases its lock before returning), so it was never at risk. + +### Technology Stack + +| Layer | Choice / Version | Role in Feature | Notes | +|-------|------------------|------------------|-------| +| Shared library | C (gnu17), `src/lib/cluster.c` | New candidate→client-config conversion helper | Extends existing file alongside `xce_parse_peers` | +| Worker / Scheduler | C (gnu17) | Consume the existing `xcecli_ctx_t` mechanism directly | No new dependency | +| apiserver | Go 1.23.4 + CGO | Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard | `sync/atomic`, consistent with existing `heartbeat.go` pattern | + +## File Structure Plan + +### Modified / New Files +``` +src/include/xce/cluster.h # + declare xce_cluster_config_to_client_config() +src/lib/cluster.c # + implement xce_cluster_config_to_client_config() +src/lib/testsuite/test_cluster.c # + unit tests for the new helper + +src/worker/src/controller_connector.c # rewrite: delete per-candidate loop in + # xcecd_join_cluster/init_client_config; + # single init+start+join using full + # candidate list; register rejoin callback +src/worker/src/main.c # fix: populate cc_ctx->cluster_cfg BEFORE + # xcecd_start() (removes init-order bug) +src/worker/include/controller_connector.h # + rejoin callback function declaration + +src/scheduler/src/scheduler.c # rewrite: sched_init uses the new helper + + # single init+start+join; delete inline + # per-candidate loop; register rejoin callback +src/scheduler/src/main.c # no ordering change needed (already correct); + # simplified call site only + +src/frontend/apiserver/backend/clib.go # rewrite RegisterToBackend: single connect + # using the full candidate list via CGO calls + # into xce_parse_peers/xce_cluster_config_validate/ + # xce_cluster_config_to_client_config; cgo + # preamble gains a static C shim (mirrors the + # existing xce_backend_handler/go_push_handler + # pattern) that calls the reconnect.go export, + # passed to xcecli_set_reconnect_handler; the + # heartbeat tick callback explicitly re-invokes + # the same rejoin path on recovery (see below) +src/frontend/apiserver/backend/health.go # NEW: backend-degraded circuit-breaker state +src/frontend/apiserver/backend/reconnect.go # NEW: exported Go rejoin logic (re-sends + # xcecli_join for HostRole_apiserver, clears + # health.go state); called from clib.go's C + # shim AND directly from the heartbeat callback +src/frontend/apiserver/backend/heartbeat.go # MODIFIED: tick callback detects a + # degraded-to-healthy transition after + # xcecli_heartbeat succeeds and explicitly + # triggers reconnect.go's rejoin logic +src/frontend/apiserver/utils/config.go # no schema change (Peers already exists) + +src/worker/testsuite/test_e2e_multimaster_join.c # NEW: candidate-list join + reconnect E2E +src/scheduler/testsuite/test_e2e_multimaster_join.c # NEW: same, scheduler +src/frontend/apiserver/backend/health_test.go # NEW: circuit-breaker unit tests +src/frontend/apiserver/backend/clib_test.go # + candidate-list parity tests (existing file) +``` + +> `controller/` and `libxce`'s failover mechanism itself are not modified — only consumed. + +## System Flows + +### Startup connect (Req 1) + +```mermaid +sequenceDiagram + participant Daemon + participant Convert as xce_cluster_config_to_client_config + participant Ctx as xcecli_ctx (client.c) + participant CandA as Candidate A (standby) + participant CandB as Candidate B (primary) + + Daemon->>Convert: full candidate list (validated) + Convert-->>Daemon: xce_client_config_t (nmasters=N) + Daemon->>Ctx: xcecli_ctx_init + xcecli_start + Ctx->>CandA: connect + CandA-->>Ctx: connected, not primary + Ctx->>Daemon: xcecli_join (role) + Ctx->>CandA: join request + CandA-->>Ctx: standby redirect + Ctx->>Ctx: rotate to next candidate + Ctx->>CandB: connect + join + CandB-->>Ctx: joined as primary + Ctx-->>Daemon: xcecli_join success +``` + +### Runtime failover (Req 2, Req 3) + +```mermaid +sequenceDiagram + participant Daemon + participant Ctx as xcecli_ctx lock held + participant OldPrimary as Old Primary dies + participant NewPrimary as New Primary + + Daemon->>Ctx: xcecli_* call existing traffic + Ctx->>OldPrimary: send + OldPrimary--xCtx: connection lost + Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off + Ctx->>NewPrimary: reconnect + NewPrimary-->>Ctx: connected + Ctx->>Daemon: invoke rejoin callback lock still held + Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0 + Ctx->>NewPrimary: resume original call + NewPrimary-->>Ctx: reply + Ctx-->>Daemon: original call result, lock released + Note over Daemon: after the call returns, lock is free + Daemon->>Daemon: check and clear needs-rejoin flag + Daemon->>Ctx: xcecli_join role separate call + Ctx->>NewPrimary: join + NewPrimary-->>Ctx: joined + Ctx-->>Daemon: rejoin complete +``` + +Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: allocator thread; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way. + +### apiserver fail-fast while degraded (Req 4) + +```mermaid +sequenceDiagram + participant Client as API Client + participant Handler as apiserver handler + participant Health as health.go + participant Ctx as xcecli_ctx (gCtx) + + Client->>Handler: HTTP request needing backend + Handler->>Health: is backend degraded? + alt degraded + Health-->>Handler: yes + Handler-->>Client: 503 service unavailable (fast) + else healthy + Health-->>Handler: no + Handler->>Ctx: backend call + alt call exhausts retry budget + Ctx-->>Handler: error + Handler->>Health: mark degraded (with TTL) + Handler-->>Client: 503 service unavailable + else call succeeds + Ctx-->>Handler: result + Handler-->>Client: 200 with result + end + end +``` + +### apiserver idle-period reconnect (Req 2.4) + +```mermaid +sequenceDiagram + participant HB as heartbeat.go tick + participant Ctx as xcecli_ctx (gCtx, FF_CALL) + participant Rejoin as reconnect.go + participant Health as health.go + participant NewPrimary as New Primary + + Note over HB: no client requests in flight + HB->>Ctx: xcecli_heartbeat (fire-and-forget) + Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect + Ctx->>NewPrimary: reconnected transport only + Ctx-->>HB: rc success (was failing, now succeeds) + HB->>HB: detect failure-to-success transition + HB->>Rejoin: explicitly trigger rejoin + Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver) + NewPrimary-->>Rejoin: joined + Rejoin->>Health: MarkHealthy() +``` + +This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (`XCECLI_FAILOVER_CALL`, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive. + +## Requirements Traceability + +| Requirement | Summary | Components | Interfaces | Flows | +|-------------|---------|------------|------------|-------| +| 1.1-1.3 | Try each candidate in order at startup | libxce (unchanged), ConfigConverter | `xcecli_start`, `xcecli_join` | Startup connect | +| 1.4 | Worker uses full list on FIRST attempt | WorkerControllerConnector | `controller_connector.c` init order fix | Startup connect | +| 1.5 | Single/no-candidate mode unchanged | All three | ConfigConverter (`nmasters=1` path) | — | +| 1.6 | Clear startup failure if all candidates fail | All three | existing `xcecli_start`/`xcecli_join` return codes | Startup connect | +| 2.1-2.3 | Automatic reconnect + retry until success or shutdown | libxce (unchanged), all three's rejoin callbacks | `xcecli_set_reconnect_handler` | Runtime failover | +| 2.4 | apiserver resumes push notifications after reconnect (incl. idle-period recovery) | ApiserverBackendBridge | `reconnect.go`'s actual-rejoin function re-sends `xcecli_join(HostRole_apiserver)`, triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from `heartbeat.go` on idle-period recovery | Runtime failover, apiserver idle-period reconnect | +| 3.1-3.2 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation preserved), SchedulerControllerClient (allocator/picker thread separation) | existing thread structure | Runtime failover | +| 4.1-4.2 | apiserver fails fast to clients while degraded | ApiserverBackendHealth | `health.go` | apiserver fail-fast flow | +| 5.1-5.2 | Per-candidate connect/reconnect logging | All three | log statements at each rotation | Startup connect, Runtime failover | +| 6.1-6.2 | Consistent candidate-list syntax | ApiserverConfigParity | CGO call to `xce_parse_peers`/`xce_cluster_config_validate` | Startup connect | + +## Components and Interfaces + +| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts | +|-----------|--------------|--------|---------------|-------------------|-----------| +| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.6, 6.1 | `xce_cluster_config_validate` (P0) | Service | +| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service | +| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service | +| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service | +| ApiserverBackendHealth | apiserver (Go) | Fail-fast circuit breaker for backend-down state | 4.1-4.2 | ApiserverBackendBridge (P0, consumes rejoin/error signals) | State | + +### Shared Library + +#### ConfigConverter + +| Field | Detail | +|-------|--------| +| Intent | Turn a validated `xce_cluster_config_t` into a fully-populated `xce_client_config_t`, clamped to `N_MAX_MASTERS` with a WARN on truncation | +| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 6.1 | + +**Responsibilities & Constraints** +- Pure, socket-free transformation (mirrors `xce_test`'s existing inline logic at `test.c:360-372`, generalized into one reusable function). +- Does not call `xce_cluster_config_validate()` itself — callers validate first (existing contract, unchanged) and pass a pre-validated config. +- Does not own `from`/`key` (identity/auth) — caller sets those on the output struct after conversion, matching how `xce_test`, worker, and scheduler already source them independently (CLI/env-specific). + +**Dependencies** +- Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0) +- Outbound: none (leaf function) + +**Contracts**: Service [x] + +##### Service Interface +```c +/* xce/cluster.h */ +/** + * xce_cluster_config_to_client_config() — convert a validated candidate list + * into an xce_client_config_t's master fields (nmasters/masters/master_ports). + * + * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order. + * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and + * logs a WARN identifying the truncated count (mirrors xce_test's existing + * clamp behavior). + * + * Does NOT set out->from / out->key — caller sets those separately. + * Does NOT validate cfg — caller must have already called + * xce_cluster_config_validate() successfully. + * + * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1. + */ +int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg, + xce_client_config_t* out); +``` +- Preconditions: `cfg` already passed `xce_cluster_config_validate()`. +- Postconditions: `out->nmasters` in `[1, N_MAX_MASTERS]`; `out->masters[i]`/`out->master_ports[i]` populated for `i < out->nmasters`, in the same order as `cfg->candidates[]`. +- Invariants: never writes beyond `N_MAX_MASTERS` entries regardless of `cfg->candidates_count`. + +### Worker + +#### WorkerControllerConnector + +| Field | Detail | +|-------|--------| +| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect | +| Requirements | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 | + +**Responsibilities & Constraints** +- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:237-311`) with: build `xce_client_config_t` via ConfigConverter once (candidates from `cc_ctx->cluster_cfg` when `--peers` given, else today's single-host fallback path unchanged per Req 1.5), then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`. +- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4). +- Registers a rejoin callback (`xcecli_set_reconnect_handler`) that only sets a `needs_rejoin` flag and returns `0` — per "Rejoin callback constraint" above, it must not call `xcecli_join` (or any `xcecli_*` function) directly, since the library invokes it while `ac->failover.lock` is still held. +- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks and clears `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup. +- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from `hardware_monitor`/`system_monitor`/`process_manager` update paths (Req 3.1) — no new thread introduced. + +**Dependencies** +- Inbound: `xcecd_start()` (P0) +- Outbound: ConfigConverter (P0), `xcecli_*` (P0) + +**Contracts**: Service [x] + +**Implementation Notes** +- Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification. +- Validation: implementation must confirm (code read, not test-only) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread — flagged as a Validation Hook, not re-verified in this design pass. +- Risks: deleting the hand-rolled loop must not silently drop the `MAX_RETRY_COUNT`/`RETRY_INTERVAL` bounded-retry-then-`handle_disconnect()` behavior in `checking_connection()` — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous. + +### Scheduler + +#### SchedulerControllerClient + +| Field | Detail | +|-------|--------| +| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect | +| Requirements | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 | + +**Responsibilities & Constraints** +- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:462-495`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call. +- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). After each `xcecli_jobdispatch` call returns, the allocator checks and clears the flag; if set, it calls `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate call before continuing. +- No new thread: `xcecli_jobdispatch` (the only backend call site, `scheduler.c:1289`) continues to run from the existing `sched_allocator` thread; blocking during a reconnect/backoff cycle is an accepted, benign degradation since the allocator has no useful work without a controller anyway, and `sched_job_picker` runs on its own separate thread unaffected (Req 3.1-3.2 satisfied by the existing thread split, not new code). + +**Dependencies** +- Inbound: `sched_init()` (P0) +- Outbound: ConfigConverter (P0), `xcecli_*` (P0) + +**Contracts**: Service [x] + +**Implementation Notes** +- Integration: preserve the existing `WARN`/`INFO` log lines around candidate parsing (`main.c`) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally). +- Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting. + +### apiserver + +#### ApiserverBackendBridge + +| Field | Detail | +|-------|--------| +| Intent | Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker | +| Requirements | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 | + +**Responsibilities & Constraints** +- `RegisterToBackend()` (`clib.go`) is rewritten: build a C `xce_cluster_config_t` from `config.Peers` (or the single `Address`/`Port` fallback, Req 1.5) by calling `xce_parse_peers` via CGO (not the existing hand-written Go `parsePeer`), validate with `xce_cluster_config_validate`, convert with ConfigConverter, then call `xce_backend_init` **once** with the resulting `nmasters > 1` config — replacing the current `for _, peer := range config.Peers { tryConnect(...) }` Go loop entirely. +- `reconnect.go` holds two plain Go functions: (1) a `//export`-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a `needs_rejoin` atomic flag and returns `0` (per "Rejoin callback constraint" above: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join`/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears `ApiserverBackendHealth`'s degraded state on success. +- The flag-setter is **registered with the C library from `clib.go`'s own cgo preamble**, via a small static C shim (`xce_backend_rejoin_handler`, mirroring the existing `xce_backend_handler`/`go_push_handler` pattern already used for push dispatch) passed to `xcecli_set_reconnect_handler` — cgo symbol resolution is per-file (see `wire_dispatch_support.go`'s documented rationale for the identical `go_push_handler` wrapper), so the shim and the registration call must live in the same file as the `extern` declaration, not standalone in `reconnect.go`. +- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check-and-clear are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper. +- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: the heartbeat tick callback (`heartbeat.go`) tracks the previous tick's outcome; when a tick transitions from failure to success (transport just recovered), it explicitly calls `reconnect.go`'s rejoin function itself, then clears `ApiserverBackendHealth`. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code, just an explicit trigger point the write-call path already gets for free from the library. + +**Dependencies** +- Inbound: `cmd/main.go` (P0) +- Outbound: ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome) + +**Contracts**: Service [x] + +##### Service Interface +```go +// backend/clib.go +// RegisterToBackend initialises the client using the full configured +// candidate list (config.Peers, or the single Address/Port fallback) and +// registers the rejoin callback. Single call — no per-peer Go loop. +func RegisterToBackend(config *BackendConfig) error +``` +- Preconditions: `StartNotificationListener` already called (unchanged from today). +- Postconditions: on success, `gCtx.ctx` is joined with `nmasters` reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered. +- Invariants: candidate-list parsing/validation errors surface the same way `xce_parse_peers`/`xce_cluster_config_validate` report them in C (Req 6.2) — not a separately-worded Go error. + +**Implementation Notes** +- Integration: `parsePeer` (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1). +- Integration: the rejoin C shim must be defined in `clib.go`'s cgo preamble (not `reconnect.go`) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise. +- Risks: CGO error surfaces (C `-EINVAL` etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code. + +#### ApiserverBackendHealth + +| Field | Detail | +|-------|--------| +| Intent | Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded | +| Requirements | 4.1, 4.2 | + +**Responsibilities & Constraints** +- New `health.go`: an atomic "degraded-until" timestamp. `IsDegraded() bool` — true if `now < degraded-until`. `MarkDegraded(ttl time.Duration)` — called by backend-call wrapper functions in `clib.go` when a call's returned error indicates retry-budget exhaustion (e.g. `E_XCE_NOT_READY`). `MarkHealthy()` — called by the rejoin callback (`reconnect.go`) on a successful rejoin. +- Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing `gCtx.ctx`-based calls in `clib.go`) check `IsDegraded()` first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch `gCtx`). + +**Dependencies** +- Inbound: ApiserverBackendBridge (sets state on outcomes) +- Outbound: none + +**Contracts**: State [x] + +##### State Management +- State model: single atomic degraded-until timestamp (int64 unix-nano via `sync/atomic`, same pattern as existing `heartbeatIntervalSecs` in `heartbeat.go`). +- Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per `tech.md`). +- Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request. + +**Implementation Notes** +- Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag. +- Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit `MarkHealthy()`. + +## Error Handling + +### Error Strategy +No new error taxonomy — reuse the existing controller status codes (`E_XCE_REDIRECT`/`E_XCE_STANDBY`/`E_XCE_NOT_READY`, `errnox.h`) that `client_failover.c` already classifies, and the existing `xce_cluster_config_validate()` `-EINVAL` cases for malformed candidate-list config. + +### Error Categories and Responses +- **Malformed candidate-list config** (Req 1.6, 6.2): `xce_parse_peers`/`xce_cluster_config_validate` failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route). +- **All candidates unreachable at startup** (Req 1.6): `xcecli_start`/`xcecli_join` returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: `RegisterToBackend` returns an error, `cmd/main.go`'s existing handling applies unchanged). +- **Runtime disconnect** (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes. +- **Candidate count exceeds `N_MAX_MASTERS`**: ConfigConverter clamps and WARNs (not an error) — matches `xce_test`'s existing behavior exactly. +- **apiserver backend degraded** (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error. + +### Monitoring +Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (`XCECD_LOG` for worker, the scheduler's existing `INFO`/`WARN` macros, `zap` for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path. + +## Testing Strategy + +- **Unit Tests**: + - `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`). + - apiserver `health.go`: degraded→healthy via `MarkHealthy()`, via TTL expiry; concurrent goroutine safety (`health_test.go`). + - apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (`clib_test.go`). + - Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real `ac->failover.lock` held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found. +- **Integration Tests**: + - Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's `testsuite/`. + - Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI. +- **E2E Tests**: + - Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new `test_e2e_multimaster_join.c` per C component plus an apiserver-side check in the existing HA soak harness. + - apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4). + - apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1). + +## Security Considerations +No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced. diff --git a/.kiro/specs/component-failover-adoption/requirements.md b/.kiro/specs/component-failover-adoption/requirements.md new file mode 100644 index 00000000..12c5c710 --- /dev/null +++ b/.kiro/specs/component-failover-adoption/requirements.md @@ -0,0 +1,90 @@ +# Requirements Document + +## Project Description (Input) +`component-failover-adoption` — pre-scoped roadmap backlog item (`roadmap.md:64`, recorded 2026-06-16, HARD-dependent on `libxce-failover`, which shipped GO 2026-07-18 — this item is now unblocked). Make the C daemons (scheduler, worker) and the Go apiserver actually exercise libxce's transparent multi-master failover, matching xce_test's `masters`-setting behavior. Two roadmap-defined integration items: +1. Each component must hand libxce the **full master/candidate list**, not a single endpoint (today even the CLI uses only `conf->masters[0]`). +2. Bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons this must be verified **liveness-safe**: it must never stall a thread that also drives heartbeats (the project-wide "never block iomgr/delaymgr" thread-role contract, `ha-async-io.md`). + +Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. `xce_test` is explicitly OUT of scope — it is the already-correct in-spec verification client, not a target. + +### Background (verification findings, 2026-07-19) + +`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates. + +None of the three production daemons match this pattern today; each has a different gap: + +- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover). +- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate. +- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only. + +In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead. + +A separate, partially-built shared primitive already exists in libxce (`src/include/xce/cluster.h`: `xce_connect_cluster()`, `xce_reconnect_init/step/fini()`, backed by `xce_parse_peers()`/`xce_cluster_config_validate()`) — but its production connect path (`do_connect()` in `src/lib/cluster.c`) is a stub that always returns `NULL` outside unit tests, and it operates on a raw `endpoint_t*`, not the higher-level `xcecli_ctx_t` the daemons actually use for the join/request/reply protocol. No production code calls it. Whether the design phase completes/adopts this primitive or builds directly on `xce_client_config_t.nmasters` (the `libxce-failover`/`client_failover.c` path xce_test already uses) is a design decision, not a requirements one — both existing mechanisms are noted here as prior art this feature must reconcile with, not duplicate. + +### Background (verification findings, 2026-07-19) + +`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates. + +None of the three production daemons match this pattern; each has a different gap: + +- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover). +- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate. +- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only. + +In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead. + +## Boundary Context + +- **In scope**: scheduler, worker, and apiserver each using their *full* configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established. +- **Out of scope**: `xce_test` (already the correct reference client, not a target of this feature); raising the validated 3-candidate limit; controller-side changes (peer authentication, `validate_host()`, redirect-reply population — already shipped by `libxce-failover`); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate `worker-rejoin-resync` item — this feature restores the *connection*, not job-state content). +- **Adjacent expectations**: depends on `libxce-failover` (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list. + +## Requirements + +### Requirement 1: Full candidate list used on initial connection +**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to attempt their initial controller connection using the complete configured candidate list, so that startup succeeds even when the first-listed candidate is not the current primary. + +#### Acceptance Criteria +1. When a component is started with more than one controller candidate configured, the component shall attempt connection to each configured candidate, in configured order, until one accepts the join as primary. +2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart. +3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error. +4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects. +5. Where only one candidate (or none) is configured, the component shall behave as it does today in single-controller mode. +6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted. + +### Requirement 2: Automatic reconnect after controller failover +**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons. + +#### Acceptance Criteria +1. When a component's active connection to its controller is lost, the component shall attempt to reconnect using the complete configured candidate list, not only the previously connected candidate. +2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator. +3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again. +4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart. + +### Requirement 3: Reconnect retry does not block other liveness-critical work +**Objective:** As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive. + +#### Acceptance Criteria +1. While a worker or scheduler is retrying a controller connection (including any backoff delay between attempts), the component shall continue performing its other liveness-critical duties (worker: hardware/process monitoring and job-state tracking; scheduler: its allocation/decision processing) without being blocked by the retry attempt or backoff delay. +2. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover. + +### Requirement 4: apiserver behavior toward API clients during backend disconnection +**Objective:** As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served. + +#### Acceptance Criteria +1. When apiserver's connection to the controller is down and a reconnect is in progress, apiserver shall respond to API requests that require the backend with a clear service-unavailable style error rather than hanging until a connection is restored. +2. While the backend reconnect is in progress, apiserver shall continue attempting to serve API requests that do not require the backend connection. + +### Requirement 5: Operator-visible connect/reconnect diagnostics +**Objective:** As an HA cluster operator, I want visibility into which controller candidates each component tried and why each attempt failed, so that I can distinguish a real outage from a configuration mistake. + +#### Acceptance Criteria +1. When a component attempts a connection or reconnection to a controller candidate, the component shall log the candidate attempted and the outcome (success, unreachable, or standby-redirect). +2. If a component exhausts all configured candidates during a connection or reconnection attempt, then the component shall log a summary identifying every candidate attempted and the reason each failed. + +### Requirement 6: Consistent candidate-list configuration across components +**Objective:** As an HA cluster operator, I want to configure the controller candidate list the same way for scheduler, worker, and apiserver, so that I do not need to learn or maintain three different configuration formats for the same setting. + +#### Acceptance Criteria +1. The candidate-list configuration accepted by apiserver shall use the same `[name=]host:port[,...]` syntax already accepted by scheduler and worker. +2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration. diff --git a/.kiro/specs/component-failover-adoption/research.md b/.kiro/specs/component-failover-adoption/research.md new file mode 100644 index 00000000..b8413735 --- /dev/null +++ b/.kiro/specs/component-failover-adoption/research.md @@ -0,0 +1,120 @@ +# Research & Design Decisions + +## Summary +- **Feature**: `component-failover-adoption` +- **Discovery Scope**: Extension (light discovery — libxce, worker, scheduler, apiserver already exist; this wires three consumers onto an existing library mechanism) +- **Key Findings**: + - `libxce-failover` (shipped, GO 2026-07-18) already built a **complete, transparent, per-call multi-master retry/rotate mechanism** (`XCECLI_FAILOVER_CALL` / `XCECLI_FAILOVER_CALL_COUNT` / `XCECLI_FF_CALL` in `client_internal.h`, wired into every public `xcecli_*` call in `client.c`, including `xcecli_join` and the initial `xcecli_start` connect). It requires only `xce_client_config_t.nmasters > 1` to activate. **No production code sets `nmasters` above 1.** + - A **rejoin hook** (`xcecli_set_reconnect_handler`) exists specifically so a consumer can re-send its join after a transparent reconnect. It has **zero production callers** — only its own setter and a unit test reference it. + - A second, older, **half-built and unused** primitive (`xce_connect_cluster()` / `xce_reconnect_init/step/fini()` in `src/lib/cluster.c`) also exists. Its production connect path (`do_connect()`) is a stub that always returns `NULL` outside unit tests, and it operates on raw `endpoint_t*`, not `xcecli_ctx_t`. No production code calls it. It predates/overlaps `libxce-failover` and is superseded by it. + - This is a genuine "wire it up" feature, not a "build failover" feature — confirmed by the roadmap: `component-failover-adoption` (roadmap.md:64) was already recorded 2026-06-16 with this exact framing, HARD-blocked on `libxce-failover`, now unblocked. + +## Research Log + +### Why do worker/scheduler/apiserver each hand-roll a candidate loop instead of using the library mechanism? +- **Context**: Initial verification (this conversation) found worker, scheduler, and apiserver each independently loop over configured candidates, calling `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` fresh for each candidate with `nmasters` hardcoded to `1` per attempt. +- **Sources Consulted**: `src/lib/client.c`, `src/lib/client_internal.h`, `src/worker/src/controller_connector.c`, `src/scheduler/src/scheduler.c`, `src/frontend/apiserver/backend/clib.go`, `roadmap.md`. +- **Findings**: `xcecli_start` (client.c:125-188) already rotates through ALL configured masters on the very first connect (bug-854 fix, comment at line 136). `xcecli_join` (client.c:291-300) is wrapped in `XCECLI_FAILOVER_CALL`, so it already retries and rotates on standby/redirect/transport-error responses. The three daemons' hand-rolled loops predate this (or were never updated after it landed) and now duplicate — imperfectly — what the library does correctly. +- **Implications**: The fix is subtractive as much as additive: delete the per-daemon candidate loops, populate `nmasters`/`masters[]`/`master_ports[]` fully once, and let the existing library mechanism do the rest. This directly satisfies Requirements 1 and most of Requirement 2 without new retry/rotation logic. + +### Does the library mechanism, once enabled, also cover runtime reconnect (Requirement 2) automatically? +- **Findings**: Yes for any component whose normal operation already calls `xcecli_*` functions periodically or on demand — every one of those calls is wrapped in `XCECLI_FAILOVER_CALL`/`XCECLI_FAILOVER_CALL_COUNT`, so a transport failure on any call transparently triggers rotate+reconnect+retry within that same call. Fire-and-forget calls (`xcecli_nop`, `xcecli_heartbeat`) use `XCECLI_FF_CALL` instead: a bounded single best-effort reconnect attempt, never the full retry budget — this is the library's own liveness-safety design for exactly the concern in Requirement 3. +- **Implications**: Scheduler's `xcecli_jobdispatch` calls (scheduler.c:1289, called from the `sched_allocator` thread) and apiserver's per-request backend calls (`gCtx.ctx` in clib.go) get transparent reconnect for free once `nmasters` is set correctly — no new polling/reconnect loop needs to be written for either component. + +### Does anything need a NEW re-join after a transparent reconnect, beyond what the library does automatically? +- **Findings**: The library's rejoin hook exists precisely because reconnecting the transport does not re-establish application-level cluster membership — a new primary does not automatically know a worker/scheduler/apiserver rejoined unless it re-sends its join (role) message. `xcecli_reconnect_cb` "runs on the calling thread" and returns 0 to resume the interrupted op or <0 to treat the rejoin as failed (loop rotates/backs off again). +- **Implications**: All three components must register a rejoin callback that re-issues `xcecli_join(ctx, , ...)` (with worker also re-sending `hostattrs_t`). Without it, a component would reconnect at the transport level but the new primary would not recognize it as joined — likely surfacing as silently-dropped pushes/notifications rather than an obvious error. + +### Thread-affinity of blocking retry calls (Requirement 3) +- **Findings**: Worker already isolates all controller-connection work (`controller_connector_start`, `xcecd_join_cluster`, and the periodic `checking_connection()` health-check) onto one dedicated thread (`start_controller_connector_thread` / `connector_to_controller`), separate from `hardware_monitor`/`system_monitor`/`process_manager` update calls in `xcecd_start()`. This isolation already satisfies Requirement 3 structurally for the worker — it is not something this feature needs to newly build, only preserve while removing the redundant candidate loop. +- Scheduler already runs `sched_job_picker` and `sched_allocator` as two separate `pthread_create`d threads (scheduler.c:511,516); `xcecli_jobdispatch` (the only call site found that could block on retry, scheduler.c:1289) executes from the allocator thread only. A blocked allocator during a controller outage is a benign, expected degradation (there is no work the allocator can do without a controller anyway) and does not stall the job picker or any liveness signal scheduler sends elsewhere — scheduler has no independent heartbeat-carrying thread to protect. No new threading is needed for the scheduler. +- apiserver: unlike the C daemons, ALL request-handling goroutines that touch the backend share the same `gCtx.ctx`, serialized by the library's per-context `failover.lock`. Under a real outage this could queue every concurrent API request behind the full retry budget, which is a client-experience problem (Requirement 4), not a cluster-liveness problem. Needs a lightweight fail-fast guard in the Go layer — see Design Decisions. +- **Implications**: No new dedicated thread is required for worker or scheduler (existing isolation already suffices — flagged as a Validation Hook to confirm during implementation, not re-derived here). apiserver needs a new, small piece of state (not a thread) to short-circuit blocking calls when the backend is known-down. + +### Configuration surface mismatch: `N_MAX_MASTERS` vs `MAX_HA_CANDIDATES` +- **Findings**: `src/include/client.h:28` defines `N_MAX_MASTERS (3)` — the hard cap on `xce_client_config_t.masters[]`. `src/include/config.h:78` defines `MAX_HA_CANDIDATES 4` — the cap on `xce_cluster_config_t.candidates[]` used by `xce_parse_peers`. These are different constants. `xce_test` (`test.c:360-367`) already handles the mismatch correctly: parse up to 4 candidates, then clamp to `N_MAX_MASTERS` (3) with a `WARN` if truncated. +- **Implications**: Any new shared conversion helper must replicate this exact clamp-and-warn behavior (not silently drop entries, not error out) so all components behave identically when 4 candidates are configured. Reconciling the two constants into one is out of scope (pre-existing, unrelated to this feature). + +### TLS/CA-cert wiring gap (pre-existing, out of boundary) +- **Findings**: `xce_client_config_t` (the struct actually consumed by `xcecli_ctx_init`/`xcecli_start`, i.e. the real connection path) has no CA-cert field at all — only `nmasters`/`masters`/`master_ports`/`from`/`key`. The `ca_cert_path` that worker/scheduler's `--ca-cert` flag populates on `xce_cluster_config_t` is validated by `xce_cluster_config_validate()` but is only ever read by the orphaned `xce_connect_cluster()` stub, never by the real connect path. +- **Implications**: This is a pre-existing gap unrelated to candidate-list wiring — the real connection already establishes TLS today (single-master) via whatever global SSL context setup exists outside this struct. Not this feature's concern; noted here so it isn't mistaken for something this design should fix. + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks / Limitations | Notes | +|--------|-------------|-----------|---------------------|-------| +| Adopt existing `xce_client_config_t` / `XCECLI_FAILOVER_CALL` mechanism (chosen) | Populate the full candidate list once per component, register a rejoin callback, delete hand-rolled loops | Already built, tested (`libxce-failover`), and exactly what `xce_test` uses today; minimal new code | None material — this is the reference-verified path | Matches roadmap's own framing of this backlog item | +| Complete and adopt `xce_connect_cluster()`/`xce_reconnect_*` | Finish the stubbed `do_connect()`, wire it under each daemon | N/A | Duplicate mechanism to the one already shipped; raw-`endpoint_t` API doesn't carry the join/request/reply protocol the daemons need — would need its own substantial new integration layer on top | Rejected — would build a second, redundant failover mechanism | +| Build a new per-component reconnect loop | Fix each daemon's existing hand-rolled loop in place, independently | Minimal library changes | Perpetuates 3 divergent implementations (the root problem this feature exists to fix); does not gain the library's already-tested retry/backoff/classification logic | Rejected — directly contradicts the roadmap item's stated goal | + +## Design Decisions + +### Decision: Adopt the existing `xcecli_ctx_t` / `XCECLI_FAILOVER_CALL` mechanism; do not build new failover logic +- **Context**: Requirements 1-3 need multi-candidate connect + runtime reconnect + liveness-safe retry. +- **Alternatives Considered**: see Architecture Pattern Evaluation above. +- **Selected Approach**: Each component populates a fully-formed `xce_client_config_t` (all configured candidates, not just the first) once, calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once, and registers a rejoin callback via `xcecli_set_reconnect_handler`. All subsequent `xcecli_*` calls already retry/rotate/reconnect transparently. +- **Rationale**: The mechanism is shipped, tested, and is the exact one `xce_test`'s `masters` setting already uses correctly. Building anything new would duplicate it. +- **Trade-offs**: None material for worker/scheduler. For apiserver, moving off the current Go-level per-peer loop means the CGO layer becomes responsible for candidate rotation instead of Go — acceptable since it is strictly more capable (transparent runtime reconnect, not just startup). +- **Follow-up**: Confirm during implementation that worker's hardware/process-monitor update calls truly run off the connector thread (Validation Hook, not re-verified byte-for-byte in this research pass). + +### Decision: One new shared C helper converts `xce_cluster_config_t` → `xce_client_config_t` +- **Context**: All three components need to turn a parsed/validated candidate list into the `nmasters`/`masters[]`/`master_ports[]` triple, replicating `xce_test`'s clamp-to-`N_MAX_MASTERS`-with-WARN behavior (see mismatch finding above). Today this logic exists ad hoc in `test.c` only. +- **Alternatives Considered**: (1) Duplicate the clamp/copy logic in worker, scheduler, and apiserver's CGO layer separately. (2) One shared helper in `src/lib/cluster.c` (where `xce_parse_peers`/`xce_cluster_config_validate` already live), declared in `xce/cluster.h`. +- **Selected Approach**: (2) — a single `xce_cluster_config_to_client_config()` helper, reused by worker (C), scheduler (C), and apiserver (via CGO). +- **Rationale**: DRY; the clamp-and-warn behavior must be identical across components (Requirement 6's config-syntax-parity spirit extends naturally to the derived struct, not just the input string); one function is one thing to test. +- **Trade-offs**: None — this is pure generalization of logic that would otherwise be copy-pasted three times. + +### Decision: apiserver calls the existing C `xce_parse_peers`/`xce_cluster_config_validate` via CGO instead of maintaining a second Go parser +- **Context**: Requirement 6 requires apiserver's candidate-list syntax to match scheduler/worker exactly. apiserver currently has its own hand-written Go `parsePeer` function. +- **Alternatives Considered**: (1) Keep/harden the Go parser to match C syntax by inspection. (2) Call the real C parser via CGO (apiserver already links libxce via CGO for everything else). +- **Selected Approach**: (2). +- **Rationale**: Guarantees byte-identical parsing forever (no drift risk), and apiserver already pays the CGO cost for every other backend interaction. +- **Trade-offs**: One more CGO surface, consistent with the existing `backend/` package pattern. + +### Decision: apiserver gets a small Go-side "backend health" circuit breaker for Requirement 4 +- **Context**: Requirement 4 needs apiserver to fail fast toward its own HTTP clients when the backend is down, rather than queue every request behind the library's full retry budget. +- **Alternatives Considered**: (1) Let every request block on the existing per-ctx lock and rely on Gin/client-side timeouts. (2) Track backend health from call outcomes + the rejoin callback, and check it before issuing a blocking backend call. +- **Selected Approach**: (2) — new `backend/health.go`: an atomic "degraded since" state, set when a call's classified outcome is retry-budget-exhaustion, cleared when the rejoin callback fires (successful reconnect+rejoin). Backend-call wrappers in `clib.go` check it first and return a clear error immediately while degraded, instead of entering the blocking call. +- **Rationale**: Small, localized, reuses signals the library already produces (call rc, rejoin callback) rather than adding new polling. +- **Trade-offs**: A short window exists where the flag is stale (a request arrives just as the backend goes down, before any call has failed) — acceptable, matches typical circuit-breaker behavior; the request in flight simply experiences the one blocking retry cycle before the flag is set for subsequent requests. + +### Decision: rejoin callbacks must be lock-free flag-setters, never direct `xcecli_join` calls +- **Context**: A second `/kiro-validate-design` pass, cross-checking the library-invoked rejoin path against `client_failover.c` directly rather than only its header comments, found that `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` — a plain, non-recursive `pthread_mutex_t` (`mylock_init` → `pthread_mutex_init(l, NULL)`) — for its entire retry loop, and `xcecli_invoke_rejoin()` (which calls the registered rejoin callback) runs synchronously *inside* that loop, lock held (`client_failover.c:397`). Every component's original design had its rejoin callback call `xcecli_join(...)` directly — itself `XCECLI_FAILOVER_CALL`-wrapped, so it would try to re-lock the same mutex on the same thread: a guaranteed self-deadlock on the very first real failover. This was never caught because `xcecli_set_reconnect_handler` has zero production callers and the existing unit test's callback never calls back into the client (it's a no-op probe). +- **Alternatives Considered**: (1) Have the callback call `xcecli_join` directly (original design — deadlocks). (2) Expose a new, lock-free/raw join primitive from libxce for rejoin callbacks to use (would require modifying `libxce-failover`, violating this spec's Allowed Dependencies boundary of consuming it read-only). (3) Callback only sets a lightweight flag and returns immediately (no I/O, no lock); the daemon's own calling code checks and clears the flag *after* the triggering call returns (lock free by then) and issues the real `xcecli_join` as a separate call. +- **Selected Approach**: (3), applied uniformly to worker (`checking_connection`), scheduler (the allocator, after `xcecli_jobdispatch`), and apiserver (each backend-call wrapper, after its `C.xcecli_*` call). apiserver's heartbeat-triggered path (Task 4.4) was already structured this way by construction (it acts after `xcecli_heartbeat` returns) and needed no change. +- **Rationale**: Fixes the deadlock without touching `libxce` at all, staying inside this spec's stated boundary; the fix is a small, uniform pattern applied identically across all three components rather than a special case. +- **Trade-offs**: One extra check-and-clear step after each relevant call site, in exchange for correctness; negligible cost. +- **Follow-up**: Worth flagging upstream to `libxce-failover`'s owners that the rejoin-hook contract (`client.h`'s doc comment for `xcecli_reconnect_cb`) doesn't document this lock-held constraint — a future consumer without this spec's context could reintroduce the deadlock. Not fixed here (out of boundary), but the regression-guard unit test added in design.md's Testing Strategy exists specifically because of this gap in the upstream contract's documentation. + +### Decision: apiserver's rejoin registration uses a C shim in `clib.go`, not a standalone `reconnect.go` export +- **Context**: `/kiro-validate-design` review flagged that design.md originally described `reconnect.go` as "a `//export`ed Go function registered via `xcecli_set_reconnect_handler`," implying direct registration. This codebase already documents (`wire_dispatch_support.go:154-159`, explaining why `go_push_handler` needs the `xce_backend_handler` C shim) that cgo symbol resolution is per-file — a `C.` reference only resolves in the file holding its own matching `extern` declaration. +- **Alternatives Considered**: (1) Register the exported Go function directly from `reconnect.go`. (2) Add a static C shim + `extern` declaration in `clib.go`'s own cgo preamble (mirroring the proven `xce_backend_handler` pattern) and register that. +- **Selected Approach**: (2). +- **Rationale**: Matches the codebase's own documented, working pattern instead of risking the same resolution failure `xce_backend_handler` was built to avoid. +- **Trade-offs**: None — same amount of code, just correctly placed. +- **Follow-up**: none; resolved by construction in design.md. + +### Decision: apiserver's heartbeat prober explicitly triggers rejoin on recovery +- **Context**: `/kiro-validate-design` review found, by direct inspection of `client_internal.h`/`client_failover.c`, that `XCECLI_FF_CALL` (backing `xcecli_heartbeat`, apiserver's only idle-period liveness probe) never calls `xcecli_invoke_rejoin` — only `XCECLI_FAILOVER_CALL` (the full-retry path used by write calls) does. Left unaddressed, apiserver would repair its TCP connection via the heartbeat but never re-join, so push notifications (Req 2.4) would stay dead during any failover where no write request happens to arrive first. +- **Alternatives Considered**: (1) Switch the periodic prober from `xcecli_heartbeat` (fire-and-forget) to `xcecli_hello` (full retry, rejoin-bearing) — the same mechanism worker's `checking_connection()` already uses. (2) Keep `xcecli_heartbeat` (preserves its existing wire semantics/side effects on the controller) and have the Go-side tick callback explicitly detect a failure-to-success transition and manually invoke the same rejoin logic used elsewhere. +- **Selected Approach**: (2). +- **Rationale**: Preserves `xcecli_heartbeat`'s existing wire behavior (unknown/unverified whether the controller treats the heartbeat opcode differently from hello for apiserver accounting) while closing the gap with a small, self-contained Go-side check — no library changes, no behavior change to any other component. +- **Trade-offs**: A few extra lines in `heartbeat.go`; the rejoin logic now has two call sites (the cgo shim and the heartbeat callback), both invoking the same `reconnect.go` function, so no duplicated join-payload logic. +- **Follow-up**: none; resolved by construction in design.md. + +### Decision: Do not touch `xce_connect_cluster()`/`xce_reconnect_*` +- **Context**: This second, unused, stubbed primitive exists in the same area of the codebase. +- **Selected Approach**: Leave as-is; do not adopt, complete, or remove it in this spec. +- **Rationale**: Out of this feature's boundary — removing dead code is a separate hardening decision, not required to satisfy any requirement here, and touching it risks scope creep into an unrelated cleanup. +- **Follow-up**: Worth a future hardening backlog note (superseded/dead code) — not filed as part of this spec. + +## Risks & Mitigations +- Deleting worker's/scheduler's hand-rolled candidate loops could regress subtle existing behavior (e.g., worker's specific WARN logging on standby redirect) — mitigate by preserving equivalent log statements around the new single-call path, verified in code review. +- apiserver's circuit breaker could mask a fixed backend as still-degraded if the clearing signal (rejoin callback) doesn't fire in some edge case — mitigate with a bounded TTL on the degraded flag (re-probe after a short timeout) rather than relying solely on the rejoin callback. +- `N_MAX_MASTERS` (3) vs `MAX_HA_CANDIDATES` (4) mismatch could silently drop a 4th configured candidate on any component — mitigate by making the shared helper's WARN-and-clamp behavior loud and identical everywhere (already the plan). + +## References +- `src/lib/client.c`, `src/lib/client_internal.h`, `src/include/xce/client_failover.h` — the shipped `libxce-failover` mechanism this feature adopts. +- `src/controller/testsuite/test.c:345-382` — the reference (`xce_test`) usage pattern. +- `.kiro/steering/roadmap.md:64` — `component-failover-adoption` backlog item this spec formalizes. +- `.kiro/steering/ha-async-io.md` — thread-role contract informing Requirement 3. diff --git a/.kiro/specs/component-failover-adoption/spec.json b/.kiro/specs/component-failover-adoption/spec.json new file mode 100644 index 00000000..7d25c76a --- /dev/null +++ b/.kiro/specs/component-failover-adoption/spec.json @@ -0,0 +1,22 @@ +{ + "feature_name": "component-failover-adoption", + "created_at": "2026-07-20T04:23:58Z", + "updated_at": "2026-07-20T05:20:00Z", + "language": "en", + "phase": "tasks-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": true + }, + "design": { + "generated": true, + "approved": true + }, + "tasks": { + "generated": true, + "approved": false + } + }, + "ready_for_implementation": false +} diff --git a/.kiro/specs/component-failover-adoption/tasks.md b/.kiro/specs/component-failover-adoption/tasks.md new file mode 100644 index 00000000..6c1b3a2c --- /dev/null +++ b/.kiro/specs/component-failover-adoption/tasks.md @@ -0,0 +1,126 @@ +# Implementation Plan + +- [ ] 1. Foundation: shared candidate-list conversion helper +- [ ] 1.1 Implement the candidate-list-to-client-config conversion helper in libxce + - Add `xce_cluster_config_to_client_config()` to `src/lib/cluster.c`, declared in `src/include/xce/cluster.h`, generalizing the clamp-to-`N_MAX_MASTERS`-with-WARN logic `xce_test` already applies inline + - Copies candidates in order; leaves `from`/`key` untouched for the caller to set + - Observable completion: given a validated `xce_cluster_config_t` with 1, 3, and 4 candidates, the function returns a correctly populated `xce_client_config_t` in each case, truncating and logging a WARN only for the 4-candidate case + - _Requirements: 1.5_ + +- [ ] 1.2 Unit test the conversion helper + - Cover: candidate order preserved, truncation at `N_MAX_MASTERS` with WARN when given more, `NULL`/empty-candidate-list rejection with `-EINVAL` + - Add cases to `src/lib/testsuite/test_cluster.c` + - Observable completion: new test cases pass under the existing libxce unit test run + - _Requirements: 1.5_ + +- [ ] 2. Worker: adopt full-candidate-list connect and transparent reconnect +- [ ] 2.1 (P) Replace worker's per-candidate join loop with a single library-driven connect/join + - In `src/worker/src/controller_connector.c`: delete `xcecd_join_cluster`'s manual per-candidate iteration and `init_client_config`'s hardcoded `nmasters=1`; build the client config via the Task 1 helper from `cc_ctx->cluster_cfg` (or the existing single-host fallback when no `--peers` given) and call `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once + - Register a rejoin callback via `xcecli_set_reconnect_handler` (declared in `src/worker/include/controller_connector.h`) that ONLY sets a `needs_rejoin` flag and returns `0` — it must NOT call `xcecli_join` or any other `xcecli_*` function itself, because the library invokes it while `ac->failover.lock` is still held (a non-recursive mutex) and re-entering that lock via a wrapped call self-deadlocks + - In `checking_connection()`, immediately after each `xcecli_hello` call returns (lock now released), check and clear `needs_rejoin`; if set, call `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate, subsequent call + - Preserve existing WARN/INFO log statements for standby-redirect and per-candidate outcomes around the new single-call path + - Reconcile `checking_connection()`'s bounded `MAX_RETRY_COUNT`/`RETRY_INTERVAL`-then-`handle_disconnect()` loop with the library's own indefinite retry: the library now owns connection recovery (Requirement 2.3 — keep retrying, never permanently give up), so `checking_connection()`'s outer bound must no longer tear the worker down after 5 failed hello probes; it becomes a liveness-probe cadence only + - Observable completion: starting the worker against a 3-candidate config where candidate[0] is a live standby results in the worker joined as expected, with log lines showing the standby redirect and the successful candidate; a simulated outage longer than the old 5-retry window no longer triggers `handle_disconnect()`; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking + - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2_ + - _Boundary: WorkerControllerConnector_ + - _Depends: 1.1_ + +- [ ] 2.2 Fix worker's candidate-list startup ordering + - In `src/worker/src/main.c`: move the `cc_ctx->cluster_cfg = cluster_cfg` assignment to before `xcecd_start()` is called, so the very first connection attempt (inside `xcecd_start`/`controller_connector_start`) already sees the full `--peers` candidate list instead of only the single-host fallback + - Observable completion: starting the worker with `--peers` where the FIRST listed candidate is unreachable succeeds on the first connect attempt (no restart needed), verified by a log trace showing candidate rotation during the initial join, not only during a later reconnect + - _Requirements: 1.4_ + +- [ ] 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage + - Confirm (via code read) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread that now carries the library's blocking retry/backoff — fix isolation if this is not already true + - Add `src/worker/testsuite/test_e2e_multimaster_join.c`: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect + successful re-join to a surviving candidate without a worker restart + - Observable completion: the new test passes, and worker's hardware/process monitoring continues updating (observable via existing monitor state) while a connect/reconnect retry is in progress + - _Requirements: 1.6, 3.1, 3.2_ + - _Depends: 2.1, 2.2_ + +- [ ] 3. Scheduler: adopt full-candidate-list connect and add runtime reconnect +- [ ] 3.1 (P) Replace scheduler's inline per-candidate loop with a single library-driven connect/join + - In `src/scheduler/src/scheduler.c`: delete `sched_init`'s inline per-candidate loop; build the client config via the Task 1 helper from `sched->cluster_cfg` (or the existing single-host fallback) and call `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once + - Register a rejoin callback via `xcecli_set_reconnect_handler` that ONLY sets a `needs_rejoin` flag and returns `0` — same constraint as worker's Task 2.1: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself or it self-deadlocks + - After each `xcecli_jobdispatch` call returns (lock now released), check and clear `needs_rejoin`; if set, call `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate, subsequent call before continuing + - Preserve/add per-candidate outcome logging equivalent to what the deleted loop produced + - Observable completion: starting the scheduler against a 3-candidate config where candidate[0] is a live standby results in the scheduler joined as expected, with log lines showing the standby redirect and the successful candidate — and killing the joined primary afterward results in automatic reconnect and re-join to a surviving candidate, which the scheduler has never done before this task; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking + - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2_ + - _Boundary: SchedulerControllerClient_ + - _Depends: 1.1_ + +- [ ] 3.2 Add scheduler candidate/reconnect test coverage + - Add `src/scheduler/testsuite/test_e2e_multimaster_join.c`: scheduler starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join without a scheduler restart + - Confirm `sched_job_picker` continues running on its own thread, unaffected, while `sched_allocator`'s `xcecli_jobdispatch` call is blocked in reconnect/backoff (existing thread split, not new code) + - Observable completion: the new test passes, and the job-picker thread's activity (e.g. its own state/counters) continues advancing during a simulated reconnect delay on the allocator thread + - _Requirements: 3.1, 3.2_ + - _Depends: 3.1_ + +- [ ] 4. apiserver: adopt full-candidate-list connect via CGO, config parity, rejoin, and fail-fast +- [ ] 4.1 (P) Route apiserver's candidate-list parsing through the real C parser + - In `src/frontend/apiserver/backend/clib.go`: call `xce_parse_peers`/`xce_cluster_config_validate` via CGO to parse `config.Peers` into a `xce_cluster_config_t`, replacing the hand-written Go `parsePeer` function and its call sites + - Map C validation errors (`-EINVAL` cases) to clear Go error messages naming the malformed entry + - Observable completion: the same malformed-candidate-string test cases that scheduler/worker already reject (from Task 1.2's neighboring validation coverage) are rejected identically when passed through apiserver's config path + - _Requirements: 6.1, 6.2_ + - _Boundary: ApiserverBackendBridge_ + - _Depends: 1.1_ + +- [ ] 4.2 Replace apiserver's per-peer connect loop with a single library-driven connect + - In `src/frontend/apiserver/backend/clib.go`: replace `RegisterToBackend`'s `for _, peer := range config.Peers { tryConnect(...) }` loop with one call that builds the client config via the Task 1 helper (through CGO) from the full parsed candidate list (or the single `Address`/`Port` fallback) and calls `xce_backend_init` once + - Preserve/extend existing connect-outcome logging (`zap`) to report each candidate attempted + - Observable completion: starting apiserver with 3 configured peers where the first is a live standby results in apiserver registered against the actual primary in a single `RegisterToBackend` call, with log lines showing the standby redirect + - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 5.1, 5.2_ + - _Depends: 4.1_ + +- [ ] 4.3 Add apiserver's rejoin logic with a correctly-placed cgo registration + - New `src/frontend/apiserver/backend/reconnect.go`: two plain Go functions — (1) a `//export`-annotated flag-setter that ONLY sets a `needs_rejoin` atomic flag and returns `0` (no I/O — the library invokes it while `ac->failover.lock` is held, so calling `xcecli_join` directly here self-deadlocks); (2) a plain function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears backend-health degraded state on success + - In `src/frontend/apiserver/backend/clib.go`'s cgo preamble: add a static C shim (mirroring the existing `xce_backend_handler`/`go_push_handler` pattern) with its own `extern` declaration for the flag-setter export, and pass the shim to `xcecli_set_reconnect_handler` — the registration must live in `clib.go`, not in `reconnect.go`, because cgo symbol resolution is per-file (same reason `go_push_handler` needs `xce_backend_handler`, documented in `wire_dispatch_support.go`) + - Add one shared internal helper in `clib.go` that wraps a `C.xcecli_*` call: after the call returns (lock now released), check and clear `needs_rejoin`; if set, call `reconnect.go`'s actual-rejoin function as a separate step. Route all four backend-call wrappers (job submit, job control, queue control, host control) through this ONE helper instead of duplicating the check in each (Task 4.5 extends this same helper with the pre-call health check, rather than adding a second one) + - Observable completion: after killing and restoring connectivity to apiserver's joined controller while a write request is in flight (job submit retried through the full failover path), apiserver resumes receiving backend push notifications (existing `go_push_handler` path) without an apiserver process restart; invoking the registered flag-setter while the ctx's failover lock is held returns immediately without blocking + - _Requirements: 2.1, 2.2, 2.3, 2.4_ + - _Depends: 4.2_ + +- [ ] 4.4 Trigger apiserver's rejoin from the idle-period heartbeat path + - In `src/frontend/apiserver/backend/heartbeat.go`'s tick callback: track the previous tick's `xcecli_heartbeat` outcome; when a tick transitions from failure to success (the fire-and-forget call's own best-effort transport reconnect just succeeded), explicitly call Task 4.3's actual-rejoin function (function (2), not the flag-setter) directly — safe here since `XCECLI_FF_CALL` has already released the lock by the time the heartbeat call returns. `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) never invokes the library's own rejoin hook, unlike the full-retry path write calls use, so without this the idle-period case never re-joins + - Observable completion: killing the primary while apiserver has no in-flight write request (only the heartbeat ticking) still results in apiserver rejoining and resuming push notifications, verified by the same push-notification check as Task 4.3 but with no write traffic present during the outage + - _Requirements: 2.4_ + - _Depends: 4.3_ + +- [ ] 4.5 Add apiserver's backend-health circuit breaker + - New `src/frontend/apiserver/backend/health.go`: atomic degraded-until state with `IsDegraded()`, `MarkDegraded(ttl)`, `MarkHealthy()` + - Extend Task 4.3's shared internal helper (not a second, separate hook) to also check `IsDegraded()` before the `C.xcecli_*` call and return a clear "backend unavailable" error immediately instead of invoking it when degraded; wire `MarkDegraded` into the same helper's post-call path when a call's result indicates retry-budget exhaustion; wire `MarkHealthy` into both Task 4.3's rejoin function (write-path recovery) and Task 4.4's heartbeat-triggered rejoin (idle-path recovery) + - Observable completion: with the flag forced degraded in a unit test, a wrapped backend call returns immediately with the unavailable error instead of invoking the C call + - _Requirements: 4.1, 4.2_ + - _Depends: 4.4_ + +- [ ] 4.6 apiserver unit test coverage + - `src/frontend/apiserver/backend/health_test.go`: degraded→healthy via `MarkHealthy()` and via TTL expiry; concurrent-goroutine safety + - Extend `src/frontend/apiserver/backend/clib_test.go` with candidate-list parity cases from Task 4.1 + - Observable completion: `make test` in `src/frontend/` passes with the new cases included + - _Requirements: 4.1, 4.2, 6.1, 6.2_ + - _Depends: 4.5_ + +- [ ] 5. Integration and cross-component validation +- [ ] 5.1 E2E: full-cluster reconnect after primary failover + - Extend the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern to a scenario with worker, scheduler, and apiserver all configured with the full 3-candidate list; kill the primary mid-session with a write request in flight + - Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted + - Observable completion: the soak scenario passes with all three components showing a successful re-join to the new primary in their logs and no manual intervention + - _Requirements: 2.1, 2.2, 2.3, 2.4_ + - _Depends: 2.3, 3.2, 4.6_ + +- [ ] 5.2 E2E: apiserver reconnects during an idle period (no in-flight write traffic) + - Kill the primary while apiserver has only its heartbeat prober active (no pending write requests); assert apiserver still rejoins and resumes push notifications via the Task 4.4 heartbeat-triggered path + - Observable completion: the scenario passes without requiring any client-triggered write request during the outage window + - _Requirements: 2.4_ + - _Depends: 4.6_ + +- [ ] 5.3 E2E: apiserver fails fast while backend is degraded + - With apiserver's backend connection killed and not yet reconnected, issue an API request that requires the backend and assert it returns a clear 503-style error within a bounded, sub-second time — not after the full retry budget + - Observable completion: the timed assertion passes against a live apiserver instance + - _Requirements: 4.1, 4.2_ + - _Depends: 4.6_ + +- [ ] 5.4 Regression: confirm existing single-controller mode is unaffected + - Run the full worker, scheduler, and apiserver test suites (ASAN where applicable) plus the existing non-HA (no `--peers`/no `backend.peers`) startup path for each component + - Observable completion: all existing tests pass unchanged, and each component still starts and operates normally when no candidate list is configured (Requirement 1.5's single/no-candidate fallback) + - _Requirements: 1.5_ + - _Depends: 5.1, 5.2, 5.3_ diff --git a/.kiro/steering/config-design.md b/.kiro/steering/config-design.md new file mode 100644 index 00000000..59d0b763 --- /dev/null +++ b/.kiro/steering/config-design.md @@ -0,0 +1,72 @@ +# Configuration Design: What Belongs in Shared Config + +Governs any XCE distributed-config schema decision — `ClusterConfig` (controller.capnp), and any future worker/scheduler/apiserver shared-config struct. Applies whenever a new field is proposed for a config blob that gets pushed/replicated to multiple nodes (e.g. via `OP_upgradeConfig` or an equivalent future mechanism). + +## The Question + +A parameter needs a home. Two choices: +- **Shared config** — part of the cluster-wide config blob, distributed to every node, changed via a single coordinated push (build → validate → replicate → majority ACK). +- **Local/instance config** — set per-node (CLI flag, local config file, env var), changed independently, no cluster-wide coordination. + +Getting this wrong in either direction has a cost: putting a per-instance value in shared config forces every routine node-level change through a cluster-wide config push; putting a cluster-invariant value in local config lets nodes silently drift out of sync. + +## Rule 0: Bootstrap availability (checked first, overrides everything else) + +> If a parameter is needed to establish consensus, elect a coordinator, or otherwise receive the shared config in the first place, it CANNOT live inside that shared config. + +This isn't a judgment call like rules 1–2 below — it's a structural/chicken-egg constraint. A node can't depend on config it doesn't have yet to obtain that same config. Example: an HA/Raft election heartbeat interval/timeout — a controller needs this value to participate in the election that determines who's primary and delivers the cluster config; it must be locally available (CLI flag, hardcoded default) before any config exists. This is independent of whether the field would otherwise pass or fail the independence/frequency rules below — even a field that's cluster-invariant and rarely-changing (which would normally mean "shared") is disqualified from shared config if it's needed to bootstrap access to that config. + +If a candidate field passes Rule 0 (i.e., it isn't a bootstrap dependency), proceed to the two rules below. + +## The Two Rules + +Apply both to every candidate field. **Excluding on either rule is sufficient** — a field only belongs in shared config if it fails both. Note: failing both rules means shared config is the *default* home, not automatic — see "Operational-uniformity override" below for when a field that would otherwise qualify for local config gets deliberately centralized anyway. + +### 1. Independence rule + +> If a parameter can change on one node without requiring every other node to match, it does NOT belong in shared config. + +Test: does divergence between nodes on this field cause a correctness problem, or is it just... fine? If fine, it's independent — keep it local. + +### 2. Frequency rule + +> If a parameter is expected to change more often than the rest of the config, it does NOT belong in shared config. + +Test: would this field's natural rate of change force config pushes far more often than everything else in the blob needs? If yes, bundling it in shared config makes routine operational changes as heavyweight as a schema change. + +### Default: both rules fail → include + +A field that must be identical across every node AND changes rarely belongs in shared config. That's the actual reason shared config exists — to keep genuinely cluster-invariant values from drifting. + +## Operational-uniformity override + +Passing the independence rule (no correctness requirement to match across nodes) means a field is *eligible* for local config — it does not mean it *must* go local. A field can still be deliberately kept in shared config when centralizing it has operational value beyond pure correctness: predictable behavior across the cluster, one less per-node setting for an operator to track, avoiding "why does this node behave differently" support questions. This is a judgment call, not automatic — when exercising it, document the reasoning inline (don't just leave a field in shared config by default without stating why it's an exception). + +Worked example: `ClusterConfig`'s `heartbeatInterval`/`inactiveHeartbeatCount` fields govern controller→**worker** liveness (when a controller marks one of its own workers unavailable) — not the HA/Raft controller-to-controller election heartbeat (see Rule 0 example above; those are two different "heartbeat" concepts sharing similar names, easy to conflate). Each controller only manages its own connected workers, so nothing breaks if this value differs node-to-node — it passes the independence rule cleanly. Decided (2026-07-20) to keep it in shared config anyway: operators want uniform, predictable worker-timeout behavior across the cluster rather than per-node drift, even though no correctness invariant demands it. + +## Worked Examples + +| Field | Rule 0 (bootstrap) | Independence | Frequency | Verdict | Why | +|---|---|---|---|---|---| +| Cluster name | passes (not a bootstrap dep) | fails — every node must agree | fails — set once at bootstrap | **Shared** | Identity mismatch across nodes is a correctness bug | +| HA/Raft election heartbeat interval/timeout | **fails Rule 0** — needed to run the election that delivers config in the first place | n/a — disqualified before reaching this rule | n/a | **Local** (CLI flag; currently hardcoded, see roadmap) | Can't depend on config you don't have yet to obtain that config | +| Controller→worker liveness heartbeat/timeout | passes (not a bootstrap dep) | passes — one controller's worker-timeout tolerance has no cross-node dependency | — | **Shared** (operational-uniformity override, not a rule outcome — see above) | Would qualify for local by the rules alone; kept shared for predictable cluster-wide behavior | +| Listen port | passes | passes — each node's own bind port has no cross-node dependency | — | **Local** | Purely a per-process network setting | +| Peer/candidate list (cluster membership) | passes | passes — adding a standby doesn't require the rest of the config to change | passes — routine operational action | **Local** (own mechanism) | Coupling membership changes to a full config push makes scaling the cluster as heavyweight as a schema migration | +| Internal hash-table sizes, buffer sizes (perf tuning) | passes | passes — effect is local to the process that sets them | — | **Local** | No node needs to know another node's internal tuning | +| Any monotonic version/generation counter | n/a — not operator input | n/a | n/a | **Not config at all** | Protocol state the coordinator (primary) assigns, not a value an operator sets | + +The last row is a distinct fourth category, not a rule outcome: some fields aren't "config" in the operator-input sense — they're state the system itself maintains (e.g. a config generation counter incremented by the primary on every push). Don't force these into the shared-config-vs-local-config framing; they're neither. + +## Applying This + +When a new field is proposed for a shared config blob: +1. Check Rule 0 first — if the field is needed to bootstrap/obtain the config itself, it's local, full stop, regardless of what rules 1–2 would say. +2. Otherwise, run it through the independence rule, then the frequency rule. +3. If it fails either, its default home is local (CLI flag, per-node config file, a dedicated dynamic-membership mechanism, etc.) — unless the operational-uniformity override applies and is explicitly documented as a deliberate exception. +4. If it passes both (fails neither exclusion), add it to shared config — and document why in the owning spec, so the next person doesn't have to re-derive it. +5. If it's coordinator-assigned protocol state rather than operator input, keep it out of the operator-facing config schema entirely. + +## Origin + +Derived 2026-07-20 while classifying `ClusterConfig`'s 21 fields for `xceadmin config build` (`.kiro/specs/admin-config-tool/`) — 6 fields were being pulled into shared YAML config by default with no principled reason to include vs. exclude them. See `brief.md`'s "Configuration Inclusion Rules" section in that spec for the full field-by-field classification of `ClusterConfig` specifically; this file holds the general rule for reuse beyond that one spec. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index e50e8d11..25201541 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -88,7 +88,8 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o - [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump `** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare `** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_ - [ ] xce_test shell command support — add `!` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log. - [→] scheduler framework enhancement — **PROMOTED to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/scheduler-framework/brief.md`). Original backlog seed superseded by the full brief with iterator-driven allocation architecture, configurable job picker, combinatorial parallel job allocation, and session-level unsatisfiable resreq cache. Scope narrowed from the original seed: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). Next: `/kiro-spec-requirements scheduler-framework`. -- [ ] **perf-tuning-extraction** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). Audit `ClusterConfig` fields in `controller.capnp` and identify parameters used only for controller performance tuning (e.g., heartbeat intervals, timeout values, pool sizes, WAL segment sizing). Extract them from the cluster config and create a separate integration plan document for future performance tuning (how to expose, configure, validate, and apply perf tunables without requiring a full cluster config upgrade). This is direct-implementation work (documentation + field audit), not a spec. +- [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass. +- [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`. ## Superseded / Absorbed Items diff --git a/.wolf/anatomy.md b/.wolf/anatomy.md index a65a6646..33194ea5 100644 --- a/.wolf/anatomy.md +++ b/.wolf/anatomy.md @@ -1,7 +1,7 @@ # anatomy.md -> Auto-maintained by OpenWolf. Last scanned: 2026-07-19T08:21:33.657Z -> Files: 684 tracked | Anatomy hits: 0 | Misses: 0 +> Auto-maintained by OpenWolf. Last scanned: 2026-07-21T08:07:27.029Z +> Files: 731 tracked | Anatomy hits: 0 | Misses: 0 ## ../../../../../../tmp/claude-1000/ @@ -43,6 +43,10 @@ - `run_full_suite.sh` — Full controller suite, suite-by-suite under ASAN (dodges the 300s aggregate cap). (~450 tok) +## ../../../../../claude/.claude/plans/ + +- `ancient-beaming-kahn.md` — Bug-1454: Scheduler --peers Minor Issues (Narrowed) (~913 tok) + ## ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/ - `MEMORY.md` — Memory Index (~1906 tok) @@ -325,10 +329,26 @@ ## .kiro/specs/admin-config-tool/ -- `brief.md` — Brief: admin-config-tool (~9037 tok) -- `requirements.md` — Requirements Document (~3532 tok) +- `brief.md` — Brief: admin-config-tool (~10183 tok) +- `requirements.md` — Requirements Document (~4705 tok) - `spec.json` (~138 tok) +## .kiro/specs/clusterconfig-schema-split/ + +- `design.md` — Overview (~7008 tok) +- `requirements.md` — Requirements Document (~4621 tok) +- `research.md` — Research & Design Decisions (~4955 tok) +- `spec.json` (~133 tok) +- `tasks.md` — Implementation Plan (~2668 tok) + +## .kiro/specs/component-failover-adoption/ + +- `design.md` — Design Document: component-failover-adoption (~10302 tok) +- `requirements.md` — Requirements Document (~3325 tok) +- `research.md` — Research & Design Decisions (~5526 tok) +- `spec.json` (~132 tok) +- `tasks.md` — Implementation Plan (~3873 tok) + ## .kiro/specs/config-upgrade-snapshot/ - `brief.md` — Brief: config-upgrade-snapshot (~923 tok) @@ -352,6 +372,11 @@ - `spec.json` (~129 tok) - `tasks.md` — Implementation Plan (~3693 tok) +## .kiro/specs/ha-client-multimaster/ + +- `requirements.md` — Requirements Document (~621 tok) +- `spec.json` (~130 tok) + ## .kiro/specs/ha-commit-barrier-rollout/ - `brief.md` — Brief: ha-commit-barrier-rollout (~626 tok) @@ -627,11 +652,12 @@ - `api-standards.md` — API Standards (~1291 tok) - `build-deployment.md` — Build & Deployment (~1451 tok) +- `config-design.md` — Configuration Design: What Belongs in Shared Config (~2053 tok) - `error-handling.md` — Error Handling Standards (~1026 tok) - `ha-async-io.md` — HA Protocol & Async I/O (~4180 tok) - `persistence.md` — WAL & Persistence Durability (~2600 tok) - `product.md` — Product Overview (~1429 tok) -- `roadmap.md` — Roadmap (~11258 tok) +- `roadmap.md` — Roadmap (~11743 tok) - `security.md` — Security Standards (~804 tok) - `structure.md` — Project Structure (~1170 tok) - `tech.md` — Technology Stack (~1155 tok) @@ -969,18 +995,19 @@ - `apiserver.c` — include (~5962 tok) - `CLAUDE.md` — XCE Controller (~3491 tok) -- `cluster.c` — include "config.h" (~18669 tok) +- `cluster.c` — include "config.h" (~19487 tok) - `commit.c` — Declares is (~12825 tok) -- `config_repl.c` — define _GNU_SOURCE (~32213 tok) +- `config_repl.c` — define _GNU_SOURCE (~32202 tok) - `config_repl.h` — pragma once (~6155 tok) - `ha_convergence.c` — Declares char (~7830 tok) - `ha.c` — define _GNU_SOURCE (~111605 tok) - `ha.h` — ifndef HA_H; +demotion_convergence_inflight field, ha_set_demotion_offload_for_test/ha_set_catchup_offload_for_test decls (2026-07-18) (~31200 tok) - `job.c` — include (~24947 tok) - `join.c` — include "config.h" (~2795 tok) -- `main.c` — include (~3035 tok) +- `main.c` — include (~4682 tok) - `meson.build` (~2391 tok) - `myapp.c` — define _GNU_SOURCE (~5215 tok) +- `myconf.c` — define _GNU_SOURCE (~1088 tok) - `persist.c` — define _GNU_SOURCE (~72815 tok) - `sched.c` — include (~6155 tok) @@ -1004,10 +1031,10 @@ - `test_apply_seam.c` (~2936 tok) - `test_apply_seam.c` — raft-apply-decoupling task 2.1: calls apply_job_start/finish/fail/update_transition() directly on a bare jobmgr+job_t (no cluster_t), bypassing on_job_start/finish/fail, to prove the extracted guard cores work outside the receipt path (~2900 tok) - `test_commit_barrier_reply.c` — Declares uint8_t (~41011 tok) -- `test_commit_barrier_rollout.c` — Declares explicitly (~26498 tok) +- `test_commit_barrier_rollout.c` — Declares explicitly (~26488 tok) - `test_commit_durability.c` — define _GNU_SOURCE (~32879 tok) - `test_commit_resolve.c` (~14018 tok) -- `test_config_repl.c` — define _GNU_SOURCE; +test_catchup_offload_probe_runs_promptly / test_catchup_inline_probe_blocks_RED (2026-07-18, bug-1451 RED/GREEN via real single-worker delaymgr + probe task) (~48500 tok) +- `test_config_repl.c` — define _GNU_SOURCE (~45079 tok) - `test_convergence_verification.c` — define _GNU_SOURCE; pre-existing ha_walack_on_ack/ha_catchup_continuation_fn fixture, NO slowlane_init — the surface that broke when catchup offload defaulted ON (see bug-1451) (~18715 tok) - `test_demotion_convergence.c` — define _GNU_SOURCE; dc_setup +slowlane_init, +test_demotion_offload_task_returns_promptly / test_demotion_inline_task_blocks_RED (2026-07-18, bug-1450 RED/GREEN) (~18600 tok) - `test_e2e_capnp.c` — include (~3729 tok) @@ -1054,23 +1081,89 @@ - `wire_dispatch_support.go` — go:build cgo (~2254 tok) - `wire_dispatch_test.go` — go:build cgo (~1337 tok) +## src/frontend/xcecli/cmd/auth/ + +- `logon.go` — NewLogonCmd (~1388 tok) + +## src/frontend/xcecli/cmd/clusters/ + +- `list.go` — NewClustersCmd (~505 tok) + +## src/frontend/xcecli/cmd/hosts/ + +- `controller.go` — Struct: HostController (~1272 tok) +- `list.go` — Struct: HostsQuerier (~1502 tok) +- `select.go` — Struct: HostSelector (~595 tok) + +## src/frontend/xcecli/cmd/jobs/ + +- `controller.go` — Struct: JobController (~2393 tok) +- `list.go` — Struct: JobsQuerier (~3544 tok) +- `restart.go` — Struct: JobRestart (~501 tok) +- `submit.go` — Struct: JobSubmitter (~6452 tok) + +## src/frontend/xcecli/cmd/permissions/ + +- `list.go` — Struct: PermsQuerier (~874 tok) + +## src/frontend/xcecli/cmd/queues/ + +- `controller.go` — Struct: QueueController (~1787 tok) +- `list.go` — Struct: QueuesQuerier (~1449 tok) + +## src/frontend/xcecli/cmd/resources/ + +- `list.go` — Struct: ResourcesDistQuerier (~2638 tok) + +## src/frontend/xcecli/cmd/roles/ + +- `addrole.go` — Struct: RoleSubmitter (~2124 tok) +- `deleterole.go` — Struct: DeleteRole (~583 tok) +- `list.go` — Struct: RolesQuerier (~1633 tok) + +## src/frontend/xcecli/cmd/users/ + +- `assignrole.go` — Struct: UserSubmitter (~862 tok) +- `list.go` — Struct: UserRolesQuerier (~1919 tok) +- `perm4user.go` — Struct: Perms4User (~1056 tok) +- `unassignrole.go` — Struct: UnassignUser (~711 tok) + +## src/frontend/xcecli/common/request/ + +- `request_test.go` — TestAuthHeaderConsistency (~112 tok) +- `request.go` — Struct: APIResponse (~2465 tok) + +## src/frontend/xcecli/config/ + +- `config.go` — Struct: APIServerInfo (~1488 tok) + +## src/frontend/xcecli/models/ + +- `logon.go` — Struct: LogonData (~147 tok) + ## src/include/ - `client.h` — if !defined(_CLIENT_H_) (~1707 tok) +- `clusterconfig.h` — if !defined(_CLUSTERCONFIG_H_) (~248 tok) - `config.h` — if !defined(_CONFIG_H_) (~1257 tok) - `delay.h` — if !defined(_DELAY_H_) (~414 tok) - `kvmgr.h` — if !defined(__KVMGR_H__) (~631 tok) -- `myconf.h` — if !defined(_MYCONF_H_) (~560 tok) +- `myconf.h` — if !defined(_MYCONF_H_) (~874 tok) - `netio.h` — if !defined(_NETIO_H_) (~2608 tok) - `persist.h` — pragma once (~24030 tok) - `typemap.h` — if !defined(_TYPEMAP_H_) (~5207 tok) - `wal_types.h` — pragma once (~2076 tok) - `zarray.h` — if !defined(_ZARRAY_H_) (~1286 tok) +## src/include/xce/ + +- `cluster.h` — A single controller candidate. (~1107 tok) + ## src/lib/ - `client_internal.h` — if !defined(_CLIENT_INTERNAL_H_) (~4475 tok) - `client.c` — include (~6767 tok) +- `cluster.c` — include (~1526 tok) - `delay.c` — include (~1520 tok) - `jobj.c` — include (~6090 tok) - `kvmgr.c` — include (~3053 tok) @@ -1088,6 +1181,7 @@ ## src/lib/testsuite/ +- `test_cluster.c` — include (~1725 tok) - `test_ctrlstatus.c` (~4821 tok) - `test_jobj_hardening.c` — include (~4939 tok) - `test_kvmgr_reserve.c` — Declares instance (~1260 tok) @@ -1099,9 +1193,15 @@ - `test_zarray_reserve.c` — Declares instance (~2341 tok) - `test.c` — include (~1210 tok) +## src/scheduler/include/ + +- `scheduler.h` — if !defined(_SCHEDULER_H_) (~1863 tok) + ## src/scheduler/src/ - `handler.c` — include (~4838 tok) +- `main.c` — include (~1546 tok) +- `scheduler.c` — include (~10738 tok) ## src/scheduler/testsuite/ @@ -1112,9 +1212,13 @@ ## src/schema/ -- `controller.capnp` — Declares OPCode (~9735 tok) +- `controller.capnp` — Declares OPCode (~9440 tok) ## src/tools/ - `test-commit-durability.sh` — test-commit-durability.sh — bug-661 reproduction: committed jobs lost across failover. (~9879 tok) - `xce-build.sh` (~1400 tok) + +## src/worker/src/ + +- `main.c` — include "sqlite3.h" (~5454 tok) diff --git a/.wolf/hooks/_session.json b/.wolf/hooks/_session.json index 75042b0f..2bfa225c 100644 --- a/.wolf/hooks/_session.json +++ b/.wolf/hooks/_session.json @@ -1,49 +1,12 @@ { - "session_id": "session-2026-07-19-0115", - "started": "2026-07-19T08:15:02.639Z", - "files_read": { - "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/commit.c": { - "count": 1, - "tokens": 12825, - "first_read": "2026-07-19T08:15:37.199Z" - }, - "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/include/commit.h": { - "count": 1, - "tokens": 3635, - "first_read": "2026-07-19T08:18:10.332Z" - }, - "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_persist_truncate.c": { - "count": 2, - "tokens": 5031, - "first_read": "2026-07-19T08:21:11.795Z" - } - }, - "files_written": [ - { - "file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_persist_truncate.c", - "action": "edit", - "tokens": 238, - "at": "2026-07-19T08:21:29.974Z" - }, - { - "file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_persist_truncate.c", - "action": "edit", - "tokens": 90, - "at": "2026-07-19T08:21:31.862Z" - }, - { - "file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_persist_truncate.c", - "action": "edit", - "tokens": 96, - "at": "2026-07-19T08:21:33.668Z" - } - ], - "edit_counts": { - "src/controller/testsuite/test_persist_truncate.c": 3 - }, - "anatomy_hits": 3, + "session_id": "session-2026-07-21-0123", + "started": "2026-07-21T08:23:18.398Z", + "files_read": {}, + "files_written": [], + "edit_counts": {}, + "anatomy_hits": 0, "anatomy_misses": 0, - "repeated_reads_warned": 1, + "repeated_reads_warned": 0, "cerebrum_warnings": 0, - "stop_count": 4 + "stop_count": 0 } \ No newline at end of file diff --git a/.wolf/memory.md b/.wolf/memory.md index 733ac275..3f5729ca 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -5211,3 +5211,349 @@ | 01:29 | bug-1452 root-caused+fixed: teardown order (commit_engine_fini before persist_stop) raced fsync_worker vs ce teardown, NULL commit.c:208; fix=persist_stop before commit_engine_fini x3 tests; verified 976/976 x6 runs clean; buglog.json updated | test_persist_truncate.c, buglog.json | fixed+verified | ~2100 | | 01:30 | Session end: 3 writes across 1 files (test_persist_truncate.c) | 3 reads | ~21915 tok | | 01:30 | Session end: 3 writes across 1 files (test_persist_truncate.c) | 3 reads | ~21915 tok | +| 02:28 | Session end: 3 writes across 1 files (test_persist_truncate.c) | 3 reads | ~21915 tok | + +## Session: 2026-07-20 21:14 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 21:24 | Created .kiro/specs/ha-client-multimaster/spec.json | — | ~130 | +| 21:24 | Created .kiro/specs/ha-client-multimaster/requirements.md | — | ~662 | +| 21:24 | Session end: 2 writes across 2 files (spec.json, requirements.md) | 4 reads | ~31299 tok | +| 21:27 | Edited .kiro/specs/component-failover-adoption/spec.json | inline fix | ~14 | +| 21:27 | Edited .kiro/specs/component-failover-adoption/requirements.md | expanded (+18 lines) | ~1066 | +| 21:28 | Edited .kiro/specs/component-failover-adoption/requirements.md | expanded (+53 lines) | ~1967 | +| 21:28 | Edited .kiro/specs/component-failover-adoption/requirements.md | inline fix | ~62 | +| 21:29 | Edited .kiro/specs/component-failover-adoption/spec.json | 8→8 lines | ~55 | +| 21:29 | Session end: 7 writes across 2 files (spec.json, requirements.md) | 10 reads | ~52837 tok | +| 21:33 | Edited .kiro/specs/component-failover-adoption/spec.json | 4→4 lines | ~22 | +| 21:38 | Created .kiro/specs/component-failover-adoption/research.md | — | ~4374 | +| 21:41 | Created .kiro/specs/component-failover-adoption/design.md | — | ~8513 | +| 21:41 | Edited .kiro/specs/component-failover-adoption/spec.json | 12→12 lines | ~74 | +| 21:41 | Session end: 11 writes across 4 files (spec.json, requirements.md, research.md, design.md) | 14 reads | ~80861 tok | +| 21:43 | Edited .kiro/specs/component-failover-adoption/spec.json | 4→4 lines | ~20 | +| 21:45 | Created .kiro/specs/component-failover-adoption/tasks.md | — | ~2907 | +| 21:46 | Edited .kiro/specs/component-failover-adoption/tasks.md | 3→4 lines | ~279 | +| 21:46 | Edited .kiro/specs/component-failover-adoption/spec.json | 17→17 lines | ~94 | +| 21:46 | Session end: 15 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 14 reads | ~84386 tok | +| 21:51 | Session end: 15 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 15 reads | ~86640 tok | +| 23:34 | Edited .kiro/specs/component-failover-adoption/design.md | expanded (+10 lines) | ~538 | +| 23:35 | Edited .kiro/specs/component-failover-adoption/design.md | modified itself() | ~654 | +| 23:35 | Edited .kiro/specs/component-failover-adoption/design.md | 3→4 lines | ~149 | +| 23:35 | Edited .kiro/specs/component-failover-adoption/design.md | expanded (+24 lines) | ~335 | +| 23:35 | Edited .kiro/specs/component-failover-adoption/design.md | "xcecli_join(HostRole_apis" → "reconnect.go" | ~95 | +| 23:35 | Edited .kiro/specs/component-failover-adoption/design.md | 2→3 lines | ~247 | +| 23:36 | Edited .kiro/specs/component-failover-adoption/research.md | modified registered() | ~792 | +| 23:37 | Edited .kiro/specs/component-failover-adoption/tasks.md | modified that() | ~912 | +| 23:37 | Edited .kiro/specs/component-failover-adoption/tasks.md | expanded (+6 lines) | ~455 | +| 23:37 | Edited .kiro/specs/component-failover-adoption/tasks.md | 5→5 lines | ~136 | +| 23:40 | Edited .kiro/specs/component-failover-adoption/design.md | expanded (+9 lines) | ~580 | +| 23:40 | Edited .kiro/specs/component-failover-adoption/design.md | 23→27 lines | ~404 | +| 23:41 | Edited .kiro/specs/component-failover-adoption/design.md | 2→2 lines | ~233 | +| 23:41 | Edited .kiro/specs/component-failover-adoption/design.md | 3→3 lines | ~263 | +| 23:42 | Edited .kiro/specs/component-failover-adoption/design.md | logic() → returns() | ~378 | +| 23:43 | Edited .kiro/specs/component-failover-adoption/design.md | modified callbacks() | ~406 | +| 23:43 | Edited .kiro/specs/component-failover-adoption/research.md | expanded (+8 lines) | ~778 | +| 23:43 | Edited .kiro/specs/component-failover-adoption/tasks.md | modified itself() | ~524 | +| 23:44 | Edited .kiro/specs/component-failover-adoption/tasks.md | 4→5 lines | ~362 | +| 23:44 | Edited .kiro/specs/component-failover-adoption/tasks.md | modified that() | ~477 | +| 23:44 | Edited .kiro/specs/component-failover-adoption/tasks.md | inline fix | ~171 | +| 23:45 | Edited .kiro/specs/component-failover-adoption/design.md | "clib.go" → "IsDegraded()" | ~104 | +| 23:46 | Edited .kiro/specs/component-failover-adoption/design.md | inline fix | ~147 | +| 23:46 | Edited .kiro/specs/component-failover-adoption/tasks.md | inline fix | ~130 | +| 23:46 | Edited .kiro/specs/component-failover-adoption/tasks.md | 7→6 lines | ~241 | +| 23:47 | Edited .kiro/specs/component-failover-adoption/design.md | inline fix | ~105 | +| 23:47 | Edited .kiro/specs/component-failover-adoption/spec.json | inline fix | ~12 | +| 23:48 | Session end: 42 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 19 reads | ~115172 tok | + +## Session: 2026-07-20 23:54 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 23:59 | Edited .kiro/specs/admin-config-tool/requirements.md | 2→4 lines | ~257 | +| 23:59 | Edited .kiro/specs/admin-config-tool/requirements.md | 4→4 lines | ~172 | +| 23:59 | Edited .kiro/specs/admin-config-tool/brief.md | "ha/data/xce.crt" → "xce.crt" | ~91 | +| 23:59 | Session end: 3 writes across 2 files (requirements.md, brief.md) | 3 reads | ~13381 tok | +| 00:06 | Edited .kiro/specs/admin-config-tool/requirements.md | 16→18 lines | ~587 | +| 00:06 | Edited .kiro/specs/admin-config-tool/requirements.md | 3→5 lines | ~210 | +| 00:07 | Edited .kiro/specs/admin-config-tool/brief.md | inline fix | ~139 | +| 00:07 | Edited .kiro/specs/admin-config-tool/brief.md | 2→2 lines | ~175 | +| 00:07 | Edited .kiro/specs/admin-config-tool/brief.md | inline fix | ~56 | +| 00:08 | Edited .kiro/specs/admin-config-tool/brief.md | 2→2 lines | ~352 | +| 00:08 | Edited .kiro/specs/admin-config-tool/brief.md | inline fix | ~116 | +| 00:09 | Edited .kiro/specs/admin-config-tool/spec.json | inline fix | ~12 | +| 00:09 | Session end: 11 writes across 3 files (requirements.md, brief.md, spec.json) | 3 reads | ~15538 tok | +| 00:16 | Edited .kiro/specs/admin-config-tool/requirements.md | 2→3 lines | ~232 | +| 00:16 | Edited .kiro/specs/admin-config-tool/requirements.md | 1→3 lines | ~268 | +| 00:17 | Edited .kiro/specs/admin-config-tool/requirements.md | inline fix | ~121 | +| 00:17 | Edited .kiro/specs/admin-config-tool/brief.md | modified Rules() | ~555 | +| 00:18 | Edited .kiro/specs/admin-config-tool/brief.md | expanded (+23 lines) | ~650 | +| 00:18 | Edited .kiro/specs/admin-config-tool/requirements.md | 1→3 lines | ~182 | +| 00:18 | Edited .kiro/specs/admin-config-tool/spec.json | inline fix | ~12 | +| 00:19 | Session end: 18 writes across 3 files (requirements.md, brief.md, spec.json) | 3 reads | ~18252 tok | +| 00:22 | Session end: 18 writes across 3 files (requirements.md, brief.md, spec.json) | 3 reads | ~18252 tok | +| 00:24 | Created .kiro/steering/config-design.md | — | ~1264 | +| 00:24 | Edited .kiro/specs/admin-config-tool/brief.md | 3→5 lines | ~148 | +| 00:29 | Created .kiro/specs/clusterconfig-schema-split/spec.json | — | ~132 | +| 00:29 | Created .kiro/specs/clusterconfig-schema-split/requirements.md | — | ~912 | +| 00:29 | Edited .kiro/specs/admin-config-tool/requirements.md | "ClusterConfig" → "port" | ~138 | +| 00:30 | Edited .kiro/specs/admin-config-tool/requirements.md | 4→5 lines | ~191 | +| 00:30 | Edited .kiro/specs/admin-config-tool/requirements.md | inline fix | ~108 | +| 00:31 | Edited .kiro/specs/admin-config-tool/requirements.md | 1→3 lines | ~265 | +| 00:31 | Session end: 26 writes across 4 files (requirements.md, brief.md, spec.json, config-design.md) | 8 reads | ~25514 tok | +| 00:33 | Session end: 26 writes across 4 files (requirements.md, brief.md, spec.json, config-design.md) | 8 reads | ~25514 tok | +| 00:46 | Edited .kiro/steering/roadmap.md | 1→2 lines | ~596 | +| 00:46 | Edited .kiro/steering/config-design.md | expanded (+8 lines) | ~401 | +| 00:47 | Edited .kiro/steering/config-design.md | expanded (+8 lines) | ~1225 | +| 00:47 | Edited .kiro/specs/admin-config-tool/brief.md | 9→10 lines | ~392 | +| 00:48 | Edited .kiro/specs/admin-config-tool/brief.md | inline fix | ~262 | +| 00:48 | Edited .kiro/specs/admin-config-tool/requirements.md | inline fix | ~131 | +| 00:50 | Created .kiro/specs/clusterconfig-schema-split/requirements.md | — | ~3201 | +| 00:50 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | 8→8 lines | ~55 | +| 00:50 | Session end: 34 writes across 5 files (requirements.md, brief.md, spec.json, config-design.md, roadmap.md) | 21 reads | ~73596 tok | +| 00:54 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | 11→14 lines | ~433 | +| 00:54 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | 3→3 lines | ~347 | +| 00:55 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | 1→2 lines | ~185 | +| 00:55 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | inline fix | ~12 | +| 00:55 | Session end: 38 writes across 5 files (requirements.md, brief.md, spec.json, config-design.md, roadmap.md) | 22 reads | ~77973 tok | + +## Session: 2026-07-20 01:05 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 01:14 | Created ../../../../../claude/.claude/plans/ancient-beaming-kahn.md | — | ~932 | +| 01:16 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | 1→3 lines | ~337 | +| 01:16 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | inline fix | ~168 | +| 01:17 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | "port" → "candidates" | ~422 | +| 01:17 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | listener() → mechanism() | ~434 | +| 01:17 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | inline fix | ~91 | +| 01:17 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | 1→2 lines | ~316 | +| 01:18 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | 2→1 lines | ~48 | +| 01:18 | Edited .kiro/specs/clusterconfig-schema-split/requirements.md | inline fix | ~76 | +| 01:18 | Created ../../../../../claude/.claude/plans/ancient-beaming-kahn.md | — | ~1104 | +| 01:19 | Edited .kiro/steering/roadmap.md | inline fix | ~380 | +| 01:19 | Session end: 11 writes across 3 files (ancient-beaming-kahn.md, requirements.md, roadmap.md) | 24 reads | ~170578 tok | +| 01:19 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | inline fix | ~12 | +| 01:19 | Session end: 12 writes across 4 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json) | 24 reads | ~170590 tok | +| 01:21 | Session end: 12 writes across 4 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json) | 27 reads | ~170590 tok | +| 01:22 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | 4→4 lines | ~22 | +| 01:23 | Session end: 13 writes across 4 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json) | 39 reads | ~186735 tok | +| 01:24 | Created ../../../../../claude/.claude/plans/ancient-beaming-kahn.md | — | ~691 | +| 01:26 | Created .kiro/specs/clusterconfig-schema-split/research.md | — | ~2107 | +| 01:27 | Created ../../../../../claude/.claude/plans/ancient-beaming-kahn.md | — | ~495 | +| 01:28 | Created ../../../../../claude/.claude/plans/ancient-beaming-kahn.md | — | ~974 | +| 01:29 | Created .kiro/specs/clusterconfig-schema-split/design.md | — | ~5870 | +| 01:29 | Session end: 18 writes across 6 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json, research.md) | 40 reads | ~203106 tok | +| 01:30 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 3→8 lines | ~188 | +| 01:30 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | 12→12 lines | ~74 | +| 01:30 | Session end: 20 writes across 6 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json, research.md) | 40 reads | ~203381 tok | +| 01:30 | Session end: 20 writes across 6 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json, research.md) | 40 reads | ~203381 tok | +| 01:36 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 2→3 lines | ~94 | +| 01:37 | Edited .kiro/specs/clusterconfig-schema-split/design.md | added 1 condition(s) | ~582 | +| 01:38 | Edited .kiro/specs/clusterconfig-schema-split/design.md | modified node() | ~948 | +| 01:39 | Edited .kiro/specs/clusterconfig-schema-split/research.md | modified decision() | ~563 | +| 01:39 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | 12→12 lines | ~73 | +| 01:39 | Session end: 25 writes across 6 files (ancient-beaming-kahn.md, requirements.md, roadmap.md, spec.json, research.md) | 42 reads | ~209293 tok | + +## Session: 2026-07-20 01:41 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 01:43 | Edited src/scheduler/src/scheduler.c | modified if() | ~109 | +| 01:47 | Edited src/lib/cluster.c | modified do_connect() | ~101 | +| 01:47 | Edited src/lib/cluster.c | 3→1 lines | ~9 | +| 01:47 | Edited src/lib/cluster.c | modified for() | ~57 | +| 01:48 | Edited src/include/xce/cluster.h | modified struct() | ~170 | +| 01:48 | Edited src/include/xce/cluster.h | 6→5 lines | ~46 | +| 01:48 | Edited src/include/xce/cluster.h | modified void() | ~53 | +| 02:11 | Edited src/lib/testsuite/test_cluster.c | modified mock_connect() | ~30 | +| 02:11 | Edited src/lib/testsuite/test_cluster.c | 4→2 lines | ~26 | +| 02:12 | Edited src/worker/src/main.c | modified main() | ~157 | +| 02:12 | Edited src/worker/src/main.c | 6→5 lines | ~59 | +| 02:12 | Edited src/worker/src/main.c | modified VIRTUAL_CLUSTER() | ~143 | +| 02:12 | Edited src/worker/src/main.c | modified if() | ~209 | +| 02:12 | Edited src/scheduler/src/main.c | 14→12 lines | ~106 | +| 02:13 | Edited src/scheduler/src/main.c | 7→4 lines | ~20 | +| 02:13 | Edited src/scheduler/src/main.c | modified if() | ~196 | +| 02:13 | Edited src/controller/main.c | removed 5 lines | ~16 | +| 09:19 | ran xce-build.sh all asan (bug-1453/1454 C-side validation); 1st run hit known bug-1439 ninja capnp.h race, retry succeeded 353/353, all 9 meson suites OK incl test_cluster+scheduler | src/xce.build, .wolf/buglog.json | success (build+tests green) | ~1200 | +| 02:22 | Edited src/frontend/xcecli/config/config.go | 6→5 lines | ~56 | +| 02:22 | Edited src/frontend/xcecli/models/logon.go | 7→6 lines | ~41 | +| 02:22 | Edited src/frontend/xcecli/common/request/request.go | 19→17 lines | ~48 | +| 02:22 | Edited src/frontend/xcecli/common/request/request.go | removed 29 lines | ~10 | +| 02:23 | Edited src/frontend/xcecli/common/request/request.go | 8→3 lines | ~51 | +| 02:23 | Edited src/frontend/xcecli/common/request/request.go | 8→3 lines | ~47 | +| 02:23 | Edited src/frontend/xcecli/common/request/request.go | 8→3 lines | ~48 | +| 02:23 | Edited src/frontend/xcecli/cmd/auth/logon.go | 17→15 lines | ~53 | +| 02:23 | Edited src/frontend/xcecli/cmd/auth/logon.go | modified updateServerConfig() | ~232 | +| 02:24 | Edited src/frontend/xcecli/cmd/auth/logon.go | reduced (-14 lines) | ~35 | +| 02:24 | Edited src/frontend/xcecli/cmd/auth/logon.go | 2→2 lines | ~27 | +| 02:24 | Edited src/frontend/xcecli/cmd/jobs/list.go | 4→3 lines | ~22 | +| 02:24 | Edited src/frontend/xcecli/cmd/jobs/list.go | inline fix | ~27 | +| 02:24 | Edited src/frontend/xcecli/cmd/jobs/list.go | inline fix | ~31 | +| 02:24 | Edited src/frontend/xcecli/cmd/jobs/list.go | ", serverInfo.CACert, " → ", " | ~30 | +| 02:25 | Edited src/frontend/xcecli/cmd/jobs/restart.go | 32→30 lines | ~219 | +| 02:25 | Edited src/frontend/xcecli/cmd/jobs/controller.go | 15→14 lines | ~118 | +| 02:25 | Edited src/frontend/xcecli/cmd/jobs/controller.go | 30→29 lines | ~230 | +| 02:25 | Edited src/frontend/xcecli/cmd/jobs/submit.go | 5→4 lines | ~33 | +| 02:25 | Edited src/frontend/xcecli/cmd/jobs/submit.go | 8→7 lines | ~48 | +| 02:26 | Edited src/frontend/xcecli/cmd/jobs/submit.go | 7→6 lines | ~46 | +| 02:26 | Edited src/frontend/xcecli/cmd/roles/list.go | 4→3 lines | ~22 | +| 02:26 | Edited src/frontend/xcecli/cmd/roles/list.go | inline fix | ~28 | +| 02:26 | Edited src/frontend/xcecli/cmd/roles/addrole.go | 23→21 lines | ~134 | +| 02:26 | Edited src/frontend/xcecli/cmd/roles/addrole.go | 25→23 lines | ~147 | +| 02:27 | Edited src/frontend/xcecli/cmd/roles/deleterole.go | 12→10 lines | ~63 | +| 02:27 | Edited src/frontend/xcecli/cmd/users/list.go | 8→7 lines | ~95 | +| 02:27 | Edited src/frontend/xcecli/cmd/users/list.go | 4→3 lines | ~22 | +| 02:27 | Edited src/frontend/xcecli/cmd/users/list.go | inline fix | ~28 | +| 02:27 | Edited src/frontend/xcecli/cmd/users/assignrole.go | 23→21 lines | ~145 | +| 02:28 | Edited src/frontend/xcecli/cmd/users/unassignrole.go | 12→10 lines | ~67 | +| 02:28 | Edited src/frontend/xcecli/cmd/users/perm4user.go | 4→3 lines | ~22 | +| 02:28 | Edited src/frontend/xcecli/cmd/users/perm4user.go | inline fix | ~22 | +| 02:28 | Edited src/frontend/xcecli/cmd/hosts/list.go | 4→3 lines | ~22 | +| 02:28 | Edited src/frontend/xcecli/cmd/hosts/list.go | inline fix | ~28 | +| 02:28 | Edited src/frontend/xcecli/cmd/hosts/controller.go | 4→3 lines | ~21 | +| 02:28 | Edited src/frontend/xcecli/cmd/hosts/controller.go | 7→6 lines | ~30 | +| 02:28 | Edited src/frontend/xcecli/cmd/hosts/select.go | 13→12 lines | ~87 | +| 02:28 | Edited src/frontend/xcecli/cmd/queues/list.go | 4→3 lines | ~22 | +| 02:28 | Edited src/frontend/xcecli/cmd/queues/list.go | inline fix | ~28 | +| 02:28 | Edited src/frontend/xcecli/cmd/queues/controller.go | 4→3 lines | ~21 | +| 02:28 | Edited src/frontend/xcecli/cmd/queues/controller.go | 7→6 lines | ~30 | +| 02:29 | Edited src/frontend/xcecli/cmd/resources/list.go | 8→7 lines | ~52 | +| 02:29 | Edited src/frontend/xcecli/cmd/resources/list.go | inline fix | ~32 | +| 02:29 | Edited src/frontend/xcecli/cmd/resources/list.go | 8→7 lines | ~52 | +| 02:29 | Edited src/frontend/xcecli/cmd/resources/list.go | inline fix | ~32 | +| 02:29 | Edited src/frontend/xcecli/cmd/permissions/list.go | 9→8 lines | ~70 | +| 02:29 | Edited src/frontend/xcecli/cmd/clusters/list.go | 4→3 lines | ~22 | +| 02:29 | Edited src/frontend/xcecli/cmd/clusters/list.go | ", serverInfo.CACert, " → ", " | ~27 | +| 02:29 | Created src/frontend/xcecli/common/request/request_test.go | — | ~112 | +| 02:35 | bug-1453 Go-side CA-cert removal complete: config/models field, request.go SetRootCAs/loadRootCAs/certPath params, cert/cert.go+parser.go deleted, logon.go https-check block, 18 cmd files SetRootCAs calls, request_test.go 3 tests | src/frontend/xcecli/** | build+vet+test PASS, gofmt clean, grep-clean | ~4200 | +| 02:37 | Session end: 67 writes across 19 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 34 reads | ~23785 tok | +| 02:58 | Edited src/scheduler/src/scheduler.c | expanded (+8 lines) | ~148 | +| 03:01 | Session end: 68 writes across 19 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 34 reads | ~23944 tok | +| 03:02 | Edited src/scheduler/src/scheduler.c | modified sched_init() | ~46 | +| 03:03 | Edited src/scheduler/src/scheduler.c | added 1 condition(s) | ~32 | +| 03:03 | Edited src/scheduler/src/scheduler.c | modified for() | ~161 | +| 03:03 | Edited src/scheduler/src/scheduler.c | removed 10 lines | ~11 | +| 03:03 | Edited src/scheduler/include/scheduler.h | 4→3 lines | ~48 | +| 03:04 | Edited src/scheduler/src/main.c | 3→5 lines | ~54 | +| 03:04 | Edited src/scheduler/src/main.c | modified if() | ~47 | +| 03:04 | Edited src/scheduler/src/main.c | 10→10 lines | ~94 | +| 03:08 | Session end: 76 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26109 tok | +| 03:17 | Session end: 76 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26109 tok | +| 03:18 | Session end: 76 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26109 tok | +| 03:24 | Edited src/scheduler/src/scheduler.c | modified for() | ~251 | +| 03:26 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26295 tok | +| 03:34 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26295 tok | +| 03:36 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26295 tok | +| 03:53 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 35 reads | ~26295 tok | +| 03:56 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 36 reads | ~26295 tok | +| 10:56 | Verified dangling-pointer revert in scheduler.c sched_init HA candidate loop: full ASAN build 353/353, meson 9/9 OK, scheduler suite 15/15, Go tests pass. Logged bug-1456 (regression of bug-1454). Also noted: Bash run_in_background + explicit timeout param silently kills long builds — omit timeout for background tasks. | scheduler.c, buglog.json | build+tests green | ~4000 tok | +| 04:01 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 36 reads | ~26295 tok | +| 05:01 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 36 reads | ~26295 tok | +| 05:15 | Session end: 77 writes across 20 files (scheduler.c, cluster.c, cluster.h, test_cluster.c, main.c) | 36 reads | ~26295 tok | + +## Session: 2026-07-20 06:00 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 06:11 | Created .kiro/specs/clusterconfig-schema-split/tasks.md | — | ~2285 | +| 06:11 | Edited .kiro/specs/clusterconfig-schema-split/spec.json | 19→19 lines | ~105 | +| 06:11 | Session end: 2 writes across 2 files (tasks.md, spec.json) | 7 reads | ~19111 tok | +| 07:01 | Edited .kiro/specs/clusterconfig-schema-split/research.md | expanded (+11 lines) | ~676 | +| 07:01 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 9→9 lines | ~388 | +| 07:02 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 6→5 lines | ~147 | +| 07:02 | Edited .kiro/specs/clusterconfig-schema-split/design.md | inline fix | ~51 | +| 07:02 | Edited .kiro/specs/clusterconfig-schema-split/design.md | flag() → loop() | ~631 | +| 07:03 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 16→15 lines | ~561 | +| 07:03 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 24→24 lines | ~650 | +| 07:03 | Edited .kiro/specs/clusterconfig-schema-split/design.md | "--port" → "--peers" | ~49 | +| 07:04 | Edited .kiro/specs/clusterconfig-schema-split/design.md | 17→17 lines | ~496 | +| 07:04 | Edited .kiro/specs/clusterconfig-schema-split/design.md | modified passed() | ~999 | +| 07:05 | Edited .kiro/specs/clusterconfig-schema-split/design.md | inline fix | ~210 | +| 07:05 | Edited .kiro/specs/clusterconfig-schema-split/design.md | inline fix | ~45 | +| 07:06 | Created .kiro/specs/clusterconfig-schema-split/tasks.md | — | ~2261 | +| 07:15 | Edited .kiro/specs/clusterconfig-schema-split/design.md | "--peers =::" | ~8 | +| 07:15 | Edited .kiro/specs/clusterconfig-schema-split/tasks.md | "--peers =::" | ~8 | +| 07:15 | Edited .kiro/specs/clusterconfig-schema-split/design.md | modified matching() | ~271 | +| 07:15 | Edited .kiro/specs/clusterconfig-schema-split/research.md | 1→2 lines | ~211 | +| 07:16 | Edited .kiro/specs/clusterconfig-schema-split/design.md | "--peers :" → "--peers =::" → "--peers =: Killing any running code-review-graph / crg-daemon processes" -pkill -f "code-review-graph serve" 2>/dev/null || true -pkill -f "code-review-graph.*daemon" 2>/dev/null || true -pkill -f "uvx code-review-graph" 2>/dev/null || true -pkill -f "crg-daemon" 2>/dev/null || true -sleep 1 - -echo "==> Uninstalling uv package" -VIRTUAL_ENV="$UVENV" uv pip uninstall code-review-graph 2>/dev/null || true - -echo "==> Removing leftover binaries" -rm -f "$UVENV/bin/code-review-graph" "$UVENV/bin/crg-daemon" - -echo "==> Removing state directories" -rm -rf "$WT/.code-review-graph" \ - /home/rshen/projects/xce/xce/.code-review-graph \ - /home/claude/.code-review-graph \ - "/home/claude/.cache/claude-cli-nodejs/-home-rshen-projects-xce-worktrees-persistent-ha/mcp-logs-code-review-graph" - -echo "==> Removing git pre-commit hook (if it is the code-review-graph one)" -HOOK="/home/rshen/projects/xce/xce/.git/hooks/pre-commit" -if [ -f "$HOOK" ] && grep -q "code-review-graph" "$HOOK"; then rm -f "$HOOK"; fi - -echo "==> Purging uv cache entries" -find /home/claude/.cache/uv -maxdepth 4 -iname "*code_review_graph*" -exec rm -rf {} + 2>/dev/null || true - -echo "==> Verification" -if command -v code-review-graph >/dev/null 2>&1; then - echo " STILL PRESENT — a Claude session is probably still respawning it. Close it and re-run." -else - echo " code-review-graph: GONE" -fi -echo " .mcp.json refs: $(grep -c code-review-graph "$WT/.mcp.json" 2>/dev/null || echo 0)" -echo " settings.json hook refs: $(grep -c code-review-graph "$WT/.claude/settings.json" 2>/dev/null || echo 0)" -echo "Done."