From d70bf6a4021e4e7ab5ecb74269a5dc568393084d Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 10 Aug 2026 17:51:51 +0800 Subject: [PATCH] update documents --- .../clusterconfig-schema-split/design.md | 24 +- .kiro/specs/controller-status/design.md | 18 +- .kiro/specs/election-iomgr-refactor/design.md | 135 +- .../specs/ha-commit-barrier-rollout/design.md | 73 +- .kiro/specs/ha-commit-durability/design.md | 46 +- .kiro/specs/ha-commit-engine/design.md | 24 +- .kiro/specs/ha-config-bootstrap/design.md | 24 +- .../specs/ha-consistency-invariants/design.md | 26 +- .kiro/specs/ha-convergence-recovery/design.md | 64 +- .kiro/specs/ha-elect-ctx-lifecycle/design.md | 8 +- .kiro/specs/ha-entity-snapshots/design.md | 57 +- .../design.md | 8 +- .kiro/specs/ha-heartbeat-liveness/design.md | 28 +- .../specs/ha-peer-connection-reuse/design.md | 64 +- .kiro/specs/ha-prevote/design.md | 14 +- .kiro/specs/ha-repl-hardening/design.md | 60 +- .kiro/specs/ha-wal-replication/design.md | 6 +- .kiro/steering/api-standards.md | 18 +- .kiro/steering/build-deployment.md | 21 +- .kiro/steering/error-handling.md | 20 +- .kiro/steering/ha-async-io.md | 12 +- .kiro/steering/product.md | 14 +- .kiro/steering/roadmap.md | 22 +- .kiro/steering/security.md | 10 +- .kiro/steering/structure.md | 19 +- .kiro/steering/tech.md | 30 +- .kiro/steering/testing.md | 6 +- CLAUDE.md | 4 +- ...-05-07-p0-single-controller-persistence.md | 2 + .../plans/2026-05-07-p1-ha-foundation.md | 2 + .../plans/2026-05-07-p2-replication.md | 2 + .../plans/2026-05-07-p3-hardening.md | 2 + .../2026-05-07-persistence-ha-overview.md | 2 + .../plans/2026-05-13-config-replication.md | 2 + .../2026-05-20-config-repl-wire-protocol.md | 2 + reports/aug03-aug09-2026.md | 86 ++ src/controller/CLAUDE.md | 37 +- src/docs/FUTURE_WORK.md | 2 +- src/docs/apiserver-backend-tasks.md | 10 +- .../archive/cluster-config-split-analysis.md | 2 + src/docs/archive/job_peek.md | 2 + src/docs/config-replication-design.md | 356 +---- src/docs/config-replication-impl-status.md | 47 +- src/docs/deployment.md | 51 +- src/docs/p1-2-host-select.md | 3 +- src/docs/persistence-and-ha-design.md | 1210 +---------------- src/docs/resreq-ast-cse-design.md | 274 ++-- src/docs/xce.md | 12 +- src/lib/CLAUDE.md | 15 +- src/scheduler/CLAUDE.md | 6 +- src/worker/CLAUDE.md | 10 +- 51 files changed, 995 insertions(+), 1997 deletions(-) create mode 100644 reports/aug03-aug09-2026.md diff --git a/.kiro/specs/clusterconfig-schema-split/design.md b/.kiro/specs/clusterconfig-schema-split/design.md index b8ae1752..6d81b948 100644 --- a/.kiro/specs/clusterconfig-schema-split/design.md +++ b/.kiro/specs/clusterconfig-schema-split/design.md @@ -83,7 +83,7 @@ graph TB **Architecture Integration**: - Selected pattern: no new CLI surface — `port` resolution reuses the existing `--peers` candidate-match loop (a single self-referencing `--peers =:` entry already gives single-node mode a custom port today) plus a hardcoded fallback constant for the true zero-`--peers` case. - Domain/feature boundaries: `ClusterConfig` (schema) retains only fields that are genuinely cluster-shared (per `config-design.md`); `peers_cfg` (CLI-sourced, per-process, already existing) owns port resolution for everything this spec touches. -- Existing patterns preserved: `--peers` remains the sole port/membership source; `init_controller_tunables()`'s hardcoded constants are untouched (Requirement 2.2 — no behavior change). +- Existing patterns preserved: `--peers` remains the sole port/membership source; `init_controller_tunables()`'s hardcoded constants are the default values and reproduce today's exact behavior when no tunable flag is passed (Requirement 2.2 — no behavior change by default). (Note: the optional Requirement 6 CLI-sourcing was in fact implemented — `init_controller_tunables()` now reads the corresponding `myconf_t` field first, falling back to the hardcoded constant only when that field is 0/unset — so the function itself is no longer literally untouched, though default behavior is unchanged.) - New components: one new hardcoded constant (`cluster.c` or `clusterconfig.h`) — the minimum needed to satisfy Requirement 1.7 without any invocation-contract or CLI-surface change. - Steering compliance: enforces `.kiro/steering/config-design.md`'s independence/frequency/Rule-0 classification; follows `src/controller/CLAUDE.md`'s HA protocol verification checklist for the touched files. @@ -100,7 +100,7 @@ graph TB ### Modified Files - `src/schema/controller.capnp` — remove `port` (@1), `candidates` (@2), `vcpus` (@7), `jobmgr` (@13), `workermgr` (@14), `qmgr` (@15), `resmgr` (@16), `xstrings` (@17) from `ClusterConfig`; remove the `MasterCandidate` struct entirely; remove the now-orphaned `HashZArrayConfig`/`ResmgrConfig` structs (used only by the removed fields — discovered during `/kiro-impl` task 2.1, their `$C.mapname` bindings broke the capnp-generated code once task 1.2 removed the C typedefs they referenced). Cap'n Proto requires gapless ordinals, so the 13 retained fields are renumbered to a contiguous `0..12` range as a direct consequence — see `research.md`'s ordinal-gap finding (Req 1.1, 1.2, 1.6, 2.1, 2.5, 3.1). -- `ha/data/cluster.bin` (binary fixture, not source) — regenerated in place against the renumbered schema, preserving the retained fields' semantic values from the pre-spec blob (Req 3.2, 5.3). +- `ha/data/cluster.bin` (binary fixture, not source) — regenerated in place against the renumbered schema, preserving the retained fields' semantic values from the pre-spec blob (Req 3.2, 5.3). (Note: superseded by `admin-config-tool` (landed 2026-07-22, one day after this spec closed) — `ha/data/` was deleted entirely and `deploy-ha.sh` now builds `cluster.bin` fresh via `xceadmin config build` from `src/configs/*.yaml` on every deploy run, not from a checked-in regenerated blob. See that spec's design.md.) - `src/include/clusterconfig.h` — remove the matching `cluster_config_t` members: `port`, `ncandidates`, `candidates`/`master_candidate_t**`, `vcpus`, `jobmgr`, `workermgr`, `qmgr`, `resmgr`, `xstrings` (Req 1.5, 2.4). - `src/controller/cluster.c` — (a) remove the `candidates`/`ncandidates` KV-mirror loop in `cluster_build_config` (Req 1.4); (b) change `cluster_open_listener`'s fallback branch from reading `cluster_config->port` to a hardcoded default constant, reached only when `--peers` was never passed at all — the `--peers` candidate-match branch itself is unchanged (Req 1.3, 1.7); `init_controller_tunables()` itself is unchanged (Req 2.2). - `src/controller/myconf.c` — remove the `SET_DEFAULT_IF_UNSET(c->port, 0, 9990)` call (the field it defaults no longer exists) (part of Req 1.7's cleanup). @@ -111,9 +111,9 @@ graph TB ### Verification Infrastructure (existing, no new files) - `controller_unit_test` (meson target, existing) — runs the full existing suite including the two updated test files above (Req 5.2). -- A small throwaway capnp-encoding tool (written and discarded during implementation, not part of the shipped codebase) — decodes `ha/data/cluster.bin`'s retained-field values under the pre-spec schema and re-encodes them under the post-spec (renumbered) schema, overwriting the fixture (Req 3.2, 5.3). -- `/xce-deploy-ha` skill (existing, `src/tools/deploy-ha.sh`) — deploys the 3-node live cluster for the config-push cycle, using the regenerated `ha/data/cluster.bin` (Req 5.4). -- Direct `xce_controller` invocation with no `--peers` flag (existing supported mode, `main.c:293-295`) — the single-node startup verification (Req 5.5). +- A small throwaway capnp-encoding tool (written and discarded during implementation, not part of the shipped codebase) — decodes `ha/data/cluster.bin`'s retained-field values under the pre-spec schema and re-encodes them under the post-spec (renumbered) schema, overwriting the fixture (Req 3.2, 5.3). (Note: the fixture this tool targeted no longer exists — see the `ha/data/cluster.bin` note above.) +- `/xce-deploy-ha` skill (existing, `src/tools/deploy-ha.sh`) — deploys the 3-node live cluster for the config-push cycle, using the regenerated `ha/data/cluster.bin` (Req 5.4). (Note: superseded by `admin-config-tool` — `deploy-ha.sh` now generates `cluster.bin` fresh via `xceadmin config build` instead of consuming this fixture. See that spec's design.md.) +- Direct `xce_controller` invocation with no `--peers` flag (existing supported mode, `main.c:441-443` — line numbers shifted from the original `main.c:293-295` after this spec's own Requirement 6 CLI flags were added earlier in the file) — the single-node startup verification (Req 5.5). - Direct `xce_controller` invocation with a single self-referencing `--peers =:` entry (existing supported mode) — verifies the custom-port replacement mechanism itself (Req 5.5). ### Optional (Requirement 6, nice-to-have) @@ -149,8 +149,8 @@ flowchart TD ``` **Key decisions** (corrected from the initial draft during `/kiro-validate-design`; the `--port` CLI flag was further revised away in favor of the mechanism below during `/kiro-impl` kickoff — see `research.md`'s "Design Revision" section): -- **Fallback trigger is port value, not just match presence.** A matched candidate whose port field is `0` (a malformed `--peers` entry — already WARNed about at `main.c:289-291`) falls through to the hardcoded-default tier exactly like today's `if (my_port == 0)` check, rather than binding to literal port 0. This preserves Requirement 1.3's "exactly as it does today" for the already-tolerated malformed-entry case. -- **Error path is preserved, but only for one specific case.** `candidates_count == 0` can only happen when `--peers` was never passed at all — a passed-but-empty or malformed `--peers` string is rejected earlier by `xce_parse_peers` itself (`src/lib/cluster.c:58` returns `-1` unless `candidates_count > 0`), which `main.c` treats as a startup error before `cluster_open_listener` is ever reached (corrected citation, 2026-07-20: this guarantee comes from `xce_parse_peers`'s contract, not `validate_peer_count`, which only rejects even non-zero counts) — that's genuine single-node mode, and gets the hardcoded-default fallback per Requirement 1.7. An operator who wants a non-default port in this mode passes a single self-referencing `--peers =:` entry instead — an already-existing, already-supported invocation (this resolves via the unchanged `NameFound`/`PortValid` branches above, not a new mechanism). But when `--peers` WAS passed and the local name simply isn't in it, that's a likely operator misconfiguration (own name typo'd or omitted) — this spec keeps today's `-EINVAL`/refuse-to-start behavior for that case rather than silently defaulting, per the project's fail-loud convention (`src/controller/CLAUDE.md` process-lifecycle invariant). Decided interactively during design review rather than defaulting to "simplest single code path." +- **Fallback trigger is port value, not just match presence.** A matched candidate whose port field is `0` (a malformed `--peers` entry — already WARNed about at `main.c:434-439`, was `main.c:289-291`; line numbers shifted after this spec's own Requirement 6 CLI flags were added earlier in the file) falls through to the hardcoded-default tier exactly like today's `if (my_port == 0)` check, rather than binding to literal port 0. This preserves Requirement 1.3's "exactly as it does today" for the already-tolerated malformed-entry case. +- **Error path is preserved, but only for one specific case.** `candidates_count == 0` can only happen when `--peers` was never passed at all — a passed-but-empty or malformed `--peers` string is rejected earlier by `xce_parse_peers` itself (`src/lib/cluster.c:59` returns `-1` unless `candidates_count > 0`; was cited as `:58`), which `main.c` treats as a startup error before `cluster_open_listener` is ever reached (corrected citation, 2026-07-20: this guarantee comes from `xce_parse_peers`'s contract, not `validate_peer_count`, which only rejects even non-zero counts) — that's genuine single-node mode, and gets the hardcoded-default fallback per Requirement 1.7. An operator who wants a non-default port in this mode passes a single self-referencing `--peers =:` entry instead — an already-existing, already-supported invocation (this resolves via the unchanged `NameFound`/`PortValid` branches above, not a new mechanism). But when `--peers` WAS passed and the local name simply isn't in it, that's a likely operator misconfiguration (own name typo'd or omitted) — this spec keeps today's `-EINVAL`/refuse-to-start behavior for that case rather than silently defaulting, per the project's fail-loud convention (`src/controller/CLAUDE.md` process-lifecycle invariant). Decided interactively during design review rather than defaulting to "simplest single code path." ## Requirements Traceability @@ -165,7 +165,7 @@ flowchart TD | 1.7 | Replacement port-sourcing mechanism (existing `--peers` self-reference + hardcoded default) | Listener/Port-Resolution Component | `cluster.c`, `myconf.c`, `config_repl.c` | | 2.1 | Remove 6 perf-tuning fields from schema | Schema Component | `controller.capnp` | | 2.5 | Remove orphaned `HashZArrayConfig`/`ResmgrConfig` structs | Schema Component | `controller.capnp` | -| 2.2 | Hardcoded tunables unchanged | Tunables Component | `cluster.c` (no change) | +| 2.2 | Hardcoded tunables unchanged (as defaults) | Tunables Component | `cluster.c` (values unchanged; function itself later extended by the now-implemented Req 6.2 CLI-sourcing wrapper) | | 2.3 | Controller stops attempting to read removed tunable fields | Tunables Component | `cluster.c` (implicit — field no longer exists) | | 2.4 | `cluster_config_t` drops tunable members | C Struct Component | `clusterconfig.h` | | 3.1 | Retained fields unchanged in type/number | Schema Component | `controller.capnp` (no change to retained fields) | @@ -175,7 +175,7 @@ flowchart TD | 4.2 | No deprecate-in-place transition | Schema Component | `controller.capnp` | | 5.1 | Build + test-fixture compile cleanly | Test Fixture Updates | `test_commit_barrier_rollout.c`, `test_config_repl.c` | | 5.2 | No new unit test failures | Verification | controller unit test suite | -| 5.3 | Regenerate `ha/data/cluster.bin` fixture against the new schema | Verification | `ha/data/cluster.bin` | +| 5.3 | Regenerate `ha/data/cluster.bin` fixture against the new schema | Verification | `ha/data/cluster.bin` (Note: `ha/data/` was later deleted entirely by `admin-config-tool`; see the "Note" on this file in File Structure Plan above) | | 5.4 | 3-node live soak (config-push, peer/listener/worker-liveness) | Verification | live HA cluster | | 5.5 | Single-node + bare-peers port-override + `-EINVAL` live verification | Verification | live single-node controller | | 6.1 | (Nice-to-have) CLI tunable flags | CLI Tunables Component (optional) | `main.c`, `myconf.h` | @@ -251,20 +251,20 @@ int cluster_open_listener(cluster_t *c); No new error categories are introduced, and the existing `-EINVAL` error path (`cluster_open_listener`'s "no port could be determined," `cluster.c:1226` today) is **preserved**, not removed — corrected during design review, since single-code-path simplicity would have traded a clear startup error for a silent wrong-default in a likely-misconfiguration scenario (`--peers` given, local name absent). It now applies to exactly one case: `--peers` was passed and the local name isn't in it. The genuinely-new case (`--peers` never passed at all, i.e. single-node) and the pre-existing-tolerated case (name matched, that entry's port is `0`) both resolve via the hardcoded-default fallback and never error — unchanged from today's tolerance for the latter, newly-defined for the former. All other error handling (schema decode failures, capnp field-absence for legacy blobs) is capnp's existing default behavior: an old `cluster.bin` with data in now-unused field slots decodes without error, and those values are simply never read (Requirement 4.1) — no new error branch needed for that case either. ### Monitoring -No new logging beyond what already exists for `--peers` parsing (`main.c:284-291`) and controller startup (`INFO`/`WARN`/`ERROR` via the existing `mylog` macros). No new metrics. +No new logging beyond what already exists for `--peers` parsing (`main.c:411-443`, was `main.c:284-291` — shifted after this spec's own Requirement 6 CLI flags were added earlier in the file) and controller startup (`INFO`/`WARN`/`ERROR` via the existing `mylog` macros). No new metrics. ## Testing Strategy - **Unit**: build-time enforcement is the primary "test" for Requirements 1.1-1.2, 1.5-1.6, 2.1, 2.4 — a removed field/struct member that's still referenced anywhere is a compile error, not a silent gap (this is why Requirement 5.1's test-fixture fix is explicit rather than assumed). - **Unit**: `controller_unit_test` full suite run (Requirement 5.2) — existing coverage for `cluster_build_config`, `init_controller_tunables`, and the two touched test files, now compiling against the shrunk struct. -- **Integration**: regenerate the `ha/data/cluster.bin` fixture (Requirement 3.2, 5.3) — since retained-field renumbering breaks binary compatibility with any pre-spec blob, decode the current fixture's semantic field values under the pre-spec schema and re-encode them under the new schema with a small throwaway tool; confirm the regenerated blob decodes correctly under the post-spec schema and `deploy-ha.sh`'s verbatim-copy usage keeps working unmodified. +- **Integration**: regenerate the `ha/data/cluster.bin` fixture (Requirement 3.2, 5.3) — since retained-field renumbering breaks binary compatibility with any pre-spec blob, decode the current fixture's semantic field values under the pre-spec schema and re-encode them under the new schema with a small throwaway tool; confirm the regenerated blob decodes correctly under the post-spec schema and `deploy-ha.sh`'s verbatim-copy usage keeps working unmodified. (Note: superseded by `admin-config-tool` — `deploy-ha.sh` no longer does verbatim-copy of a fixture at all; it builds `cluster.bin` fresh via `xceadmin config build` from YAML on every run. See that spec's design.md.) - **Live/E2E**: 3-node HA cluster deploy + config-push cycle (Requirement 5.4) — using the regenerated fixture; peer discovery via `--peers`, listener binding, worker-liveness tracking all behave identically to pre-spec. - **Live/E2E**: single-node controller startup with no `--peers` flag, AND with a single self-referencing `--peers` entry for a custom port (Requirement 5.5) — the actual regression risk this spec's design resolves; must be verified live, not inferred from the 3-node soak (single-node never exercises the `--peers`-match branch at all). - **Live/E2E (optional, Requirement 6)**: if the nice-to-have CLI tunable flags are implemented, verify each flag overrides its corresponding hardcoded default and that omitting all of them reproduces today's exact hardcoded values. ## Migration Strategy -No data migration is required for the fields this spec targets — Requirement 4.2 explicitly rules out a deprecate-in-place transition, and Requirement 4.1 confirms retained-field renumbering is an accepted, unguarded compatibility break for this pre-1.0 product: a `cluster.bin` built before this spec is not expected to be usable as-is against a controller built after it, and no backward-read path is supported or verified for such an artifact. The one thing this spec DOES migrate is the checked-in `ha/data/cluster.bin` test fixture (Requirement 3.2, 5.3): its retained-field values are decoded under the pre-spec schema and re-encoded under the new schema with a small throwaway tool, so `/xce-deploy-ha` and dependent live-verification tooling keep working. There is no forward path either (a post-spec `cluster.bin` is not expected to be readable by a pre-spec controller, but that direction isn't a supported downgrade path in this project regardless of this spec). +No data migration is required for the fields this spec targets — Requirement 4.2 explicitly rules out a deprecate-in-place transition, and Requirement 4.1 confirms retained-field renumbering is an accepted, unguarded compatibility break for this pre-1.0 product: a `cluster.bin` built before this spec is not expected to be usable as-is against a controller built after it, and no backward-read path is supported or verified for such an artifact. The one thing this spec DOES migrate is the checked-in `ha/data/cluster.bin` test fixture (Requirement 3.2, 5.3): its retained-field values are decoded under the pre-spec schema and re-encoded under the new schema with a small throwaway tool, so `/xce-deploy-ha` and dependent live-verification tooling keep working. (Note: superseded by `admin-config-tool`, landed 2026-07-22 — `ha/data/cluster.bin` was deleted entirely and `/xce-deploy-ha` now generates `cluster.bin` fresh via `xceadmin config build` from YAML, so this migrated fixture no longer exists or is consumed. See that spec's design.md.) There is no forward path either (a post-spec `cluster.bin` is not expected to be readable by a pre-spec controller, but that direction isn't a supported downgrade path in this project regardless of this spec). ## Supporting References - `src/docs/cluster-config-split-analysis.md` — prior team analysis; primary design input for the dead-field classification. diff --git a/.kiro/specs/controller-status/design.md b/.kiro/specs/controller-status/design.md index 1c79b85a..244e2cb6 100644 --- a/.kiro/specs/controller-status/design.md +++ b/.kiro/specs/controller-status/design.md @@ -47,8 +47,8 @@ ### Existing Architecture Analysis - **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.** -- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:1784`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation). -- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:2059`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch. +- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:3802`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation). +- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:4525`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch. - **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`). ### Pattern & dependency direction @@ -249,18 +249,18 @@ Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `co ### Boundary Commitments (delta) - **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`. -- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only. -- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) and `ha->last_known_committed_config_tag` (`uint64_t`, `ha.h:172`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering. -- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. +- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only. (Note: superseded by **ha-heartbeat-committed-propagation** — that spec implemented exactly this, adding `ha_consume_heartbeat_committed`/`ha_observe_heartbeat_committed`, invoked from the primary's heartbeat in `cluster.c`, which now refreshes `last_known_committed_lsn`/`last_known_committed_config_tag` on every heartbeat, term-guarded and monotonic. See that spec's design.md.) +- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:183`) and `ha->last_known_committed_config_tag` (`ha.h:185`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering. (Note: `last_known_committed_config_tag` is now also `_Atomic uint64_t` — migrated by ha-heartbeat-committed-propagation from the plain `uint64_t` this design was written against.) +- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, `ha_observe_heartbeat_committed` on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.) ### Locked design decisions (from research items) -1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:2570`, `ha.c:4126`.) -2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2). +1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:5319` (`ha_vote_self_freshness`), `ha.c:6732`.) +2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2). (Note: superseded by **ha-heartbeat-committed-propagation** — `last_known_committed_lsn`/`_config_tag` are now also refreshed on every heartbeat via `ha_observe_heartbeat_committed`, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.) 3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.) 4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line. ### Research item carried to implementation -- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` is non-atomic (`ha.h:172`, written under lock at `ha_convergence.c:377`); the status read is a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. +- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` was non-atomic at design time (written under lock in `ha_convergence.c`); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by **ha-heartbeat-committed-propagation**, which migrated the field to `_Atomic uint64_t` at `ha.h:185`, written via `atomic_store_explicit` at `ha_convergence.c:368` and `ha.c:9011`; the as-built `ctrlstatus_fill` correctly reads it with `atomic_load_explicit`, not a bare aligned load.) ### Schema delta (Cap'n Proto) Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder): @@ -310,4 +310,4 @@ No new files (the capnp field is generated; all logic slots into existing TUs). **Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility. -**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. +**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.) diff --git a/.kiro/specs/election-iomgr-refactor/design.md b/.kiro/specs/election-iomgr-refactor/design.md index 13bee004..3857b305 100644 --- a/.kiro/specs/election-iomgr-refactor/design.md +++ b/.kiro/specs/election-iomgr-refactor/design.md @@ -37,10 +37,10 @@ is handled normally. No semaphores, no wbuf signals, no changes to `cluster_run` - `vote_request_thread` + `am_i_primary_quorum` blocking loop → replaced by `ha_start_election_async()` - `on_connected` callback added to `netio.c` - `client_ssl_ctx` in `ha_state_t` + `myapp_run` -- `vote_reply_msg_handler` for iomgr-native reply handling +- `vote_reply_msg_handler` for iomgr-native reply handling (Note: this handler was later generalized into `peer_ctrl_msg_handler`, a single state-aware dispatcher that also routes repl acks and PreVote replies — superseded by ha-prevote, see that spec's design.md) - Election timeout watchdog thread - `ha_elect_evaluate_and_signal` extended to call `cluster_init_phase2` then set role -- `handle_request_vote` extended with role-based deny gate +- `handle_request_vote` extended with role-based deny gate (Note: `handle_request_vote` is now `#ifdef XCE_TESTING`-only; production vote dispatch is `cluster.c`'s `case OPCode_requestVote:` calling `raft_on_vote_request()` — superseded by raft-adapter-conformance, see that spec's design.md) - `myapp_run` restructured: phase1 + SSL + `ha_start_election_async` + `cluster_run` directly ### Out of Boundary @@ -102,12 +102,12 @@ graph TD ### Key Decisions -- **`ha_start_election_async()` is fully non-blocking**: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately. +- **`ha_start_election_async()` is fully non-blocking**: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately. (Note: peer vote-endpoint connection and vote tallying are no longer driven directly by this function — they are now owned by the Raft library (`src/lib/raft/raft.c`): `ha_start_election_async()` calls `raft_start_election()`, which fires `raft_cb_send_vote_request` per peer and `raft_cb_become_leader`/`raft_cb_become_follower` on a terminal result. Superseded by raft-log-replication and raft-commit-safety, see those specs' design.md.) - **Two promotion paths in `ha_elect_evaluate_and_signal`**: - *Initial startup* (`ha->phase2_done == 0`): calls `cluster_init_phase2` (loads config data, inits structures), sets `ha->phase2_done = 1`. Phase2 runs exactly once per process lifetime — config upgrade always does a full restart. - - *RDHUP failover* (`ha->phase2_done == 1`): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly. -- **`is_first_boot` = `ha->phase2_done == 0`**: Not `!c->running` (which is 0 until `cluster_run` sets it, making all elections look like first boot). `phase2_done` is a dedicated bool in `ha_state_t`. -- **`CTRL_PRIMARY_COMMITTING`**: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary). + - *RDHUP failover* (`ha->phase2_done == 1`): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly. (Note: see CTRL_PRIMARY_COMMITTING note below — this now enters CTRL_PRIMARY directly, no COMMITTING state.) +- **`is_first_boot` = `ha->phase2_done == 0`**: Not `!c->running` (which is 0 until `cluster_run` sets it, making all elections look like first boot). `phase2_done` is a dedicated bool in `ha_state_t`. (Still accurate — verified current in `ha.c`.) +- **`CTRL_PRIMARY_COMMITTING`**: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary). **(Note: superseded by ha-single-leader-role, see that spec's design.md — `CTRL_PRIMARY_COMMITTING` was subsequently removed. `ctrl_role_t` now has only 4 values (`CTRL_PRIMARY`, `CTRL_STANDBY`, `CTRL_INIT`, `CTRL_ELECTING`); a winning candidate enters `CTRL_PRIMARY` directly via `raft_cb_become_leader`/`ha_elect_evaluate_and_signal`, with no transitional committing state and no committing-timeout watchdog. Job-traffic admission during startup is now gated by `ha_has_write_quorum()` inside the `CTRL_PRIMARY` case of `ha_dispatch_gate`, not by a separate FSM state. Every other reference to CTRL_PRIMARY_COMMITTING in this document — the sequence diagrams, the `ha_repl_ack_received()`/`ha_dispatch_gate` COMMITTING-case code samples, the Error Handling and Testing Strategy sections below — describes this now-removed mechanism.)** - **`cluster_run` unchanged**: Still just `cbuf_get(&wbuf)` for stop signal. - **`delaymgr_start` and `persist_start` move to `myapp_run`**: Currently called inside `cluster_start()`. In the new design `cluster_start` is not called from `myapp_run`. These two calls move explicitly into `myapp_run` before `ha_start_election_async`. - **`bootstrap_config_if_needed` removed entirely**: Replaced by the freshness gate in voting + primary replication push. No pre-election config probe needed. @@ -152,16 +152,28 @@ src/controller/ ├── ha.h # Add to ha_state_t: client_ssl_ctx, phase2_done, repl_ack_count, n_peers │ # Add to ha_config_t: committing_timeout_ms (default 10000ms) │ # Add CTRL_PRIMARY_COMMITTING to ctrl_role_t enum +│ # (Note: repl_ack_count, committing_timeout_ms, and CTRL_PRIMARY_COMMITTING +│ # do not exist in current ha.h — superseded by ha-single-leader-role, +│ # see that spec's design.md. client_ssl_ctx, phase2_done, n_peers verified current.) │ # Declare: ha_start_election_async(), ha_election_cleanup(), │ # ha_repl_ack_received(), ha_arm/disarm_committing_timeout(), │ # ha_send_ctrlhello_to_all_standbys() +│ # (Note: ha_repl_ack_received/ha_arm/disarm_committing_timeout do not +│ # exist — superseded by ha-single-leader-role. ha_send_ctrlhello_to_all_standbys +│ # verified current, called from raft_cb_become_leader.) ├── ha.c # Remove: vote_request_thread, vote_thread_ctx_t │ # Add: vote_ep_ctx_t, on_vote_connected(), vote_reply_msg_handler() +│ # (Note: vote_reply_msg_handler was generalized into peer_ctrl_msg_handler — +│ # superseded by ha-prevote, see that spec's design.md) │ # Add: ha_start_election_async(), election_timeout_watchdog │ # Extend ha_elect_evaluate_and_signal: COMMITTING path + replication push │ # Extend handle_request_vote: role-based deny gate +│ # (Note: handle_request_vote is now #ifdef XCE_TESTING-only — superseded +│ # by raft-adapter-conformance, see that spec's design.md) │ # Add: ha_repl_ack_received(), ha_arm/disarm_committing_timeout() │ # Extend ha_dispatch_gate: CTRL_PRIMARY_COMMITTING case +│ # (Note: none of these three exist — superseded by ha-single-leader-role; +│ # see the CTRL_PRIMARY_COMMITTING note in Key Decisions above) ├── cluster.c # cluster_init_phase2: vtask-safe (managers already in phase1) │ # ha_send_ctrlhello_to_all_standbys() or reuse handle_ctrlhello path │ # cluster_run: UNCHANGED @@ -259,33 +271,42 @@ sequenceDiagram | 2.2 | TLS client context | `client_ssl_ctx` in `ha_state_t` | 3-node startup | | 2.3 | Send vote after handshake | `on_vote_connected` callback | 3-node startup | | 2.4 | Connect/TLS failure → deny | `vote_ep_err_handler` | — | -| 3.1 | Reply via iomgr packet | `vote_reply_msg_handler` | 3-node startup | +| 3.1 | Reply via iomgr packet | `vote_reply_msg_handler`§ | 3-node startup | | 3.2 | No capnp UAF | vtask handler reads before free | 3-node startup | -| 3.3 | Extract reply fields | `vote_reply_msg_handler` decodes | 3-node startup | +| 3.3 | Extract reply fields | `vote_reply_msg_handler`§ decodes | 3-node startup | | 3.4 | Term fencing on reply | `handle_request_vote_reply` (existing) | — | -| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING | `ha_elect_evaluate_and_signal` → committing state | 3-node startup | -| 4.2 | Freshness gate guarantees ground-truth | `handle_request_vote` config_tag check | — | +| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING† | `ha_elect_evaluate_and_signal` → committing state† | 3-node startup | +| 4.2 | Freshness gate guarantees ground-truth | `handle_request_vote` config_tag check‡ | — | | 4.3 | Cold start: only highest config_tag wins | freshness gate + role gate combined | — | | 4.4–4.7 | Retry, timeout, heartbeat abort | watchdog, re-arm, heartbeat watchdog | Primary crash | -| 5.1 | ELECTING gate: NOT_READY for non-election | `ha_dispatch_gate` (existing + COMMITTING) | NOT_READY flow | -| 5.2 | COMMITTING: only repl acks allowed | `ha_dispatch_gate` new COMMITTING case | 3-node startup | +| 5.1 | ELECTING gate: NOT_READY for non-election | `ha_dispatch_gate` (existing + COMMITTING)† | NOT_READY flow | +| 5.2 | COMMITTING: only repl acks allowed | `ha_dispatch_gate` new COMMITTING case† | 3-node startup | | 5.3 | PRIMARY: full traffic | `ha_dispatch_gate` (existing) | — | | 5.4–5.5 | STANDBY gate, term persistence | `ha_dispatch_gate`, `ha_persist_term` (existing) | — | -| 6.1 | PRIMARY denies all votes | `handle_request_vote` role gate | — | -| 6.2 | STANDBY+fresh hb denies votes | `handle_request_vote` heartbeat check | — | +| 6.1 | PRIMARY denies all votes | `handle_request_vote` role gate‡ | — | +| 6.2 | STANDBY+fresh hb denies votes | `handle_request_vote` heartbeat check‡ | — | | 6.3 | HB timeout → ELECTING | heartbeat watchdog fires RDHUP path | Primary crash | -| 6.4 | INIT/ELECTING evaluates Raft gates | `handle_request_vote` existing gates | — | -| 6.5 | Primary crash → new primary | HB timeout → re-election → COMMITTING → PRIMARY | Primary crash | +| 6.4 | INIT/ELECTING evaluates Raft gates | `handle_request_vote` existing gates‡ | — | +| 6.5 | Primary crash → new primary | HB timeout → re-election → COMMITTING → PRIMARY† | Primary crash | | 7.1 | Winner pushes config to standbys | `push_config_to_standbys()` in committing path | 3-node startup | | 7.2 | Winner replicates WAL to standbys | `handle_ctrlhello()` catch-up (existing) | 3-node startup | | 7.3 | config_tag=0 standby gets config via push | no independent fetch; primary pushes | 3-node startup | -| 7.4 | Ack tracked per standby | `ha_state_t.repl_ack_count` | 3-node startup | -| 8.1 | COMMITTING blocks job traffic | `ha_dispatch_gate` COMMITTING case | 3-node startup | -| 8.2 | Majority ack → CTRL_PRIMARY | `ha_repl_ack_received()` threshold check | 3-node startup | +| 7.4 | Ack tracked per standby | `ha_state_t.repl_ack_count`† | 3-node startup | +| 8.1 | COMMITTING blocks job traffic | `ha_dispatch_gate` COMMITTING case† | 3-node startup | +| 8.2 | Majority ack → CTRL_PRIMARY | `ha_repl_ack_received()` threshold check† | 3-node startup | | 8.3 | CTRL_PRIMARY opens full traffic | `ha_dispatch_gate` PRIMARY case (existing) | 3-node startup | -| 8.4 | Ack timeout → degraded open | timeout watchdog in committing state | — | +| 8.4 | Ack timeout → degraded open | timeout watchdog in committing state† | — | | 9.1–9.4 | Regressions | Single-node, ground-truth invariant, RDHUP hooks-only, priority stagger | — | +† Rows marked † describe the `CTRL_PRIMARY_COMMITTING` mechanism, removed by ha-single-leader-role — +see the Key Decisions note above (a winner now enters `CTRL_PRIMARY` directly; job-traffic admission +during startup is gated by `ha_has_write_quorum()` inside `ha_dispatch_gate`'s `CTRL_PRIMARY` case). +‡ Rows marked ‡ cite `handle_request_vote`, now `#ifdef XCE_TESTING`-only; production vote-request +handling is `cluster.c`'s `case OPCode_requestVote:` → `raft_on_vote_request()` (superseded by +raft-adapter-conformance, see that spec's design.md). +§ Rows marked § cite `vote_reply_msg_handler`, generalized into `peer_ctrl_msg_handler` (superseded +by ha-prevote, see that spec's design.md). + ## Components and Interfaces ### Summary @@ -296,18 +317,30 @@ sequenceDiagram | `ha_state_t.client_ssl_ctx` | ha.h/myapp.c | TLS client context | 2.2 | | `vote_ep_ctx_t` | ha.c | Per-peer context for callbacks | 1.1, 2.1–2.4 | | `on_vote_connected()` | ha.c | Send vote RPC after handshake | 2.3, 3.1 | -| `vote_reply_msg_handler()` | ha.c | Decode reply; update tally | 3.1–3.4 | -| `ha_start_election_async()` | ha.c | Non-blocking: connect peers + arm watchdog | 1.1–1.3, 4.1–4.3 | +| `vote_reply_msg_handler()`§ | ha.c | Decode reply; update tally | 3.1–3.4 | +| `ha_start_election_async()`¶ | ha.c | Non-blocking: connect peers + arm watchdog | 1.1–1.3, 4.1–4.3 | | Election timeout watchdog | ha.c | Fire evaluate if deadline expires | 1.3, 4.2, 4.3 | -| `handle_request_vote` role gate | ha.c | Deny votes from PRIMARY/healthy-STANDBY | 6.1–6.4 | -| `ha_elect_evaluate_and_signal` extended | ha.c | Enter COMMITTING; push config+WAL; set role | 4.1, 7.1–7.4 | -| `CTRL_PRIMARY_COMMITTING` state | ha.h + ha_dispatch_gate | Block job traffic; allow repl acks | 5.2, 8.1 | -| `ha_repl_ack_received()` | ha.c | Count standby acks; transition to PRIMARY on quorum | 8.2–8.3 | -| `ha_state_t.repl_ack_count` | ha.h | Atomic ack counter per election round | 7.4, 8.2 | +| `handle_request_vote` role gate‡ | ha.c | Deny votes from PRIMARY/healthy-STANDBY | 6.1–6.4 | +| `ha_elect_evaluate_and_signal` extended | ha.c | Enter COMMITTING; push config+WAL; set role† | 4.1, 7.1–7.4 | +| `CTRL_PRIMARY_COMMITTING` state† | ha.h + ha_dispatch_gate | Block job traffic; allow repl acks | 5.2, 8.1 | +| `ha_repl_ack_received()`† | ha.c | Count standby acks; transition to PRIMARY on quorum | 8.2–8.3 | +| `ha_state_t.repl_ack_count`† | ha.h | Atomic ack counter per election round | 7.4, 8.2 | + +† = superseded by ha-single-leader-role (CTRL_PRIMARY_COMMITTING removed; see Key Decisions note above). +‡ = `handle_request_vote` is now test-only; production path is `raft_on_vote_request()` (raft-adapter-conformance). +§ = renamed/generalized to `peer_ctrl_msg_handler` (ha-prevote). +¶ = peer connect + vote tally now delegated to `raft_start_election()`/`raft_on_vote_reply()` (raft-log-replication, raft-commit-safety). ### Key Interfaces #### `ha_start_election_async()` + +**(Note: superseded by raft-log-replication and raft-commit-safety, see those specs' design.md. This +function no longer connects vote endpoints or tallies votes directly — it delegates to +`raft_start_election()`, which fires `raft_cb_send_vote_request` per peer and, on a terminal +result, `raft_cb_become_leader`/`raft_cb_become_follower`. The signature and non-blocking contract +below remain accurate.)** + ```c /* Non-blocking. Arms heartbeat watchdog + election watchdog. * Connects outgoing vote endpoints. On terminal state: @@ -323,6 +356,11 @@ void ha_start_election_async(ha_state_t *ha, #### `ha_elect_evaluate_and_signal` (extended) +**(Note: superseded by ha-single-leader-role, see that spec's design.md. The COMMITTING transition +below — `CTRL_PRIMARY_COMMITTING`, `repl_ack_count`, `ha_arm_committing_timeout` — no longer exists; +a winning candidate is set to `CTRL_PRIMARY` directly. The `is_first_boot`/`phase2_done` gating and +the STANDBY branch below remain accurate and current.)** + ```c /* After election determines role: */ int is_first_boot = !ha->phase2_done; /* phase2_done set after first cluster_init_phase2 call */ @@ -352,6 +390,11 @@ if (result == CTRL_PRIMARY) { #### `ha_repl_ack_received()` — called when a standby acks config+WAL catch-up +**(Note: this function does not exist in current code — superseded by ha-single-leader-role, see +that spec's design.md. Standby ack counting toward promotion was removed along with +`CTRL_PRIMARY_COMMITTING`; job-traffic admission is now `ha_has_write_quorum()`'s live count of +connected standbys, evaluated per-request inside `ha_dispatch_gate`, not a one-time per-round tally.)** + ```c void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) { int acks = atomic_fetch_add(&ha->repl_ack_count, 1) + 1; @@ -366,6 +409,14 @@ void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) { Called from the existing `handle_ctrlhello` reply path (CtrlHelloReply ack from standby after catch-up) and from `push_config_to_standbys` ack handler. #### `handle_request_vote` role gate (extended) + +**(Note: superseded by raft-adapter-conformance, see that spec's design.md. `handle_request_vote` +is now `#ifdef XCE_TESTING`-only, kept as a test harness entry point; production vote-request +dispatch is `cluster.c`'s `case OPCode_requestVote:`, which applies its own freshness gate and then +calls `raft_on_vote_request()` — the PRIMARY-role deny is now `on_vote_eligible`/`RAFT_LEADER` +inside the raft library, not this snippet. `CTRL_PRIMARY_COMMITTING` in the snippet below also no +longer exists — see the Key Decisions note above.)** + ```c ctrl_role_t my_role = atomic_load_explicit(&ha->role, memory_order_acquire); if (my_role == CTRL_PRIMARY || my_role == CTRL_PRIMARY_COMMITTING) @@ -376,11 +427,20 @@ if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) ``` #### `ha_heartbeat_is_fresh()` + +**(Fixed: signature and field name were stale — corrected to match current `ha.c`. The function is +non-static (exported) and reads `ha->last_activity`, a whole-second `time_t`, not a +`struct timespec last_heartbeat` — the design's original nsec-precision sketch was implemented as +whole-second precision, and it now null-guards `last_activity == 0` for "no primary message seen yet".)** + ```c -static int ha_heartbeat_is_fresh(const ha_state_t *ha) { - struct timespec now; clock_gettime(CLOCK_MONOTONIC, &now); - long ms = (now.tv_sec - ha->last_heartbeat.tv_sec) * 1000 - + (now.tv_nsec - ha->last_heartbeat.tv_nsec) / 1000000; +int ha_heartbeat_is_fresh(ha_state_t *ha) { + time_t last = ha->last_activity; /* time_t load is atomic on LP64 */ + if (last == 0) + return 0; /* no primary message seen yet */ + struct timespec now; + clock_gettime(CLOCK_MONOTONIC, &now); + long ms = (long)(now.tv_sec - last) * 1000L; return ms < (long)ha->config.heartbeat_timeout_ms; } ``` @@ -409,11 +469,19 @@ int myapp_run(myconf_t *myconf) { ``` **Note**: `vote_ep_ctx_t` carries `cluster_t *c` so that `ha_elect_evaluate_and_signal` -(called from vtask context via `vote_reply_msg_handler`) has access to `cluster_t` for +(called from vtask context via `vote_reply_msg_handler`§) has access to `cluster_t` for `cluster_init_phase2`, `push_config_to_standbys`, and `ha_send_ctrlhello_to_all_standbys`. +§ (renamed/generalized to `peer_ctrl_msg_handler` — superseded by ha-prevote.) #### `ha_dispatch_gate` — CTRL_PRIMARY_COMMITTING case +**(Note: superseded by ha-single-leader-role, see that spec's design.md. This `case` does not exist — +`ha_dispatch_gate`'s switch has only `CTRL_PRIMARY`, `CTRL_INIT`, `CTRL_ELECTING`, `CTRL_STANDBY`. +The write-admission gating this case describes now lives inside the `CTRL_PRIMARY` case itself, via +`ha_has_write_quorum()`: state-changing ops are refused with `XCE_ERR_NOT_READY` when the live +connected-standby count is below majority, while the same replication/read-only opcode allowlist +below — plus fact-bearing ops (`OPCode_notify`/`update`/`setpreason`/`clrpreason`) — stays admitted.)** + ```c case CTRL_PRIMARY_COMMITTING: /* Allow replication acks (config push ack, CtrlHelloReply, WAL ack) so @@ -437,13 +505,14 @@ case CTRL_PRIMARY_COMMITTING: `OPCode_replCatchUpDone` handler in `cluster.c` calls `ha_repl_ack_received(ha, c)` to increment the ack counter and conditionally transition to `CTRL_PRIMARY`. +(Note: `ha_repl_ack_received` does not exist — see notes above; superseded by ha-single-leader-role.) ## Error Handling - **Connect failure**: endpoint skipped; peer absent from election. After watchdog timeout, election evaluates with available votes only. - **SSL handshake failure**: `vote_ep_err_handler` calls `handle_request_vote_reply(grant=0)`. - **`cluster_init_phase2` failure from vtask**: logs ERROR; posts stop to wbuf → clean shutdown. -- **Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING**: `ha_dispatch_gate` returns `XCE_ERR_NOT_READY`; client retries. +- **Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING**: `ha_dispatch_gate` returns `XCE_ERR_NOT_READY`; client retries. (Note: `CTRL_PRIMARY_COMMITTING` no longer exists — superseded by ha-single-leader-role; the equivalent current behavior is `ha_dispatch_gate`'s `CTRL_PRIMARY` case refusing state-changing ops via `ha_has_write_quorum()` until majority standbys are connected.) - **Committing timeout (Req 8.4)**: if majority ack not received within `committing_timeout_ms`, watchdog fires `ha_promote_to_primary` anyway (degraded mode) with a WARN log. Prevents deadlock when standbys are unavailable at startup. - **Vote re-arm on STANDBY**: `ha_elect_evaluate_and_signal` re-arms `ha_start_election_async`. Heartbeat watchdog fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING → re-election. diff --git a/.kiro/specs/ha-commit-barrier-rollout/design.md b/.kiro/specs/ha-commit-barrier-rollout/design.md index 122b972e..aa3e7e26 100644 --- a/.kiro/specs/ha-commit-barrier-rollout/design.md +++ b/.kiro/specs/ha-commit-barrier-rollout/design.md @@ -8,10 +8,10 @@ **Users**: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed. -**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:921-934), `do_queuecontrol_op` (queue.c:119-132), `do_hostcontrol_op` (worker.c:689-702), and `handle_worker_join` (join.c:33-119) to commit-first-then-reply; admits fact-bearing opcodes through the dispatch gate during quorum loss (ha.c:1527); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:866-930) to the `upgradeConfig` requester (config_repl.c:1057-1094); adds state-machine guards to `mark_job_*` (job.c:583/638/696/726); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:386-462) while hardening its configAck branch. +**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:2198-2224), `do_queuecontrol_op` (queue.c:128-154), `do_hostcontrol_op` (worker.c:796-822), and `handle_worker_join` (join.c:48-149) to commit-first-then-reply [line numbers re-verified against current source — task-evidence line numbers had drifted]; admits fact-bearing opcodes through the dispatch gate during quorum loss (`ha_dispatch_gate`, ha.c:4525, allowlist at ha.c:4644-4648); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:1609-1654) to the `upgradeConfig` requester (`do_upgrade_config_with_bytes`, config_repl.c:1696-1855 — Note: as of ha-term-safety, the requester's commit gate is `config_commit_change`'s WAL_CONFIG_MARKER barrier, which internally calls `push_config_to_standbys`; see Config Barrier section below); adds state-machine guards to `mark_job_*` (job.c: `mark_job_fail` 1830, `mark_job_update` 1929 — Note: `mark_job_start`/`mark_job_finish` were renamed and restructured by raft-apply-decoupling, see Idempotency Guards section below); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:589-700) while hardening its configAck branch. ### Goals -- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:495-581) is the reference implementation and stays unchanged +- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:1412 onward) is the reference implementation (Note: superseded by raft-apply-decoupling — `handle_jobsubmit` no longer "stays unchanged"; it was rearchitected to append via an async commit continuation (`jobsubmit_commit_continuation`/`jobsubmit_apply_complete`) rather than a synchronous append-then-reply, and job-array-single-wal-record added a sibling array-submit path. The commit-first-then-reply *outcome* for the requester is preserved, but the reference shape this design points to has changed; see those specs' design.md) - Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by `lsn > committed_lsn`, never dropped while running+leading, accepted even during quorum loss - An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback) - Config upgrade fails to the requester on missed majority and does not restart into the failed config @@ -22,7 +22,7 @@ - Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual) - Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery) - Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then) -- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands) +- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands — confirmed landed: `wal_classify` (persist.c:5796) now classifies WAL_NOOP as RETAIN exactly as pre-declared below) ## Boundary Commitments @@ -61,12 +61,12 @@ ## Architecture ### Existing Architecture Analysis -- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:1068-1088) and append. Only WAL_JOB_SUBMIT uses the sync barrier (persist.c:1107-1129); everything else is `persist_wal_async` fire-and-forget -- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:932), `do_queuecontrol_op` (queue.c:130), `do_hostcontrol_op` (worker.c:700), `handle_worker_join` (join.c:109) — all violate umbrella 2.1 today. `handle_jobsubmit` (job.c:563-578) is the proven correct shape -- Queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:76-112, job.c:629-631/687-689); job-submit's hook runs after the job locks are released (job.c:553) — this asymmetry drives WAIT placement (below) -- Facts arrive via `do_notify` (cluster.c:118-157, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (ha.c:1527) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing) -- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:306-438) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md -- `push_config_to_standbys` (config_repl.c:866-930) calls `repl_wait_majority_ack(configAck)` at 916-917 but only logs the result and returns the push count; the caller (config_repl.c:1057) ACKs the admin tool with status=0 and restarts regardless. The configAck counting (repl.c:418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers` +- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:6509/6589/6638 — re-verified, was 1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:5769-5789, was 1068-1088) and append. WAL_JOB_SUBMIT is now committed via an explicit call from `handle_jobsubmit` (`persist_submit_job_payloads_and_commit`) strictly before any job object is constructed, never from this hook — the hook's WAL_MODE_WAIT branch is now a pure post-commit no-op (Note: superseded by raft-apply-decoupling task 3.1; the "in-hook sync barrier" description below and elsewhere in this doc no longer matches — see that spec's design.md). Everything else still routes through `wal_hook_append_classified` (append + RETAIN registration), functionally equivalent to the `persist_wal_async` fire-and-forget this doc describes +- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:2198), `do_queuecontrol_op` (queue.c:128), `do_hostcontrol_op` (worker.c:796), `handle_worker_join` (join.c:48) — all violate umbrella 2.1 today [historical framing — this spec fixes it; see Reply-Ordering Sweep]. `handle_jobsubmit` (job.c:1412) is the proven correct shape at the time this spec closed (Note: superseded by raft-apply-decoupling — see Goals section note above) +- Queue/worker/job-state hooks fire while the entity WRITE lock is held; job-submit's hook now fires post-commit outside any entity lock as a pure notification (raft-apply-decoupling task 3.1) — this asymmetry still drives WAIT placement (below), though the job-submit half of the asymmetry is now "no hook work at all" rather than "hook runs lock-free" +- Facts arrive via `do_notify` (cluster.c, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (`ha_dispatch_gate`, ha.c:4525) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing; fixed by this spec, see Fact Gate Admission) +- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:957) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md +- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) calls `repl_wait_majority_ack(configAck)` but only logs the result and returns the push count; the caller ACKed the admin tool with status=0 and restarted regardless [historical framing — this spec fixes it]. The configAck counting (repl.c, was 418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers` [historical — fixed, see Config Barrier] ### Architecture Pattern & Boundary Map @@ -143,16 +143,17 @@ src/include/repl.h # MODIFIED — configAck evaluation reuse ``` ### Modified Files (what and why) +[Line citations below re-verified against current source (was accurate as of this spec's 2026-06-13 close; substantial drift since from file growth — see per-line notes)] - `src/include/persist.h` — declares `wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }`, `wal_commit_mode_t wal_classify(uint32_t wal_type, int detail)`, `int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)` -- `src/controller/persist.c` — `wal_classify` table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (snapshot `current_lsn`, `commit_submit(WAIT)`) -- `src/controller/job.c` — `do_jobcontrol_op` (job.c:921-934): barrier between `job_ctrl` success and the reply. `mark_job_start/finish/fail/update` (job.c:583/638/696/726): transition guards before mutation -- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:119-132): barrier between `queue_ctrl` success and the reply -- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:689-702): barrier between `worker_ctrl` success and the reply -- `src/controller/join.c` — `handle_worker_join` (join.c:33-119): barrier between the AFTER_EVENT hook and the reply at join.c:107-109; join failure (idx < 0) keeps replying immediately -- `src/controller/ha.c` — `ha_dispatch_gate` PRIMARY no-quorum allowlist (ha.c:1537-1556): add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` -- `src/controller/config_repl.c` — `push_config_to_standbys` returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1057): on failure send error reply, skip restart -- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack`: legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds -- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c` +- `src/controller/persist.c` — `wal_classify` (persist.c:5796) table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (persist.c:5875 — snapshot `current_lsn`, `commit_submit(WAIT)`). Note: the table has since grown two rows this design never described — `WAL_JOB_ARRAY_SUBMIT` (WAIT, added by job-array-single-wal-record) and `WAL_CONFIG_MARKER` (RETAIN, added by ha-term-safety task 4.2) — see both specs' design.md +- `src/controller/job.c` — `do_jobcontrol_op` (job.c:2198-2224, was 921-934): barrier between `job_ctrl` success and the reply. `mark_job_fail`/`mark_job_update` (job.c:1830/1929, was 696/726): transition guards before mutation. `mark_job_start`/`mark_job_finish` (was job.c:583/638) no longer exist under those names — renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734) with the guard+mutation split across receipt time and apply time (Note: superseded by raft-apply-decoupling, see Idempotency Guards below and that spec's design.md) +- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:128-154, was 119-132): barrier between `queue_ctrl` success and the reply +- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:796-822, was 689-702): barrier between `worker_ctrl` success and the reply +- `src/controller/join.c` — `handle_worker_join` (join.c:48-149, was 33-119): barrier between the AFTER_EVENT hook and the reply at join.c:128-135 (was 107-109); join failure (idx < 0) keeps replying immediately +- `src/controller/ha.c` — `ha_dispatch_gate` (ha.c:4525, was implicitly ~1527) PRIMARY no-quorum allowlist (ha.c:4644-4648, was 1537-1556): adds `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` — confirmed present in current source. The gate has since grown two additional fence layers checked before this allowlist (`config_serve_fenced`, `upgrade_materialization_fenced` — ha-term-safety / config-upgrade-snapshot work); those fences do NOT include the fact opcodes in their own allowlists, so a config-fenced primary rejects facts too — a narrower, config-specific case this design did not anticipate and does not contradict (quorum-loss admission, this spec's concern, is unaffected) +- `src/controller/config_repl.c` — `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1057): on failure send error reply, skip restart. Note: superseded by ha-term-safety task 4.2 — the direct caller of `push_config_to_standbys` is now `config_commit_change` (config_repl.c:535), which `do_upgrade_config_with_bytes` calls instead; the requester's success/failure now gates on a WAL_CONFIG_MARKER consensus record reaching the committed prefix, with `push_config_to_standbys`'s majority-ACK wait as the out-of-band byte-delivery quorum underneath it. See Config Barrier section and ha-term-safety's design.md +- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack` (repl.c:589-700, was 386-462): legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds — confirmed unchanged in current source +- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c` (confirmed present) ## System Flows @@ -190,6 +191,8 @@ Gate admits the opcode even without write quorum (3.2) → handler applies the t ### Configuration upgrade +> **Note: the diagram and "request restart" step below are superseded by ha-term-safety (task 4.2), see that spec's design.md.** Current source inserts `config_commit_change` (config_repl.c:535, appending a WAL_CONFIG_MARKER) between `do_upgrade_config_with_bytes` and `push_config_to_standbys`, and no restart is requested on success — materialization now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the committed marker. The majority-ACK shape below (push → parallel threads → per-standby ack → majority-of-n_peers) is otherwise still accurate as the mechanism `push_config_to_standbys` uses underneath the marker. + ```mermaid sequenceDiagram participant A as admin tool @@ -214,7 +217,7 @@ sequenceDiagram end ``` -Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart is gated on success (4.4 — closes the roadmap residual "don't restart before the commit barrier passes"); config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time). +Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved, now framed by ha-term-safety as a non-authoritative cache of the marker-derived committed config); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart was originally gated on success (4.4 — closed the roadmap residual "don't restart before the commit barrier passes") but ha-term-safety later removed the restart step entirely in favor of marker-driven materialization on every member — see the supersession note above; config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time). ## Requirements Traceability @@ -267,7 +270,9 @@ Flow decisions: `committed_config_tag` advances only inside the majority branch | WAL_WORKER_CTRL, detail JOIN/UPDATE | handle_worker_join, do_hostcontrol_op | request | WAIT (handler barrier) | error reply to worker/client | | WAL_WORKER_CTRL, detail DEAD | controller-detected (worker.c:565/578) | fact | RETAIN | none — retained until committed | | WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON | worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier) | fact | RETAIN | none — retained until committed | -| WAL_NOOP (future) | promotion (ha-single-leader-role) | internal | RETAIN | added downstream; guard forces classification | +| WAL_NOOP | promotion (ha-single-leader-role) | internal | RETAIN | landed and confirmed classified exactly as pre-declared (persist.c:5835-5839) — no longer future | +| WAL_JOB_ARRAY_SUBMIT | handle_jobsubmit_array (client, job-array-single-wal-record) | request | WAIT (handler barrier, single barrier for the whole array) | error to client — row added by job-array-single-wal-record, not in this design's original table (Note: superseded/extended, see that spec's design.md) | +| WAL_CONFIG_MARKER | config_commit_change (ha-term-safety task 4.2) | request-adjacent consensus record | RETAIN | commits through the same barrier as job-state facts; the upgrade requester's own barrier is owned by `do_upgrade_config_with_bytes`, not this row — row added by ha-term-safety, not in this design's original table (Note: superseded/extended, see that spec's design.md and the Config Barrier section below) | | WAL_CHECKPOINT | local checkpoint marker | local-only | LOCAL | unchanged (never replicated, by design) | **Responsibilities & Constraints** @@ -338,13 +343,13 @@ int persist_commit_barrier(persist_t *p, uint32_t timeout_ms); | Requirements | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 | **Responsibilities & Constraints** -- Restructure pattern (uniform): `rc = (...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Applies to: - - `do_jobcontrol_op` (job.c:921-934) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately - - `do_queuecontrol_op` (queue.c:119-132) - - `do_hostcontrol_op` (worker.c:689-702) - - `handle_worker_join` (join.c:33-119) — barrier inserted between the AFTER_EVENT hook (join.c:104-105) and the success reply (join.c:107-109); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply -- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:459-491) — early reply at job.c:466-471 is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only -- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table is committed with the task evidence. Known control-path population at design time: job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c (gate/shutdown/hello/notify-path replies); config_repl.c:619-1129 (config/replication track); ha.c:1035; opget.c:222 (read-only); reply.c:61; task.c:97 — line numbers re-verified by grep during implementation +- Restructure pattern (uniform): `rc = (...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Confirmed still in place at (line numbers re-verified against current source, drifted from file growth since this spec's 2026-06-13 close — see File Structure Plan §Modified Files for the specific old→new lines): + - `do_jobcontrol_op` (job.c:2198-2224) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately + - `do_queuecontrol_op` (queue.c:128-154) + - `do_hostcontrol_op` (worker.c:796-822) + - `handle_worker_join` (join.c:48-149) — barrier inserted between the AFTER_EVENT hook (join.c:118-120) and the success reply (join.c:137-139); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply +- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:1119, was 459-491) — early reply is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:2330/2358, was 1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c:48/64/88) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only +- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table was committed with the task evidence. The design-time line-number snapshot originally recorded here (job.c/queue.c/worker.c/join.c/opupd.c/cluster.c/config_repl.c/ha.c/opget.c/reply.c/task.c line lists) has drifted wholesale with subsequent file growth across a dozen-plus later specs and is no longer independently meaningful as a citation list; the audit's *coverage claim* (every reply site classified, zero unclassified) is not re-verifiable line-by-line within this pass's scope — re-run the grep sweep against current source before relying on the exhaustiveness claim - A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4) **Contracts**: Service [x] — external reply semantics: success now implies majority durability; new error codes `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary) @@ -365,9 +370,12 @@ Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode | Requirements | 4.1, 4.2, 4.3, 4.4 | **Responsibilities & Constraints** -- `push_config_to_standbys` (config_repl.c:866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait (config_repl.c:916-917) passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (config_repl.c:918-922 — 4.3 preserved) -- configAck branch of `repl_wait_majority_ack` (repl.c:418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track"): count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine) -- Caller `do_upgrade_config_with_bytes` (config_repl.c:1056-1094): on non-zero return — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip `restart_requested` (4.4). Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary) + +> **Note: partially superseded by ha-term-safety (task 4.2), see that spec's design.md.** As of this spec's 2026-06-13 close, `do_upgrade_config_with_bytes` called `push_config_to_standbys` directly and its return code alone gated the requester's success/failure reply (described below, still architecturally accurate for `push_config_to_standbys` itself). ha-term-safety inserted a layer: `do_upgrade_config_with_bytes` now calls `config_commit_change` (config_repl.c:535), which appends a `WAL_CONFIG_MARKER` as the authoritative (term,lsn) consensus record and *internally* calls `push_config_to_standbys` as the out-of-band byte-delivery quorum underneath that marker. The requester's success/failure now gates on the marker reaching the committed prefix, not directly on `push_config_to_standbys`'s return. The majority-ACK arithmetic itself (below) is unchanged and still lives in `push_config_to_standbys`/`repl_wait_majority_ack`. + +- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — confirmed unchanged — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (4.3 preserved, now additionally described by ha-term-safety as a "non-authoritative cache" of the marker-derived committed config) +- configAck branch of `repl_wait_majority_ack` (repl.c:616-699, was 418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track") — confirmed unchanged: count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine) +- Caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1056-1094): on non-zero return from `config_commit_change` (was directly from `push_config_to_standbys`, see supersession note above) — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip restart. Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary). Materialization on success now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the marker — no restart is requested or required (a further ha-term-safety change beyond this spec's original "restart is gated on success" framing; see the Flow decisions paragraph below, now stale on this point) - One in-flight config change at a time (umbrella-locked) — no per-entry registry needed **Contracts**: Service [x] — `upgradeConfig` reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart @@ -385,7 +393,10 @@ Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode | Requirements | 5.1, 5.2, 5.3 | **Responsibilities & Constraints** -- Guard placement: at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire) + +> **Note: partially superseded by raft-apply-decoupling, see that spec's design.md.** `mark_job_fail`/`mark_job_update` (job.c:1830/1929) still exist essentially as described below — guard-then-mutate synchronously under the caller-held WRITE lock, used by the client-request path (`job_ctrl`). `mark_job_start`/`mark_job_finish` no longer exist under those names: they were renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734, worker-fact receipt path only) and split into two phases — a receipt-time "peek" of the identical guard (log-only, for the rejected-transition WARN; no mutation) plus a durable-append-only WAL write with no immediate mutation and no AFTER_EVENT hook, followed by the *actual* guarded mutation + hook firing later, deferred to `apply_committed_entry()` (via the extracted `apply_job_start_transition`/`apply_job_finish_transition` cores) once the entry reaches the committed prefix and the apply pump dispatches it. The guard *rules* (duplicate → no-op, backward → rejected+WARN) and their *outcome* are unchanged from what this design specifies below; the *timing* — guard-before-append vs. guard-at-apply-time — is not. + +- Guard placement (as designed; see supersession note above for the current split for START/FINISH): at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire) - Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers): - duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1) - backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return `-EALREADY`-style rejection, state preserved (5.2) diff --git a/.kiro/specs/ha-commit-durability/design.md b/.kiro/specs/ha-commit-durability/design.md index 5ed218c7..f5e03c21 100644 --- a/.kiro/specs/ha-commit-durability/design.md +++ b/.kiro/specs/ha-commit-durability/design.md @@ -2,7 +2,7 @@ ## Overview -**Purpose:** Make acknowledgment imply durability in the XCE HA controller. Today the controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution from a page-cache write (`commit.c:majority_candidate_locked` uses `highest_appended_lsn`), while background `fdatasync` runs on a timer (`fsync_worker`). A "committed" operation can therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover. +**Purpose:** Make acknowledgment imply durability in the XCE HA controller. Before this spec, the controller advanced its commit position on a majority of replication **acknowledgments**, but the primary counted its own contribution from a page-cache write (`commit.c:majority_candidate_locked` used `highest_appended_lsn`), while background `fdatasync` ran on a timer (`fsync_worker`). A "committed" operation could therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover. (Post-implementation, `majority_candidate_locked` caps the self-contribution at `persist_durable_lsn(ce->p)` — verified current in `src/controller/commit.c`.) **Users:** Cluster operators and every client whose acknowledged job/queue/worker operations must survive controller failover. @@ -53,12 +53,12 @@ ### Existing Architecture Analysis -- **Commit gate already off the iomgr thread:** the AFTER_EVENT hook "writes WAL and waits for majority replication" before the reply (`job.c:568-569`); `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)` blocks on a condvar. Adding `fdatasync` to this path is liveness-safe. -- **Standby already persists-before-ACK:** `persist_wal_write_replicated` `fdatasync`s before returning (`persist.c:515`). The primary is the asymmetric gap. -- **Recovery is silent-truncating:** `replay_wal` `break`s on CRC chain break / mismatch / short read (`persist.c:1021/1050/1042`) and returns the replayed count as success; `persist_recover` never sets `committed_lsn` (stays 0). -- **Promotion has no integrity gate:** `raft_cb_become_leader` computes `safe_lsn` from votes then `persist_truncate_wal_above(safe_lsn)` (`ha.c:572/577`) with no check that the local log is intact or ≥ `safe_lsn`. -- **Freshness floor is volatile:** `ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn)` (`ha.c:2127`), in-memory atomics not re-derived from durable WAL. -- **Test-seam convention:** `#ifdef XCE_TESTING` hook shims (`ha.c:2270-2335`); `wal_sync_ms` is a hard constant (`config.h:86`) with no override. +- **Commit gate already off the iomgr thread:** the AFTER_EVENT hook fires before the reply and `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)` blocks on a condvar (`job.c:2214`, `do_jobcontrol_op`; AFTER_EVENT hook itself e.g. `job.c:2189`). (Note: citation corrected — file has grown substantially since this spec closed; original cited `job.c:568-569`.) Adding `fdatasync` to this path is liveness-safe. +- **Standby already persists-before-ACK:** `persist_wal_write_replicated` `fdatasync`s before returning (`persist.c:1417`, function defined at `persist.c:1193`). The primary is the asymmetric gap. (Note: citation corrected from the original `persist.c:515`.) +- **Recovery is silent-truncating:** `replay_wal` (`persist.c:4624`) `break`s on CRC chain break / mismatch / short read (now around `persist.c:4704-4784`, several distinct break sites) and returns a verdict; `persist_recover` never sets `committed_lsn` (stays 0). (Note: citation corrected — original cited `persist.c:1021/1050/1042`; `replay_wal` has since moved substantially within the file.) +- **Promotion has no integrity gate:** `raft_cb_become_leader` (`ha.c:1615`) computes `safe_lsn` from votes (`ha.c:1641-1644`) then `persist_truncate_wal_above(safe_lsn)` (`ha.c:1662`) with no check that the local log is intact or ≥ `safe_lsn`. (Note: this describes the pre-spec baseline; citation corrected from the original `ha.c:572/577`. The gate this spec adds is now implemented immediately after, at `ha.c:1699-1822`.) +- **Freshness floor is volatile:** `ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn)` (now `ha.c:5310`, corrected from the original `ha.c:2127`), in-memory atomics not re-derived from durable WAL. +- **Test-seam convention:** `#ifdef XCE_TESTING` hook shims are now spread across dozens of sites in `ha.c` (the file has roughly doubled since this spec closed); `wal_sync_ms` was a hard constant (`config.h:86`, still `DEFAULT_WAL_SYNC_MS = 10`) with no override — this spec added the `--wal-sync-ms` override (confirmed live in `main.c`/`persist.h`). (Note: the original `ha.c:2270-2335` range no longer corresponds to a single representative hook-shim cluster.) ### Architecture Pattern & Boundary Map @@ -113,11 +113,11 @@ graph TB ## File Structure Plan ### Modified Files -- `src/controller/persist.c` — `persist_wal_async`: append then **signal the flusher** (no inline per-entry fsync); convert `fsync_worker` from a timer wait into a **demand-driven group-commit flusher** (wake on append-signal, `fdatasync`, advance `durable_lsn` to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in `persist_start`/`persist_stop`/`persist_fini`. `replay_wal`/`persist_recover`: return an integrity verdict; re-derive durable floor from the recovered WAL tip. `persist_wal_sync_append` (dormant, no production caller): delete or re-route through the durable-majority evaluation — it must not advance the commit position on majority ACK without the durable precondition (task 2.4). -- `src/include/persist.h` — add `_Atomic uint64_t durable_lsn` and a recovery-verdict type to `persist_t`/the recover signature; keep `fsync_*` fields (now driving the group-commit flusher); add an override field to `persist_config_t` (or its setter). -- `src/controller/commit.c` — `majority_candidate_locked`: use `durable_lsn` (not `highest_appended_lsn`) for the self contribution `cap`. -- `src/controller/ha.c` — `raft_cb_become_leader`: insert the pre-promotion eligibility gate; `ha_vote_self_freshness`: derive floor from durable WAL state; fence-release on catch-up completion. New `XCE_TESTING` durability-verification hook in the seam region (`ha.c:2270-2335`). -- `src/controller/cluster.c` — vote / PreVote freshness comparison (`cluster.c:380-396`, `467-474`) consume the durable-backed floor; promotion path honors the fence. +- `src/controller/persist.c` — `persist_wal_async`: append then **signal the flusher** (no inline per-entry fsync); convert `fsync_worker` from a timer wait into a **demand-driven group-commit flusher** (wake on append-signal, `fdatasync`, advance `durable_lsn` to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in `persist_start`/`persist_stop`/`persist_fini`. `replay_wal`/`persist_recover`: return an integrity verdict; re-derive durable floor from the recovered WAL tip. `persist_wal_sync_append`: **resolved (task 2.4)** — re-routed rather than deleted. It now calls the guarded `sync_append_legacy_commit_store` (`persist.c:5617`), which only advances `committed_lsn` when `written_lsn <= durable_lsn` (regression-pinned by `test_legacy_sync_append_holds_committed_at_durable`). (Note: current state corrected — as of this audit `persist_wal_sync_append` (`persist.c:5639`) DOES have a live production caller, `submit_job_wal_entry_and_commit` (`persist.c:5990-6004`, wired by `raft-apply-decoupling`), contradicting this section's original "dormant, no production caller" framing; the durable-precondition guard was added as designed, so the resolution is correct even though the "dormant" premise no longer holds.) +- `src/include/persist.h` — add `_Atomic uint64_t durable_lsn` and a recovery-verdict type to `persist_t`/the recover signature; keep `fsync_*` fields (now driving the group-commit flusher); add an override field to `persist_config_t` (or its setter). Confirmed current: `durable_lsn` at `persist.h:275`, `RECOVER_CLEAN`/`RECOVER_TRUNCATED` enum, `recover_decision_t`, `recovery_unrecoverable` flag all present as designed. +- `src/controller/commit.c` — `majority_candidate_locked`: use `durable_lsn` (not `highest_appended_lsn`) for the self contribution `cap`. Confirmed current (`commit.c:135`, `persist_durable_lsn(ce->p)`). +- `src/controller/ha.c` — `raft_cb_become_leader`: insert the pre-promotion eligibility gate; `ha_vote_self_freshness`: derive floor from durable WAL state; fence-release on catch-up completion. (Note: the durability-verification `XCE_TESTING` seam actually lives in `src/controller/persist.c` — `persist_test_set_force_fsync_fail` at `persist.c:1547`, `persist_test_durable_lsn_on_disk` at `persist.c:1610` — not in `ha.c` as this section originally stated.) +- `src/controller/cluster.c` — vote / PreVote freshness comparison consume the durable-backed floor; promotion path honors the fence. (Note: citations corrected — real-vote comparison now at `cluster.c:544-557`, PreVote at `cluster.c:643-651`; original cited `cluster.c:380-396`/`467-474`.) - `src/include/config.h` — keep `DEFAULT_WAL_SYNC_MS`; reconcile the stale `50` in `persistence-and-ha-design.md`. - `src/controller/main.c` — parse `--wal-sync-ms` (and optionally `--wal-sync-disable`) into `persist_config_t`. - `src/controller/myapp.c` / `cluster.c` (`persist_config_defaults` callers) — apply the override before `persist_init`. @@ -212,8 +212,8 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb - `persist_wal_async` appends and signals the flusher; it does **not** fsync inline per entry. - The **group-commit flusher** (repurposed `fsync_worker`) wakes on the append-signal, performs one `fdatasync`, advances `durable_lsn` to the highest synced LSN, and broadcasts the commit-eval condvar — amortizing one sync across all entries pending at wake time (the high-volume RETAIN stream benefits most). - `majority_candidate_locked` caps the self contribution at `durable_lsn`, not `highest_appended_lsn`. -- Standby persist-before-ACK is preserved and regression-pinned (`persist_wal_write_replicated:515`); the standby path may later adopt the same flusher (out of scope here). -- Checkpoint markers retain their existing inline `fdatasync` (`persist.c:573`). +- Standby persist-before-ACK is preserved and regression-pinned (`persist_wal_write_replicated`, defined at `persist.c:1193`, `fdatasync` call at `persist.c:1417`; note corrected from the original `persist.c:515`); the standby path may later adopt the same flusher (out of scope here). +- Checkpoint markers retain their existing inline `fdatasync` (`persist.c:1761`; note corrected from the original `persist.c:573`, which now falls inside unrelated term-index code added by `ha-term-safety`). - `durable_lsn` is monotonic, never exceeds the live segment tip; on `persist_truncate_wal_above`, clamp `durable_lsn` to the new tip. **Dependencies** @@ -307,7 +307,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb **Contracts**: State [x] **Implementation Notes** -- Integration: reuses `in_catch_up` (ha.c:954, cluster.c:529) and the PreVote/vote freshness comparison (cluster.c:380-396, 467-474); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter." +- Integration: reuses `in_catch_up` (`ha.c:917`, `cluster.c:713`; note corrected from the original `ha.c:954`/`cluster.c:529`) and the PreVote/vote freshness comparison (`cluster.c:544-557` real-vote, `cluster.c:643-651` PreVote; note corrected from the original `cluster.c:380-396`/`467-474`); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter." - Validation: simulate a node restarting truncated; assert it is denied promotion until catch-up, then re-enabled (Req 2.2/2.6). - Risks: interaction with `ha-single-leader-role` own-term gate and `ha-prevote` — re-verify the combined eligibility predicate against `raft.tla`. @@ -339,7 +339,7 @@ voter_floor_tag = max( local_tag(M), learned_tag(M) ) // unchanged ``` The voter floor is the **max of its own durable tip and its learned cluster-committed high-water**. This is the precise reconciliation of the durable-derived floor against the existing `max(applied_lsn, last_known_committed_lsn)`: -- `applied_lsn` is **replaced** by `durable_floor` (a node must not demand of a candidate a position it itself only holds volatilely; and symmetrically must not under-advertise — see §3/§4). Note `applied_lsn` and `durable_floor` use **different conventions and are not directly comparable as raw counters**: `applied_lsn` is advanced to `last_good_lsn + 1` (a *next-expected* index; `ha.c:1234/1253`), whereas `durable_floor`/`durable_lsn` is the actual durable LSN *watermark* (the highest LSN whose bytes are `fdatasync`'d). We therefore do **not** rely on a raw `durable_floor ≤ applied_lsn` inequality. The claim that this term never raises the floor beyond what is reachable rests entirely on the `D*` argument in §4: the voter floor's local-state contribution `durable_floor(v)` is bounded above by `D*` (the most-durable reachable candidate's tip) by definition of `D*`, so swapping in `durable_floor` cannot push the floor above some reachable candidate ⇒ cannot newly induce livelock (§4). +- `applied_lsn` is **replaced** by `durable_floor` (a node must not demand of a candidate a position it itself only holds volatilely; and symmetrically must not under-advertise — see §3/§4). Note `applied_lsn` and `durable_floor` use **different conventions and are not directly comparable as raw counters**: `applied_lsn` is advanced to `last_good_lsn` — the last contiguously-held LSN — via `ha_advance_applied_lsn` (`ha.c:3050`, called at `ha.c:3426-3432`), whereas `durable_floor`/`durable_lsn` is the actual durable LSN *watermark* (the highest LSN whose bytes are `fdatasync`'d). (Note: this audit found the convention itself has since changed — the code comment at `ha.c:3426-3429` cites a later fix, bug-991, that moved `applied_lsn` from `last_good_lsn + 1` [the *next-expected* index this section originally described, cited at the now-stale `ha.c:1234/1253`] to bare `last_good_lsn`, "not `last_good_lsn + 1`," because the `+1` wedged the leader's catch-up streamer. This does not affect the argument here — the two quantities were already established as not directly comparable, and the safety/liveness proof in §3/§4 does not rely on the exact `applied_lsn` convention.) We therefore do **not** rely on a raw `durable_floor ≤ applied_lsn` inequality. The claim that this term never raises the floor beyond what is reachable rests entirely on the `D*` argument in §4: the voter floor's local-state contribution `durable_floor(v)` is bounded above by `D*` (the most-durable reachable candidate's tip) by definition of `D*`, so swapping in `durable_floor` cannot push the floor above some reachable candidate ⇒ cannot newly induce livelock (§4). - `learned_committed` is **retained** as the safety backstop established by bug-339: a voter must never grant to a candidate below what the cluster has already **committed**, even if this voter's own durable tip is colder. Committed entries are now durable on a majority (Req 1/2.x), so at least a majority of voters carry that committed LSN in their `durable_floor` *and* every voter that learned the commit carries it in `learned_committed` — keeping the floor at the committed high-water on the voter side closes the window where a lagging voter would otherwise grant to a sub-committed candidate. **(c) The exact comparison (candidate ≥ voter floor — `logOk` analog):** @@ -354,16 +354,18 @@ i.e. deny iff `cand_tag < voter_floor_tag`, or `cand_tag == voter_floor_tag && c **Candidate and voter use DIFFERENT quantities — this is the load-bearing distinction, and the implementation must keep the two paths separate:** +> **Citation note (this audit):** the site line numbers below are corrected to their current locations; `ha.c` and `cluster.c` have grown substantially since this spec closed (2026-07-18) from later work (`ha-term-safety`, `raft-apply-decoupling`, `job-array-single-wal-record`, and unlisted in-flight specs). The mechanism itself is unchanged and verified current — a dedicated helper `ha_candidate_advertise_lsn(persist_t *p)` (`ha.c:5347-5349`, added since this spec's original text, which described inline snapshotting instead) now wraps the bare `persist_durable_floor(p)` call for both candidate sites. + | Role | Site(s) | Quantity it must produce | How | |------|---------|--------------------------|-----| -| **Candidate** (advertised `cand_lsn`) | `ha.c:3660-3666` (real-vote snapshot), `ha.c:3735-3741` (PreVote snapshot) | **bare `durable_floor(M)`** — NOT folded with `learned_committed` | snapshot `persist_durable_floor(p)` **directly** into `cand_lsn`; do **NOT** route the advertised LSN through `ha_vote_self_freshness` | -| **Voter** (freshness floor `voter_floor_lsn`) | `cluster.c:388-395` (real-vote), `cluster.c:471-475` (PreVote), `ha.c:2243` (legacy test gate) | **`max(durable_floor(M), learned_committed(M))`** — the backstop is retained | keep calling `ha_vote_self_freshness`, changing only its `local_lsn` input from `applied_lsn` to `durable_floor(M)`; the function still folds in `last_known_committed_lsn` | +| **Candidate** (advertised `cand_lsn`) | `ha.c:7938` (real-vote snapshot), `ha.c:8017` (PreVote snapshot) — corrected from the original `ha.c:3660-3666`/`3735-3741` | **bare `durable_floor(M)`** — NOT folded with `learned_committed` | calls `ha_candidate_advertise_lsn(&c->persist)` (`ha.c:5347-5349`), a thin wrapper over `persist_durable_floor(p)`; do **NOT** route the advertised LSN through `ha_vote_self_freshness` | +| **Voter** (freshness floor `voter_floor_lsn`) | `cluster.c:544-557` (real-vote), `cluster.c:643-651` (PreVote), `ha.c:5436` (`handle_request_vote`, the `XCE_TESTING`-only legacy test gate) — corrected from the original `cluster.c:388-395`/`471-475`/`ha.c:2243` | **`max(durable_floor(M), learned_committed(M))`** — the backstop is retained | keep calling `ha_vote_self_freshness`, changing only its `local_lsn` input from `applied_lsn` to `durable_floor(M)`; the function still folds in `last_known_committed_lsn` | -**Why the candidate must NOT reuse `ha_vote_self_freshness`.** That function (`ha.c:2127-2142`) *unconditionally* computes `*out_lsn = max(local_applied_lsn, last_known_committed_lsn)`. Feeding it `durable_floor` as the local input therefore yields `max(durable_floor, learned_committed)` — **not** the bare `durable_floor` §1a/§3 require. A truncated node with `durable_floor = 4` but a stale volatile `learned_committed = 8` would then advertise `cand_lsn = 8` and **win** — exactly the truncated-candidate-wins hole this task closes (§5a). So 4.2 must give the candidate sites a **candidate-only path** that calls `persist_durable_floor(p)` directly and assigns it to `cand_lsn`, bypassing the `learned_committed` fold. (The candidate's `cand_tag` is still `max(local_tag, learned_tag)`; only the **LSN axis** must bypass the fold.) The two PreVote/real-vote candidate snapshots currently call `ha_vote_self_freshness` for both fields — 4.2 splits the LSN field off onto the bare-`durable_floor` path while leaving the tag field on the existing max. +**Why the candidate must NOT reuse `ha_vote_self_freshness`.** That function (`ha.c:5310-5324`; note corrected from the original `ha.c:2127-2142`) *unconditionally* computes `*out_lsn = max(local_applied_lsn, last_known_committed_lsn)`. Feeding it `durable_floor` as the local input therefore yields `max(durable_floor, learned_committed)` — **not** the bare `durable_floor` §1a/§3 require. A truncated node with `durable_floor = 4` but a stale volatile `learned_committed = 8` would then advertise `cand_lsn = 8` and **win** — exactly the truncated-candidate-wins hole this task closes (§5a). So 4.2 must give the candidate sites a **candidate-only path** that calls `persist_durable_floor(p)` directly and assigns it to `cand_lsn`, bypassing the `learned_committed` fold. (The candidate's `cand_tag` is still `max(local_tag, learned_tag)`; only the **LSN axis** must bypass the fold.) The two PreVote/real-vote candidate snapshots currently call `ha_vote_self_freshness` for both fields — 4.2 splits the LSN field off onto the bare-`durable_floor` path while leaving the tag field on the existing max. **Why safety lives on the advertisement axis and the backstop lives on the voter axis.** The candidate must not **over-advertise** beyond durable state — that is the safety requirement (§3): pinning `cand_lsn = durable_floor` is what denies a truncated candidate. The `learned_committed` backstop belongs on the **voter** axis (§3 dual guard): a voter must refuse any candidate below the cluster-committed boundary it learned, even when the voter's own durable tip momentarily lags. Folding `learned_committed` into the *candidate* advertisement would defeat the very safety property it protects on the *voter* side; folding it into the *voter* floor is exactly the bug-339 backstop. raft.tla confirms the split: `RequestVote` sends `mlastLogIndex |-> Len(log[i])` — the **bare candidate log length**, with no committed term folded into the candidate's advertisement; the committed-high-water guard lives only on the voter's `logOk` floor. This recipe is now internally consistent with §1a (bare candidate `durable_floor`), §3 (quorum-intersection on the advertised `durable_floor`), and §4 (voter floor bounded by `D*`). -**(d) `safe_committed_lsn` is UNCHANGED and still from voter replies.** This floor governs only **vote granting / advertisement**. The post-election `safe_committed_lsn` is still computed by `ha_compute_safe_committed_lsn` from the `peerLastLogLsn` values in the `RequestVoteReply` set (`ha.c:480/562`), never from this floor and never from a stale ctrlhello (CLAUDE.md "Post-Election Safe Commit Point"). The reply's `peer_last_log_lsn` (`cluster.c:426`) now carries the durable-backed `voter_floor_lsn`, which is *more* conservative (never above a durable position), so the elected leader's `safe_committed_lsn` is if anything tightened, never loosened — `persist_truncate_wal_above(safe_committed_lsn)` still only removes entries strictly above a majority-backed point. +**(d) `safe_committed_lsn` is UNCHANGED and still from voter replies.** This floor governs only **vote granting / advertisement**. The post-election `safe_committed_lsn` is still computed by `ha_compute_safe_committed_lsn` (`ha.c:1144`; note corrected from the original `ha.c:480/562`) from the `peerLastLogLsn` values in the `RequestVoteReply` set, never from this floor and never from a stale ctrlhello (CLAUDE.md "Post-Election Safe Commit Point"). The reply's `peer_last_log_lsn` field (assigned at `cluster.c:596` real-vote, `cluster.c:735` PreVote; note corrected from the original `cluster.c:426`) now carries the durable-backed `voter_floor_lsn`, which is *more* conservative (never above a durable position), so the elected leader's `safe_committed_lsn` is if anything tightened, never loosened — `persist_truncate_wal_above(safe_committed_lsn)` still only removes entries strictly above a majority-backed point. #### 2. raft.tla alignment @@ -451,7 +453,7 @@ Walking the relevant items under the pinned formula: **Contracts**: State [x] **Implementation Notes** -- Integration: follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) and the `ha.c:2270-2335` hook convention. +- Integration: follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`, `main.c:32`) convention; the actual durability test-seam functions live in `src/controller/persist.c` (`persist_test_set_force_fsync_fail` at `persist.c:1547`, `persist_test_durable_lsn_on_disk` at `persist.c:1610`), not at the originally-cited `ha.c:2270-2335`. - Validation: deterministic E2E mode in `test-commit-durability.sh` drives the seam so the soak becomes a deterministic RED on the unfixed build and GREEN after. - Risks: ensure the seam is compiled out of production (`#ifdef XCE_TESTING`). @@ -493,4 +495,4 @@ Walking the relevant items under the pinned formula: - **Unrecoverable-terminal decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Concrete policy: fence at boot, attempt catch-up from a healthy primary, and on a failure to reach a committed-consistent state (no peer) enter a **LOUD PERMANENT FENCE** — non-promotable, stays alive, keeps retrying catch-up, **NEVER self-exit** (Req 2.2, 2.4, 2.6). **RESOLVED (task 3.2; terminal corrected by the never-self-exit rework 2026-06-18 — the original self-`abort` FATAL was replaced by the loud permanent fence):** see the **Boot Policy Decision Table (task 3.2)** above; tasks 3.3 (loud-fence terminal) and 4.3 (fence + release) implement it. - **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2). - **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never the unrecoverable terminal on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2). -- **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2). +- **Synchronous-append path — RESOLVED (task 2.4), re-verified current:** `persist_wal_sync_append` (`persist.c:5639`) no longer advances the commit position unguarded — it routes through `sync_append_legacy_commit_store` (`persist.c:5617`), which only stores past `committed_lsn` when `written_lsn <= durable_lsn`, closing the ACK-without-durability hole (Req 1.1, 1.2; regression-pinned by `test_legacy_sync_append_holds_committed_at_durable`). (Note: this audit found the path is NOT dormant as originally described here — it now has a live production caller, `submit_job_wal_entry_and_commit` (`persist.c:5990-6004`), introduced by `raft-apply-decoupling`. The guard makes this safe regardless, so no further fix is needed, but the "no production caller today" premise in this section's original text no longer holds.) diff --git a/.kiro/specs/ha-commit-engine/design.md b/.kiro/specs/ha-commit-engine/design.md index b743e4f6..366a0961 100644 --- a/.kiro/specs/ha-commit-engine/design.md +++ b/.kiro/specs/ha-commit-engine/design.md @@ -8,7 +8,7 @@ **Users**: Cluster operators gain the guarantee that a confirmed job submission is durable on a majority of members; downstream sub-specs (ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) gain the engine interface they build on. -**Impact**: Replaces the counting logic of `repl_wait_majority_ack` (repl.c:386-461) for WAL ACKs with per-LSN engine evaluation; removes the `n_active == 0 → success` short-circuit for multi-member clusters; moves the `committed_lsn` store out of `persist_wal_sync_append` (persist.c:1035) into the engine; fixes `ha_has_write_quorum` (ha.c:1507) to exclude catch-up standbys. Behavior of every other write path is unchanged in this spec. +**Impact**: Replaces the counting logic of `repl_wait_majority_ack` (repl.c:589, current) for WAL ACKs with per-LSN engine evaluation; removes the `n_active == 0 → success` short-circuit for multi-member clusters; moves the `committed_lsn` store out of `persist_wal_sync_append` (now persist.c:5639) into the engine; fixes `ha_has_write_quorum` (now ha.c:4495) to exclude catch-up standbys. Behavior of every other write path is unchanged in this spec. (Note: `persist_wal_sync_append` itself was later changed by ha-term-safety task 3.1 to route the primary's append through the raft library — `ha_primary_log_append` → `raft_log_append` — before this majority-commit barrier runs; see that spec's design.md.) ### Goals - Entry N commits only when `1 (self) + count(active standbys with acked_lsn >= N) >= n_peers/2 + 1` @@ -22,7 +22,7 @@ - Setting `writable_lsn` anywhere (own-term gate consumers are ha-single-leader-role; unset = no gate) - RETAIN-mode production consumers (facts roll out in ha-commit-barrier-rollout; the mode itself ships and is tested here) - Config-change barrier (`acked_config_lsn` track untouched), demotion/egress/catch-up changes, COMMITTING removal -- Async continuation of commit waits (umbrella-documented future work; blocking condvar + admission cap is the accepted mechanism) +- Async continuation of commit waits (umbrella-documented future work; blocking condvar + admission cap is the accepted mechanism) (Note: superseded by raft-apply-decoupling, which added a per-LSN continuation registry, `commit_resolve`/async registration, and a dedicated timeout-sweep thread to `commit_engine_t` — see that spec's design.md) ## Boundary Commitments @@ -44,7 +44,7 @@ - `standby_conn_t.acked_lsn` / `in_catch_up` per-standby tracking in `replicator_t` (src/include/repl.h) — read-only consumption; the engine never manages connections - `persist_t.committed_lsn` atomic (src/include/persist.h:71) — the engine is the only production writer after init/truncation - iomgr/vtaskmgr threading rules per `ha-async-io.md`: `commit_on_ack` and `commit_on_membership_change` are called on the iomgr thread and must never block on I/O or long waits (bounded mutex hold only); `commit_submit(WAIT)` blocks only on vtask threads -- Dependency direction (umbrella contract): `include/commit.h` depends only on libc/stdatomic types plus forward-declared `persist_t`; `commit.c` is called BY `persist.c` (append/sync path), `repl.c` (ACK + membership events), and `ha.c`/`cluster.c` (init, future promotion); it calls into none of them +- Dependency direction (umbrella contract): `include/commit.h` depends only on libc/stdatomic types plus forward-declared `persist_t`; `commit.c` is called BY `persist.c` (append/sync path), `repl.c` (ACK + membership events), and `ha.c`/`cluster.c` (init, future promotion); it calls into none of them (Note: superseded — `commit.h` now `#include`s the full `persist.h` and `vtaskmgr.h` rather than only forward-declaring, and `commit.c` now calls INTO `persist.c` (`persist_durable_lsn`, `persist_set_durable_cb`) to implement the durable-majority self-cap; see ha-commit-durability and raft-drive-async design.md) ### Revalidation Triggers - Any change to `commit.h` signatures or return-code contract → revalidate ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery @@ -55,10 +55,10 @@ ## Architecture ### Existing Architecture Analysis -- Single-writer model: only the primary appends WAL entries; per-standby `acked_lsn` (Raft matchIndex equivalent) already exists and is updated by `repl_on_ack` (repl.c:176-196) on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, ha.c:2739-2748) +- Single-writer model: only the primary appends WAL entries; per-standby `acked_lsn` (Raft matchIndex equivalent) already exists and is updated by `repl_on_ack` (now repl.c:250-285) on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, now ha.c:7249 → ha.c:6630 → repl.c:250) - ACKs are cumulative: a standby ACKs its highest contiguously applied LSN (ha-wal-replication contract), so per-standby `acked_lsn` is a watermark — engine evaluation needs no per-entry bitmap -- Defects replaced: `repl_wait_majority_ack` counts `acked_lsn > 0` (repl.c:419) — any ACK ever; `n_active == 0` short-circuits to success (repl.c:432); quorum denominator is `n_active` (connected standbys) rather than `n_peers` (configured members); `ha_has_write_quorum` (ha.c:1507-1526) counts `standby_endpoints->count`, which includes catch-up standbys -- `persist_wal_sync_append` (persist.c:1003-1049) is the only sync (blocking) write path — job submit via `persist_job_hook_fun` (persist.c:1125); it blocks a vtask pool thread (default 8, `--vcpus` override, cluster.c:1082); no liveness risk since wakeups come from iomgr +- Defects replaced: `repl_wait_majority_ack` counted `acked_lsn > 0` — any ACK ever; `n_active == 0` short-circuited to success; quorum denominator was `n_active` (connected standbys) rather than `n_peers` (configured members); `ha_has_write_quorum` counted `standby_endpoints->count`, which included catch-up standbys. (Note: these are pre-spec line citations that no longer resolve — the dishonest-counting branch was fully deleted, not just bypassed: `repl_wait_majority_ack` now unconditionally delegates the walAck branch to `commit_submit`, per ha-commit-barrier-rollout task 2.2. Current `ha_has_write_quorum` is at ha.c:4495-4523.) +- `persist_wal_sync_append` (now persist.c:5639) is the only sync (blocking) write path — job submit via `persist_job_hook_fun` (now persist.c:6509); it blocks a vtask pool thread (default 8, `--vcpus` override, sized in `cluster.c` ~1743-1745; `commit_engine_init` is wired at cluster.c:1820, not cluster.c:1082 as originally cited); no liveness risk since wakeups come from iomgr (plus, since ha-commit-durability, from the group-commit flusher's durable-advance callback — see below) - `replicator_t` is embedded in `persist_t` (`p->repl`), and `ha_state_t`/`cluster_t` reach it via `c->persist.repl` — the engine can reach the standby registry through its `persist_t*` ### Architecture Pattern & Boundary Map @@ -164,6 +164,8 @@ Flow decisions: the admission cap rejects before blocking when `inflight_waits > Under the engine mutex: collect `acked_lsn` of all standbys with `ep != NULL && !in_catch_up` into a scratch array, sort descending. With self counted, entry N has majority when at least `n_peers/2` active standbys have `acked_lsn >= N`; therefore the new commit candidate is `arr[n_peers/2 - 1]` (0-indexed; 0 candidates → no advance) clamped to the highest appended LSN. Store `committed_lsn = max(committed_lsn, candidate)` (2.2); cumulative ACK watermarks make the result contiguous by construction (2.3). Broadcast the condvar; each waiter re-checks its own predicate. Catch-up standbys are excluded from the count (umbrella-locked, conservative — their ACKs count after `repl_complete_catchup` triggers `commit_on_membership_change` re-evaluation). +(Note: superseded by ha-commit-durability, Req 1.2 (bug-661 fix) — the candidate is no longer clamped only to the highest *appended* LSN; self's contribution is additionally capped at `persist_durable_lsn(ce->p)`, the highest LSN fdatasync'd locally, so an entry that is only page-cache-resident cannot count self toward its own majority. See that spec's design.md for the durable-majority barrier.) + ## Requirements Traceability | Requirement | Summary | Components | Flows | @@ -202,10 +204,10 @@ Under the engine mutex: collect `acked_lsn` of all standbys with `ep != NULL && | Requirements | 1.1–1.3, 2.1–2.3, 4.1–4.4, 5.1, 6.3, 7.1–7.2 | **Responsibilities & Constraints** -- Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`; only the engine stores to it in production after init/truncation paths — the store at persist.c:1035 is deleted; the truncation store at persist.c:1438 remains) -- Commit rule (Raft §5.3, umbrella-verbatim): entry N is committed when `count(self + active standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.1–2.3) -- Own-term gate storage: `writable_lsn` defaults to 0 = no gate (this spec never sets it); when set (by ha-single-leader-role), no waiter completes successfully until `committed_lsn >= writable_lsn` -- Admission cap: at most `max(1, n_vcpus - COMMIT_WAIT_RESERVED)` (reserved default = 2) concurrent WAIT blockers; excess fails fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`) +- Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`, now declared persist.h:274; only the engine stores to it in production after init/truncation paths — the store formerly in `persist_wal_sync_append`'s engine-wired branch is deleted; truncation stores remain in `persist_truncate_wal_above`, now persist.c:6764 — note the field comment there records several more legitimate `committed_lsn` writers added since, e.g. `persist_committed_lsn_advance_atomic`, all monotonic-max) +- Commit rule (Raft §5.3, umbrella-verbatim): entry N is committed when `count(self + active standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.1–2.3) (Note: superseded in part by ha-commit-durability — self's contribution to the count is additionally capped at the local durable watermark, not just the appended tip; see `majority_candidate_locked` in commit.c) +- Own-term gate storage: `writable_lsn` defaults to 0 = no gate (this spec never sets it); when set (by ha-single-leader-role / ha-term-safety's `ha_arm_own_term_gate`), no waiter completes successfully until `committed_lsn >= writable_lsn` +- Admission cap: at most `max(1, n_vcpus - COMMIT_WAIT_RESERVED)` (reserved default = 2) concurrent WAIT blockers; excess fails fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`) (Note: superseded by raft-drive-async task 1.6 — this local cap is now only the fallback used when no `vtaskmgr_t` pool is wired (`commit_set_pool`); the production path folds admission onto the shared vtaskmgr pool's single LivenessAdmission reservation instead. See that spec's design.md.) - `n_peers <= 1`: self-ACK constitutes majority; WAIT returns 0 immediately and advances `committed_lsn` to the entry (6.3 — intentional change: standalone nodes now advance `committed_lsn` on sync appends, Raft-correct). For `n_peers > 1`, zero peer ACKs never commit (4.3, 4.4) **Dependencies** @@ -287,7 +289,7 @@ Summary (umbrella migration note: "`repl_wait_majority_ack` becomes a thin adapt ## Data Models ### Domain Model (changes only) -- **commit_engine_t** (in-memory only; commit state volatile, Raft-conformant — umbrella-locked): `{ pthread_mutex_t lock; pthread_cond_t cond; persist_t *p; int n_peers; int n_vcpus; int inflight_waits; uint64_t writable_lsn; uint64_t scratch[MAX_STANDBYS]; }` +- **commit_engine_t** (in-memory only; commit state volatile, Raft-conformant — umbrella-locked): `{ pthread_mutex_t lock; pthread_cond_t cond; persist_t *p; int n_peers; int n_vcpus; int inflight_waits; uint64_t writable_lsn; uint64_t scratch[MAX_STANDBYS]; }` (Note: superseded — the struct has since grown a `vtaskmgr_t *pool` field (raft-drive-async), a `uint64_t raft_commit_index` advisory field (ha-term-safety), and `commit_continuation_t *continuations`/`n_continuations`/`sweep_thread`/`sweep_stop` for async continuation resolution (raft-apply-decoupling). See `src/controller/include/commit.h` and those specs' design.md.) - **replicator_t**: + `struct commit_engine *ce` (NULL until wired) - **ha_state_t**: + `replicator_t *repl` (NULL in legacy test contexts) - **cluster_t**: + `commit_engine_t commit` diff --git a/.kiro/specs/ha-config-bootstrap/design.md b/.kiro/specs/ha-config-bootstrap/design.md index 0fb7a5dc..e908a4a2 100644 --- a/.kiro/specs/ha-config-bootstrap/design.md +++ b/.kiro/specs/ha-config-bootstrap/design.md @@ -30,7 +30,7 @@ This feature eliminates three critical safety gaps in the XCE controller's LSF-s ### This Spec Owns -- Controller election protocol: full replacement of `am_i_primary()` with `am_i_primary_quorum()` +- Controller election protocol: full replacement of `am_i_primary()` with `am_i_primary_quorum()` (Note: `am_i_primary_quorum()`'s blocking pthread-per-peer implementation was itself later removed — production election now runs through async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; superseded by raft-library and election-iomgr-refactor, see those specs' design.md) - Controller FSM: 4-state lifecycle (`CTRL_INIT`, `CTRL_ELECTING`, `CTRL_PRIMARY`, `CTRL_STANDBY`) and all state transitions - Pre-election startup sequence: listener binding, iomgr/vtaskmgr init before election - Config bootstrap: `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, deferred fetch after election @@ -57,7 +57,7 @@ This feature eliminates three critical safety gaps in the XCE controller's LSF-s - `iomgr` (epoll/`EPOLLIN`/`EPOLLRDHUP`): P0 — control channel and dispatch loop - `vtaskmgr`: P0 — `vtaskmgr_init(int nvcpus)` confirmed to take vcpus directly, no cluster_config dependency -- `endpoint_connect()` / `connect_ssl_raw()`: P0 — SSL connection setup; TCP connect phase is blocking; vote request connections require per-peer pthreads +- `endpoint_connect()` / `connect_ssl_raw()`: P0 — SSL connection setup; TCP connect phase is blocking; vote request connections require per-peer pthreads (Note: superseded by election-iomgr-refactor — vote requests now use async iomgr endpoints, not per-peer pthreads; see that spec's design.md) - `repl_add/remove/start_catchup/complete_catchup/needs_snapshot/on_ack/trim_lsn`: P0 — existing repl slot management reused unchanged - `do_one_config_fetch()`: P1 — wired on branch (`2aae45f`); deferred fetch depends on it - `xce_encode()` / `xce_decode()` / `xce_request()`: P0 — wire protocol encoding @@ -131,6 +131,8 @@ graph TB end ``` +**Note (election mechanism superseded)**: The diagrams above show this spec's original blocking pthread-per-peer `am_i_primary_quorum()` election path. That implementation was removed by election-iomgr-refactor (task 5.3) and replaced by async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; `handle_request_vote()` is now test-only, production vote dispatch is `cluster.c` → `raft_on_vote_request()`. The 4-state FSM, majority-commit, and control-channel shapes shown remain accurate. See election-iomgr-refactor and raft-library design.md for the current mechanism. + ### Technology Stack | Layer | Choice | Role | Notes | @@ -138,7 +140,7 @@ graph TB | Runtime | C gnu17 | All new code | No C++ | | Wire format | Cap'n Proto | 6 new opcodes in `controller.capnp` | Appended after existing @31 | | SSL | OpenSSL (existing) | Vote request and probe connections | Reuses `connect_ssl_raw()` | -| Consensus | Raft subset (custom) | Leader election + majority-commit | No external library; bounded scope justified by static membership and no snapshot protocol need | +| Consensus | Raft subset (custom) | Leader election + majority-commit | No external library; bounded scope justified by static membership and no snapshot protocol need (Note: a dedicated `src/lib/raft/` in-repo library was later extracted from this inline logic; superseded by raft-library, see that spec's design.md) | | IPC | iomgr epoll | `EPOLLIN` + `EPOLLRDHUP` on control channel | `EPOLLRDHUP` for sub-second failover | | Persistence | Flat binary (`ha-term.bin`) | Term + voted_for across restarts | Atomic write+rename (existing pattern) | | Build | Meson + Nix | capnp schema regeneration | `src/schema/controller.capnp` | @@ -321,12 +323,12 @@ Fallback trigger: heartbeat timeout vtask fires after `election_timeout_ms + ran | Requirement | Summary | Components | Key Interface | Flow | |-------------|---------|------------|---------------|------| | 1.1–1.4 | Listener + iomgr/vtaskmgr before election; dispatch gate for init/electing states | `cluster.c`, `ha.c`, `myapp.c` | `cluster_open_listener()`, vtaskmgr dispatch gate | Flow 1 | -| 2.1–2.12 | Quorum election: voted_for enforcement, freshness gate, split-vote retry, single-node shortcircuit | `ha.c` | `am_i_primary_quorum()`, `handle_request_vote()`, `OP_requestVote` | Flow 2 | +| 2.1–2.12 | Quorum election: voted_for enforcement, freshness gate, split-vote retry, single-node shortcircuit | `ha.c` | `am_i_primary_quorum()`, `handle_request_vote()` (Note: both superseded by `ha_start_election_async()`/raft library, `handle_request_vote()` now test-only — see election-iomgr-refactor), `OP_requestVote` | Flow 2 | | 3.1–3.5 | Pre-election config probe, deferred fetch, retry backoff | `config_repl.c`, `myapp.c` | `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, `OP_probeConfigTag` | Flow 1 | | 4.1–4.4 | --vcpus CLI, even-N rejection, cert pre-flight, ASAN fix | `main.c`, `ha.c` | CLI argp, `access()`, `SSL_CTX_free` | Flow 1 | | 5.1–5.8 | Majority-commit WAL: committed_lsn, broadcast, ack-wait, backpressure, truncation | `persist.c`, `repl.c`, `ha.c` | `repl_wait_majority_ack()`, `persist_truncate_wal_above()`, `OP_walAck` | Flow 3 | | 6.1–6.8 | Persistent control channel: EPOLLRDHUP failover, state sync, catch-up, three-tier discovery | `ha.c`, `cluster.c`, `repl.c` | `ha_find_and_connect_to_primary()`, extended `OP_ctrlHello`, `OP_snapshotTransfer`, `OP_walCatchupEntry` | Flow 4 | -| 7.1–7.6 | Client retry: backoff, redirect, non-retriable passthrough | `request.c` (libxce) | `xce_request_with_retry()`, `xce_retry_config_t`, redirect fields in `OP_requestVoteReply` | — | +| 7.1–7.6 | Client retry: backoff, redirect, non-retriable passthrough | `request.c` (libxce) | `xce_request_with_retry()` (Note: removed, superseded by `client_failover.c` StatusClassifier/EndpointRotator — see libxce-failover), `xce_retry_config_t`, redirect fields in `OP_requestVoteReply` | — | | 8.1–8.5 | Majority-commit config push, committed_config_tag, heartbeat piggybacking | `config_repl.c`, `repl.c` | `push_config_to_standbys()`, `OP_upgradeConfigAck` | Flow 3 analog | | 9.1–9.3 | Term + voted_for durability across restarts via ha-term.bin | `ha.c` | `ha_persist_term()`, `ha_init()` read path | — | | 10.1–10.5 | Structured logs for election outcome, bootstrap, WAL commit, config push, FSM transitions | All new code paths | `xcelog.h` INFO/WARN macros | All flows | @@ -365,6 +367,7 @@ Fallback trigger: heartbeat timeout vtask fires after `election_timeout_ms + ran - Single allocation for `ha_elect_ctx_t` via C99 flexible array — one `calloc`, one `free` - All role transitions via `atomic_store(&c->ha.role, ...)` — no lock held across transition - `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()` are deleted in this spec +- (Note: `am_i_primary_quorum()`'s production implementation and `handle_request_vote()`/`handle_request_vote_reply()` were themselves later removed/demoted to test-only by election-iomgr-refactor — production election is now `ha_start_election_async()` → raft library; see that spec's design.md) **New `ha_state_t` fields**: ```c @@ -431,7 +434,7 @@ case CTRL_PRIMARY: **Contracts**: Service [x] / State [x] **Implementation Notes** -- Risk: `endpoint_connect()` has a blocking TCP connect phase. Vote request sends require per-peer pthreads (not pure iomgr async). Election context `ha_elect_ctx_t` accumulates replies; threads post to `ctx->done` semaphore when quorum is reached or all peers replied. +- Risk: `endpoint_connect()` has a blocking TCP connect phase. Vote request sends require per-peer pthreads (not pure iomgr async). Election context `ha_elect_ctx_t` accumulates replies; threads post to `ctx->done` semaphore when quorum is reached or all peers replied. (Note: this risk was resolved by election-iomgr-refactor — vote sends are now pure async iomgr endpoints via `ha_start_election_async()`, not per-peer pthreads; `ha_elect_ctx_t` gained a refcount and several fields since, see ha-elect-ctx-lifecycle design.md) - Integration: `ha_wait_for_heartbeat_or_timeout()` uses `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + rand_jitter`. Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`. - `voted_for` enforcement: per-term, one vote per node. If `req->term > my_term`: update term, clear `voted_for_term`/`voted_for_name`. If `req->term < my_term`: deny. If already voted for a different candidate in same term: deny. @@ -483,6 +486,7 @@ _Atomic uint64_t committed_lsn; // highest majority-ACKed LSN; invariant: commi - Sends over persistent control channel fd (no new connection per entry) - On send failure: remove standby from active set, trigger catch-up reconnect, log WARN - Returns 0 when fanout initiated (not when ACKed) +- (Note: signature and mechanism superseded — current `repl_broadcast(replicator_t *r, const wal_entry_hdr_t *hdr, const void *payload, uint32_t payload_len, uint64_t prev_log_term)` sends a per-entry AppendEntries carrying the leader's per-entry term for the raft library's log-replication reconcile, with NACK-driven retry; see raft-log-replication, ha-term-safety, raft-adapter-conformance design.md) **`repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)` contract**: - Blocks (cond_timedwait) until `acked_count + 1 >= (n_active + 1) / 2` @@ -490,6 +494,7 @@ _Atomic uint64_t committed_lsn; // highest majority-ACKed LSN; invariant: commi - `ack_op` parameter selects `OP_walAck` vs `OP_upgradeConfigAck` to disambiguate concurrent commits - Returns 0 on quorum, `-ETIMEDOUT` on timeout - Single-node shortcircuit: `n_active == 0` → quorum immediately from own write (5.8, 8.5) +- (Note: signature and mechanism superseded — current `repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn, uint32_t timeout_ms, uint8_t ack_op)` delegates the walAck track to a commit engine (`commit_submit`) rather than counting acks directly, and gates commit on `durable_lsn` (fdatasync-confirmed), not raw ACK count; see ha-commit-engine and ha-commit-durability design.md) **Contracts**: Service [x] / State [x] @@ -511,6 +516,7 @@ _Atomic uint64_t committed_lsn; // highest majority-ACKed LSN; invariant: commi 2. Any peer has `tag > 0` → set `c->ha.defer_config_fetch = true`, return (3.2) 3. No peer reachable → retry with exponential backoff 500ms→10ms cap 10s; log WARN each attempt (3.4) - Never returns an error; blocks until resolution or process killed +- (Note: this full contract was later simplified/removed — the production `bootstrap_config_if_needed()` was removed in election-iomgr-refactor task 5.3; only a stub remains for linkage, mirroring the simplified two-branch logic now inlined ahead of `ha_start_election_async()` in myapp.c. See election-iomgr-refactor design.md) **`xce_probe_config_tag(const char *host, uint16_t port, uint64_t *tag_out)` contract**: - Single-RTT SSL request via `connect_ssl_raw()` + xce encode/decode @@ -598,7 +604,7 @@ Standby side: 5. Apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each 6. On catch-up loop end from primary: `ha.in_catch_up = 0` -**Role flip on promotion** (6.8): standby closes and nulls `primary_ctrl_fd` before returning from `am_i_primary_quorum()`; new primary accepts inbound `OP_ctrlHello` connections in a per-standby fd table. +**Role flip on promotion** (6.8): standby closes and nulls `primary_ctrl_fd` before returning from `am_i_primary_quorum()` (Note: this call site moved to the raft library's `on_become_leader` callback path, `raft_cb_become_leader`, since `am_i_primary_quorum()` was removed — see election-iomgr-refactor); new primary accepts inbound `OP_ctrlHello` connections in a per-standby fd table. **Contracts**: Service [x] / State [x] @@ -611,6 +617,8 @@ Standby side: ### Client Retry (src/lib/request.c) +**(Note: superseded by libxce-failover)** — `xce_request_with_retry()` described below was later removed as an "orphaned no-op" (per `src/lib/request.c` comment at the definition of `XCE_RETRY_DEFAULTS`); it was superseded by a `StatusClassifier`/`EndpointRotator` failover loop in `src/lib/client_failover.c`, driven from the xcecli layer. `xce_retry_config_t` and `XCE_RETRY_DEFAULTS` (the tunable struct/defaults — max_retries=20, initial_retry_ms=50, max_retry_ms=5000, backoff_multiplier=2.0) still exist verbatim in `src/include/request.h` and are consumed by the new failover loop. See libxce-failover design.md for the current mechanism. + #### libxce Retry Wrapper | Field | Detail | @@ -682,6 +690,7 @@ struct UpgradeConfigAck { term @0 :UInt64; ackedTag @1 :UInt64; success @2 :Bool struct SnapshotTransfer { snapshotLsn @0 :UInt64; payload @1 :Data; } struct WalCatchupEntry { lsn @0 :UInt64; entryType @1 :UInt8; payload @2 :Data; } ``` +(Note: `WalCatchupEntry.payload` is now `List(UInt8)`, not `Data` — bug-1505 capnpc-c codec fix, unrelated to any listed spec) Also extend: - `CtrlHeartbeat`: add `committedLsn @N :UInt64` and `committedConfigTag @M :UInt64` for piggybacking (req 5.6, 8.4) @@ -719,6 +728,7 @@ typedef struct { - WAL trim guard: threshold = `committed_lsn` not `applied_lsn`; prevents trimming entries a promoted standby may need for divergence resolution - `committed_lsn <= applied_lsn` asserted at promotion time; violated invariant is a programming error - `ha-term.bin` is written atomically before any vote reply is sent; a crash between write and reply means the term is persisted but the vote may not have been counted — conservative and safe +- (Note: majority-commit via `committed_lsn` alone is superseded — ha-commit-durability added a `durable_lsn` field: a member counts toward commit majority only once its LSN is `fdatasync`-confirmed, not merely ACKed/appended; `committed_lsn` still exists but the true commit/promotion-safety floor is `persist_durable_floor()`. See ha-commit-durability design.md) --- diff --git a/.kiro/specs/ha-consistency-invariants/design.md b/.kiro/specs/ha-consistency-invariants/design.md index ddce993e..5756f695 100644 --- a/.kiro/specs/ha-consistency-invariants/design.md +++ b/.kiro/specs/ha-consistency-invariants/design.md @@ -123,6 +123,8 @@ src/controller/ ├── config_repl.c # MODIFIED — config barrier error propagation; ACK-driven catch-up ├── job.c # MODIFIED — submit_lsn tagging, mark_job_* idempotency guards ├── dataex.c # MODIFIED — egress watermark filter in collect path +│ # (Note: as-built, the filter predicate/wrapper live in a +│ # new file egress.c instead — see ha-convergence-recovery) └── testsuite/ ├── test_commit_engine.c # NEW — per-LSN majority, own-term gate, admission cap, monotonicity ├── test_ha_role_lifecycle.c # NEW — direct promotion, no-op commit, demotion truncate+restart seam @@ -136,7 +138,7 @@ src/include/persist.h # MODIFIED — WAL_NOOP type, classification e - `repl.c/repl.h` — `repl_on_ack` forwards `(ep, acked_lsn)` to `commit_on_ack`; `repl_wait_majority_ack`'s counting logic retired in favor of commit-engine evaluation (function shell may remain as adapter) - `config_repl.c` — `push_config_to_standbys` returns `-ETIMEDOUT` on missed majority; caller fails the upgrade request; `ha_stream_wal_to_standby` continuation on walAck instead of next heartbeat - `job.c` — `job_t` gains `submit_lsn`; `mark_job_*` gain state-transition guards (no DONE→RUN, idempotent re-apply) -- `dataex.c` — `collect_jobs` (scheduler export path only) skips jobs with `submit_lsn > committed_lsn` +- `dataex.c` — `collect_jobs` (scheduler export path only) skips jobs with `submit_lsn > committed_lsn` (as-built: this predicate/wrapper live in new file `egress.c` instead — see the Egress Filter section note below) ## System Flows @@ -188,9 +190,9 @@ stateDiagram-v2 RESTART --> INIT: process self restart via boot recovery ``` -Promotion: `raft_cb_become_leader` sets `CTRL_PRIMARY` directly, registers persistence hooks, appends a `WAL_NOOP` entry of the current term, and broadcasts it. Request-shaped writes are accepted immediately and simply wait behind the contiguous commit index (the no-op commits first); status queries report primary (1.4). No wall-clock promotion/force path exists (1.5, 3.3). +Promotion: `raft_cb_become_leader` sets `CTRL_PRIMARY` directly, registers persistence hooks, appends a `WAL_NOOP` entry of the current term, and broadcasts it. Request-shaped writes are accepted immediately and simply wait behind the contiguous commit index (the no-op commits first); status queries report primary (1.4). No wall-clock promotion/force path exists (1.5, 3.3). (Note: promotion is no longer unconditional — `raft_cb_become_leader` now runs several eligibility/fencing gates (recovery-truncated, apply-divergent, config serve-fence, etc.) that can refuse promotion and leave the node alive, non-promotable, and retrying instead of becoming primary; added by `ha-commit-durability` and `ha-term-safety`, see those specs' design.md.) -Demotion: on observing a higher term while PRIMARY, the node truncates its WAL above `committed_lsn`, logs the event, and self-restarts; boot recovery (`persist_recover` + replay of the committed prefix) rebuilds in-memory state, discarding provisional entries (2.5, 2.7). Crash without truncation: the restarted node replays its full local WAL, but on rejoin as standby, if its local WAL tip exceeds the primary's replication boundary, it truncates above the primary-reported `committed_lsn` and self-restarts once (rejoin reconciliation — extends the existing RDHUP truncation pattern). +Demotion: on observing a higher term while PRIMARY, the node truncates its WAL above `committed_lsn`, logs the event, and self-restarts; boot recovery (`persist_recover` + replay of the committed prefix) rebuilds in-memory state, discarding provisional entries (2.5, 2.7). Crash without truncation: the restarted node replays its full local WAL. (Note: the "rejoin as standby ... truncates + self-restarts once" rejoin-reconciliation mechanism described here is superseded by `ha-term-safety` (task 3.2) — `ha_rejoin_reconcile` is now a no-op stub; a rejoining node's divergent suffix is instead truncated precisely via the raft library's term-aware `on_truncate_from` driven off real AppendEntries `(term,lsn)` matching, not an LSN-watermark self-restart. See `ha-term-safety/design.md`.) ## Requirements Traceability @@ -235,7 +237,7 @@ Demotion: on observing a higher term while PRIMARY, the node truncates its WAL a | HA Role Lifecycle | controller/ha.c | direct promotion + no-op, COMMITTING removal, redirect fix | 1.1–1.5, 3.1, 3.3 | raft-library callbacks (P0), Commit Engine (P0) | State | | Quorum Accounting | controller/ha.c | honest gate-time quorum, catch-up exclusion | 4.1, 7.1, 7.4 | repl registry (P0) | Service | | Config Barrier | controller/config_repl.c | config majority-ACK with error propagation | 5.1–5.3 | Commit Engine pattern (P1) | Service | -| Egress Filter | controller/dataex.c, job.c | exclude uncommitted submits from scheduler export | 2.4 | committed_lsn (P0) | Service | +| Egress Filter | controller/egress.c, job.c | exclude uncommitted submits from scheduler export | 2.4 | committed_lsn (P0) | Service | | Demotion Convergence | controller/ha.c | truncate + self-restart; rejoin reconciliation | 2.5, 2.7 | persist truncation + recovery (P0) | State | | ACK-Driven Catch-up | controller/config_repl.c | continue streaming on walAck | 6.2 | repl/iomgr (P0) | Event | | Quorum Observability | controller/ha.c | log degraded-state transitions | 7.2 | — | — | @@ -250,9 +252,9 @@ Demotion: on observing a higher term while PRIMARY, the node truncates its WAL a **Responsibilities & Constraints** - Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`; only the commit engine stores to it after init/truncation paths) -- Commit rule (Raft §5.3): entry N is committed when `count(self + standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.6) +- Commit rule (Raft §5.3): entry N is committed when `count(self + standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.6). (Note: superseded by `ha-commit-durability` — the primary's own self-contribution is now capped at `persist_durable_lsn` (fdatasync'd locally), not the appended tip, so an entry only in the page cache cannot count self toward majority; see that spec's design.md.) - Own-term gate: `writable_lsn` = LSN of the promotion no-op; completion (confirmation) is withheld for any entry until `committed_lsn >= writable_lsn` (3.1, 3.2) -- Admission cap: at most `n_vcpus - COMMIT_WAIT_RESERVED` (default reserved = 2) concurrent WAIT-mode blockers; excess requests fail fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`) +- Admission cap: at most `n_vcpus - COMMIT_WAIT_RESERVED` (default reserved = 2) concurrent WAIT-mode blockers; excess requests fail fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`). (Note: superseded in part by `raft-drive-async` — when a vtaskmgr pool is wired via `commit_set_pool`, admission folds onto the pool's single `VTASK_LIVENESS_RESERVED` reservation instead of the local `COMMIT_WAIT_RESERVED` constant; the local cap is the fallback only when no pool is wired. See that spec's design.md.) - `n_peers <= 1`: self-ACK constitutes majority; WAIT mode returns immediately (7.5). For `n_peers > 1`, zero peer ACKs never commit (4.4) — the legacy `n_active == 0 → success` short-circuit is removed **Dependencies** @@ -322,10 +324,12 @@ Classification (verified against call sites during implementation; table is the | WAL_NOOP | internal (promotion) | RETAIN | none — gates writability when committed | | WAL_CHECKPOINT | local-only | none | unchanged (not replicated, by design) | +> **Note (rows added by later specs, not in the original table):** `WAL_JOB_ARRAY_SUBMIT` — request (client array submit), WAIT, single commit barrier per array regardless of size (`job-array-single-wal-record` task 1.1). `WAL_CONFIG_MARKER` — RETAIN, the consensus/ordering record for a config change appended to the same unified log (`ha-term-safety` task 4.2; see also the Config Barrier note below). Current full table verified against `wal_classify` (persist.c). + **Reply-Ordering Contract (commit-first, then reply)** - For every request-shaped operation (job control, queue control, worker join/update), the handler sequence is fixed: validate → mutate (provisional) → WAL append → `commit_submit(WAIT)` → **reply**. No success reply may be sent to the requester before `commit_submit` returns 0; on non-zero return the handler replies with the error (2.1, 2.2). -- Known reply-before-commit call sites to restructure: `do_jobcontrol_op` (job.c:921-934 — replies after mutation with only async WAL) and any queue/worker handlers following the same pattern. `handle_jobsubmit` (job.c:563-578) is the reference implementation of the correct ordering. -- `handle_jobdispatch` (job.c:459-471 — replies status=0 to the scheduler before processing allocations) is classified during implementation: if its allocations produce request-shaped state changes, the early reply must move behind the barrier; if its outcomes are fact-shaped (per-job dispatch results delivered via `notify_dispatch`), the early reply is an ACK-of-receipt and may stand. Task-level verification required; the classification assert enforces the outcome. +- Known reply-before-commit call sites to restructure: `do_jobcontrol_op` (job.c:2198-2223 — line numbers current as of this audit, job.c has grown substantially since spec close) and any queue/worker handlers following the same pattern. `handle_jobsubmit` (job.c:1412 onward — line numbers current as of this audit) is the reference implementation of the correct ordering. (Note: `do_jobcontrol_op` is now fixed — it calls `persist_commit_barrier` before replying, per `ha-commit-barrier-rollout` Req 1.1-1.4; this row is historical, not an open item.) +- `handle_jobdispatch` (job.c:1119-1131 — line numbers current as of this audit — replies status=0 to the scheduler before processing allocations) is classified during implementation: if its allocations produce request-shaped state changes, the early reply must move behind the barrier; if its outcomes are fact-shaped (per-job dispatch results delivered via `notify_dispatch`), the early reply is an ACK-of-receipt and may stand. Task-level verification required; the classification assert enforces the outcome. (Confirmed as-built: outcomes are fact-shaped via `notify_dispatch`, so the early reply stands unchanged.) **Implementation Notes** - Integration: `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` consult the table; `persist_wal_async` survives only as the RETAIN fast path (append + broadcast + register) @@ -355,13 +359,13 @@ Classification (verified against call sites during implementation; table is the Summary-only: `ha_has_write_quorum` counts standbys that are connected AND not `in_catch_up` (4.1); evaluated at the dispatch gate for fast rejection (7.1) and re-evaluated authoritatively by the commit engine at ACK time (4.2, 4.3); stateless, so quorum restoration auto-resumes acceptance (7.4). Transition logging lives here (see Quorum Observability). ### Config Barrier (`config_repl.c`) -Summary-only: `push_config_to_standbys` returns `-ETIMEDOUT` when majority ACK is not reached within the commit timeout; `committed_config_tag` does not advance (already correct); the `upgradeConfig` caller propagates the error to the requester and fails the upgrade (5.1–5.3). Uses the same majority arithmetic as the commit engine (shared helper) but tracks `acked_config_lsn`/tag — config remains a parallel, simpler track (one in-flight config change at a time). +Summary-only: `push_config_to_standbys` returns `-ETIMEDOUT` when majority ACK is not reached within the commit timeout; the `upgradeConfig` caller propagates the error to the requester and fails the upgrade (5.1–5.3). Uses the same majority arithmetic as the commit engine (shared helper). (Note: superseded in part by `ha-term-safety` task 4.2 — config is no longer a fully parallel, simpler track. `config_commit_change` now appends a `WAL_CONFIG_MARKER` consensus record to the same unified `(term,lsn)` log BEFORE the byte-delivery barrier; the derived `config_committed_marker` is the commit authority and `committed_config_tag` is retired to a non-authoritative cache. The byte-delivery majority-ack barrier mechanism itself (`push_config_to_standbys`, `-ETIMEDOUT` on failure) is unchanged. See `ha-term-safety/design.md`.) ### Egress Filter (`dataex.c`, `job.c`) -Summary-only: `job_t` gains `uint64_t submit_lsn` (set when the submit entry is appended). The scheduler export path (`collect_jobs` for dataex push/pull) skips jobs with `submit_lsn > committed_lsn` (2.4 — prevents dispatch of a job whose entry may never commit). Status/API query paths are intentionally NOT filtered (documented deviation; read consistency out of scope). Replay/standby-restored jobs carry their entry LSN (≤ committed by definition). +Summary-only: `job_t` gains `uint64_t submit_lsn` (set when the submit entry is appended). The scheduler export path skips jobs with `submit_lsn > committed_lsn` (2.4 — prevents dispatch of a job whose entry may never commit). Status/API query paths are intentionally NOT filtered (documented deviation; read consistency out of scope). Replay/standby-restored jobs carry their entry LSN (≤ committed by definition). (Note: the predicate and scheduler-bound collect wrapper this design assigns to `dataex.c` were factored into a new file `egress.c` — `job_submit_committed()` + `collect_jobs_committed()` — by `ha-convergence-recovery`, kept out of `dataex.c` for controller-unit-test linkability. `dataex.c`'s plain `collect_jobs` remains unfiltered, as intended, for status/API callers. The file's own header comment calls this "the same egress-filter component the design assigns to dataex.c, factored out for testability" — a refinement, not a contradiction.) ### Demotion Convergence (`ha.c`) -Summary-only: on higher-term observation while PRIMARY (`raft_cb_become_follower` from PRIMARY): disable persistence hooks, truncate WAL above `committed_lsn`, WARN-log with both LSNs, then self-restart the process (re-exec of saved argv; under `XCE_TESTING`, a restart hook seam is invoked instead). Boot recovery replays only the surviving committed prefix → provisional in-memory state is discarded (2.5, 2.7). Rejoin reconciliation: a standby whose local WAL tip exceeds the primary's reported committed boundary truncates above the primary's `committed_lsn` (existing RDHUP truncation generalized to the rejoin/hello path) and self-restarts once. +Summary-only: on higher-term observation while PRIMARY (`raft_cb_become_follower` from PRIMARY): disable persistence hooks, truncate WAL above `committed_lsn`, WARN-log with both LSNs, then self-restart the process (re-exec of saved argv; under `XCE_TESTING`, a restart hook seam is invoked instead) — this part still matches as-built (`ha_schedule_demotion_convergence`). Boot recovery replays only the surviving committed prefix → provisional in-memory state is discarded (2.5, 2.7). (Note: the "Rejoin reconciliation" sentence originally here — a standby truncating + self-restarting once on rejoin — is superseded by `ha-term-safety` task 3.2: `ha_rejoin_reconcile` is now a no-op stub retained only as the CtrlHelloReply hook that records the learned committed boundary; the actual divergent-suffix reconciliation is owned by the raft library's term-aware `on_truncate_from`, not a watermark-clamped self-restart. See `ha-term-safety/design.md`.) ### ACK-Driven Catch-up (`config_repl.c`) Summary-only: catch-up streaming sends a bounded batch window; receipt of a walAck from a catch-up standby (iomgr) schedules the next batch immediately via a delaymgr task, instead of waiting for the next heartbeat reply (6.2). Heartbeat-driven evaluation remains as the fallback initiator and for snapshot decisions (6.3); `repl_complete_catchup` re-inclusion unchanged (6.4). diff --git a/.kiro/specs/ha-convergence-recovery/design.md b/.kiro/specs/ha-convergence-recovery/design.md index c533abeb..6cb1d06b 100644 --- a/.kiro/specs/ha-convergence-recovery/design.md +++ b/.kiro/specs/ha-convergence-recovery/design.md @@ -8,7 +8,7 @@ **Users**: Cluster operators gain single-outcome failover semantics (no phantom state on demoted/rejoined nodes) and observable degraded periods; the scheduler is protected from orphan dispatch; lagging standbys catch up at ACK speed. -**Impact**: Extends `raft_cb_become_follower` (ha.c:627-690 — currently neither truncates nor disables hooks, verified gap) with the demotion-convergence sequence; generalizes the RDHUP truncation (ha.c:3416-3420) to the rejoin path; adds `submit_lsn` to the `job_t` wrapper and a watermark predicate on the scheduler export seams; adds an ACK-driven continuation to catch-up streaming (`ha_stream_wal_to_standby`, config_repl.c:1450 — today re-driven only by the next heartbeat reply); adds quorum-transition WARN logging. No commit semantics change: this spec consumes `committed_lsn` (ha-commit-engine) and never modifies `commit.c`. +**Impact**: Extends `raft_cb_become_follower` (ha.c:2104-2247 — line drift only, was ha.c:627-690 at design time; currently neither truncates nor disables hooks, verified gap) with the demotion-convergence sequence; generalizes the RDHUP truncation (ha.c:3416-3420 at design time) to the rejoin path (Note: superseded by ha-term-safety — the rejoin/RDHUP watermark-truncate heuristic this spec built was retired and replaced by the raft library's (term,lsn)-driven `on_truncate_from`; see the Rejoin Reconciliation component below and that spec's design.md); adds `submit_lsn` to the `job_t` wrapper and a watermark predicate on the scheduler export seams; adds an ACK-driven continuation to catch-up streaming (`ha_stream_wal_to_standby`, now config_repl.c:2351 — today re-driven only by the next heartbeat reply); adds quorum-transition WARN logging. No commit semantics change: this spec consumes `committed_lsn` (ha-commit-engine) and never modifies `commit.c`. ### Goals - Demotion = disable hooks → truncate above `committed_lsn` → WARN-log range → process self-restart via existing boot recovery; exec failure is FATAL; `XCE_TESTING` restart seam (1.1–1.7) @@ -29,7 +29,7 @@ ### This Spec Owns - The demotion path of the role lifecycle: everything `raft_cb_become_follower` does when leaving PRIMARY (hook disable, truncation, logging, self-restart) and the self-restart primitive (saved argv, exec, FATAL fallback, test seam) -- Rejoin reconciliation at the standby's `CtrlHelloReply` intake (truncate-above-boundary decision, clamp, once-guard) +- Rejoin reconciliation at the standby's `CtrlHelloReply` intake (truncate-above-boundary decision, clamp, once-guard) (Note: superseded by ha-term-safety — `ha_rejoin_reconcile` is retained only as the intake hook that records the learned boundary; the truncate-above-boundary decision, clamp, and once-guard it describes were retired in favor of the raft library's term-aware truncation, see Rejoin Reconciliation component below) - `job_t.submit_lsn` tagging (live append + replay) and the commit-watermark predicate on scheduler-bound export seams - The ACK-driven catch-up continuation trigger (walAck branch → scheduled streaming) including the single-flight guard and the parity call to `repl_complete_catchup` - Quorum-loss/restore transition logging and its `quorum_lost` state flag @@ -54,7 +54,7 @@ ### Revalidation Triggers - ha-commit-engine changing `committed_lsn` ownership or making it persisted → re-verify rejoin reconciliation and the egress watermark - ha-commit-barrier-rollout landing (fact allowlist) → re-verify the degraded-mode gate pin: the no-quorum rejected set becomes request-shaped opcodes only; fact-shaped opcodes (`OPCode_notify`/`update`/`setpreason`/`clrpreason`) are admitted -- ha-single-leader-role landing → re-anchor the quorum-transition log call site at promotion standby registration (ha.c:531), which lives inside `raft_cb_become_leader` (ha.c:445-617) — code that spec owns and rewrites wholesale +- ha-single-leader-role landing → re-anchor the quorum-transition log call site at promotion standby registration (ha.c:531), which lives inside `raft_cb_become_leader` (ha.c:445-617) — code that spec owns and rewrites wholesale. **(Resolved as-built: re-anchored to ha.c:2042 inside `raft_cb_become_leader`, now ha.c:1615-2103; the demotion insertion point re-verified at ha.c:2104-2247. Both current per the fact-check pass below.)** - ha-single-leader-role changing the become-follower/become-leader callback contract → re-verify the demotion sequence insertion point - Entity-bearing snapshots landing → re-evaluate restart-based eviction vs snapshot-based rebuild (umbrella revalidation trigger) - Any new scheduler-bound export path → must adopt the watermark predicate (same rule as new WAL types must be classified) @@ -63,13 +63,14 @@ ## Architecture ### Existing Architecture Analysis -- `raft_cb_become_follower` (ha.c:627-690) cancels timers, closes vote endpoints, sets CTRL_STANDBY — verified: no truncation, no hook disable (the disable logic exists only in the uncalled `ha_demote_to_standby`, ha.c:1712-1726) -- The RDHUP failover path already truncates above the locally known committed boundary (ha.c:3416-3420) — the pattern this spec generalizes to demotion and rejoin -- The standby learns the primary's `committed_lsn` at the `CtrlHelloReply` intake (cluster.c:283-298, stored to `ha->last_known_committed_lsn`) — the rejoin-reconciliation evaluation point -- Catch-up is heartbeat-polled: the stateReport branch (ha.c:2734) drives `ha_dispatch_repl_action_for_standby` (ha.c:1373-1478), one action per heartbeat cycle; the walAck branch (ha.c:2737-2754) only records the ACK. Replication target is already derived as `max(committed_lsn, last_written)` (ha.c:2727-2732) -- `repl_complete_catchup` (repl.c:521) has no production caller (discovered gap — in_catch_up is set by broadcast-overflow eviction, repl.c:240, and never cleared); the ACK-driven continuation supplies the missing parity trigger -- Scheduler-bound egress has two seams: collect-based full push (`xcesched_push_jobs` sched.c:169, on scheduler join) and the incremental JOB_NEW push (`sched_job_new_fun` sched.c:236-248 → `xcesched_push_joblist`); both must apply the watermark. Read paths (apiserver.c:83, opget.c:173) share `collect_jobs` (dataex.c:294) and stay unfiltered -- No saved argv / exec machinery exists anywhere in the controller — self-restart is net-new (main.c:112) +> Note: this subsection is a research-time (2026-06-11) snapshot of the PRE-implementation baseline — the "verified gap" facts are what the code lacked before this spec ran. Where a later spec has since changed the mechanism, a note is added; otherwise only stale `file:line` citations are corrected in place. +- `raft_cb_become_follower` (now ha.c:2104-2247, was ha.c:627-690 at design time) cancels timers, closes vote endpoints, sets CTRL_STANDBY — verified: no truncation, no hook disable (the disable logic exists only in the uncalled `ha_demote_to_standby`, now ha.c:5081-5094, was ha.c:1712-1726 at design time) +- The RDHUP failover path already truncates above the locally known committed boundary (ha.c:3416-3420 at design time) — the pattern this spec generalizes to demotion and rejoin. (Note: superseded by ha-term-safety — this ad hoc watermark-truncate pattern, and the rejoin generalization built on it, was itself replaced by the raft library's (term,lsn)-driven `on_truncate_from` / `ha_cb_truncate_from_stub`, ha.c:2638-2655, whose own doc comment states it "REPLACES the committed-boundary truncate heuristic on the library-driven path"; see that spec's design.md) +- The standby learns the primary's `committed_lsn` at the `CtrlHelloReply` intake (now cluster.c:411-419, was cluster.c:283-298 at design time, stored to `ha->last_known_committed_lsn`) — the rejoin-reconciliation evaluation point +- Catch-up is heartbeat-polled: the stateReport branch (now ha.c:6683, was ha.c:2734) drives `ha_dispatch_repl_action_for_standby` (now ha.c:4331-4466, was ha.c:1373-1478), one action per heartbeat cycle; the walAck branch (now ha.c:6764-6812, was ha.c:2737-2754) only records the ACK. Replication target is already derived as `max(committed_lsn, last_written)` (now via `ha_compute_repl_target_lsn`, called at ha.c:6759, was inline at ha.c:2727-2732) +- `repl_complete_catchup` (now repl.c:769, was repl.c:521 at design time) has no production caller (discovered gap — in_catch_up is set by broadcast-overflow eviction, repl.c:240, and never cleared); the ACK-driven continuation supplies the missing parity trigger +- Scheduler-bound egress has two seams: collect-based full push (`xcesched_push_jobs` now sched.c:214, was sched.c:169, on scheduler join) and the incremental JOB_NEW push (`sched_job_new_fun` now sched.c:286, was sched.c:236-248 → `xcesched_push_joblist`); both must apply the watermark. Read paths (apiserver.c:83, opget.c:173 — both confirmed current, unchanged) share `collect_jobs` (dataex.c:294 — confirmed current, unchanged) and stay unfiltered +- No saved argv / exec machinery exists anywhere in the controller — self-restart is net-new (main.c:112 at design time; landed at main.c:291-301 as-built) ### Architecture Pattern & Boundary Map @@ -135,16 +136,21 @@ graph TB ### Directory Structure ``` src/controller/ -├── ha.c # MODIFIED — demotion convergence sequence, ha_self_restart, rejoin -│ # reconciliation helper, walAck continuation trigger, -│ # quorum transition logging +├── ha.c # MODIFIED — demotion branch inline in raft_cb_become_follower, walAck +│ # continuation trigger, quorum transition call sites +├── ha_convergence.c # NEW (as-built, not in original plan) — ha_self_restart, +│ # ha_rejoin_reconcile, cluster_apply_ctrlhello_reply, +│ # ha_log_quorum_transition, ha_demotion_convergence_do_run +│ # all landed here instead of ha.c ├── ha.h # MODIFIED — quorum_lost flag, catchup_inflight tracking, restart-hook │ # seam declaration, new function prototypes ├── main.c # MODIFIED — save argv for re-exec (controller_saved_argv accessor) ├── cluster.c # MODIFIED — CtrlHelloReply intake calls rejoin reconciliation ├── config_repl.c # MODIFIED — catch-up continuation task fn (wraps ha_stream_wal_to_standby) -├── dataex.c # MODIFIED — job_submit_committed predicate + committed-only collect for -│ # scheduler-bound export +├── egress.c # NEW (as-built, not in original plan) — job_submit_committed predicate + +│ # collect_jobs_committed, factored out of dataex.c for +│ # test-binary linkage (research D3 refinement; declared in +│ # include/cluster.h, not dataex.h) ├── sched.c # MODIFIED — scheduler push paths consume the predicate (full + JOB_NEW push) ├── job.c # MODIFIED — restore_job tags submit_lsn from the replayed entry ├── persist.c # MODIFIED — persist_job_hook_fun tags submit_lsn at live append; @@ -159,14 +165,14 @@ src/controller/meson.build # MODIFIED — register new test files ``` ### Modified Files (what and why) -- `ha.c` / `ha.h` — demotion branch in `raft_cb_become_follower` (detect prior role PRIMARY → disable hooks inline → schedule convergence task); `ha_self_restart()` (exec / seam / FATAL); `ha_rejoin_reconcile()` (clamped truncation + once-guard + restart); walAck-branch continuation scheduling with per-standby single-flight guard and parity `repl_complete_catchup` call; `ha_log_quorum_transition()` called from the three standby_endpoints mutation sites (ha.c:531, 3160-3166, on_standby_connected) and the gate rejection backstop -- `main.c` — store argc/argv before parsing; expose `controller_saved_argv()` for re-exec -- `cluster.c` — `CtrlHelloReply` intake (cluster.c:283-298) additionally invokes `ha_rejoin_reconcile(c, chrep->committed_lsn)` -- `config_repl.c` — `ha_catchup_continuation_fn` (delaymgr task body): re-derive target, call `ha_stream_wal_to_standby`, clear the in-flight flag -- `dataex.c` — `job_submit_committed(cluster_t*, job_t*)` predicate; `collect_jobs_committed()` wrapper used only by scheduler-bound callers (read paths keep plain `collect_jobs`) -- `sched.c` — `xcesched_push_jobs` collects via the committed-only wrapper; `xcesched_push_joblist` (JOB_NEW incremental push) filters its list with the same predicate -- `job.c` / `persist.c` — tagging: live path sets `submit_lsn = written_lsn` in `persist_job_hook_fun` (persist.c:1107-1125); replay path threads `hdr->lsn` from `apply_wal_entry` (persist.c:691/729-731) into `restore_job` (job.c:71) -- `src/include/xce/types.h` — `job_t` wrapper (types.h:483-485) gains `uint64_t submit_lsn` (0 = unknown/pre-feature = exportable); wire type `xce_job_t` untouched +- `ha.c` / `ha.h` — demotion branch in `raft_cb_become_follower` (ha.c:2104-2247; detect prior role PRIMARY → disable hooks inline → schedule convergence task); walAck-branch (ha.c:6764-6812) continuation scheduling with per-standby single-flight guard and parity `repl_complete_catchup` call; `ha_log_quorum_transition()` called from the standby_endpoints mutation sites (now ha.c:2042, 7651, 9350) and the gate rejection backstop (ha.c:4617). As-built, `ha_self_restart()` and `ha_rejoin_reconcile()` landed in a new `ha_convergence.c` (see File Structure Plan) rather than `ha.c` directly. +- `main.c` — store argc/argv before parsing (main.c:291-301); expose `controller_saved_argv()` for re-exec +- `cluster.c` — `CtrlHelloReply` intake (now cluster.c:411-419) additionally invokes `ha_rejoin_reconcile(c, chrep->committed_lsn)` via `cluster_apply_ctrlhello_reply`. (Note: superseded by ha-term-safety — `ha_rejoin_reconcile` now only records the boundary; see Rejoin Reconciliation component.) +- `config_repl.c` — `ha_catchup_continuation_fn` (now config_repl.c:2839, delaymgr task body): re-derive target, call `ha_stream_wal_to_standby` (config_repl.c:2351), clear the in-flight flag +- `egress.c` (as-built; design originally assigned this to `dataex.c`, see File Structure Plan) — `job_submit_committed(cluster_t*, job_t*)` predicate; `collect_jobs_committed()` wrapper used only by scheduler-bound callers (read paths keep plain `collect_jobs`, dataex.c:294) +- `sched.c` — `xcesched_push_jobs` (sched.c:214) collects via the committed-only wrapper; `xcesched_push_joblist` (sched.c:70, JOB_NEW incremental push) filters its list with the same predicate +- `job.c` / `persist.c` — tagging: live path sets `submit_lsn = written_lsn` in `persist_job_hook_fun` (now persist.c:6509); replay path threads `hdr->lsn` from `apply_wal_entry` (now persist.c:3281) into `restore_job` (now job.c:274) +- `src/include/xce/types.h` — `job_t` wrapper (types.h:483-490) gains `uint64_t submit_lsn` (0 = unknown/pre-feature = exportable); wire type `xce_job_t` untouched ## System Flows @@ -194,9 +200,11 @@ sequenceDiagram OS-->>P: boot: persist_recover replays committed prefix only ``` -Key decisions: hooks are disabled synchronously in the callback (no provisional entry may be produced after the higher term is observed — 1.1); truncation and exec are deferred off the callback thread (iomgr-safe — D1); truncation always precedes restart so boot replay yields committed-only state (1.2, 1.3, 1.7); crash before/during the task is safe — the rejoin reconciliation flow covers the untruncated-suffix case. +Key decisions: hooks are disabled synchronously in the callback (no provisional entry may be produced after the higher term is observed — 1.1); truncation and exec are deferred off the callback thread (iomgr-safe — D1); truncation always precedes restart so boot replay yields committed-only state (1.2, 1.3, 1.7); crash before/during the task is safe — the rejoin reconciliation flow covers the untruncated-suffix case (Note: as of ha-term-safety, that coverage is via the raft library's term-aware truncation, not this spec's rejoin-reconcile truncate path — see Rejoin Reconciliation component). The "exec fails → FATAL (abort)" branch in the diagram above is also superseded: see the Self-Restart Primitive component and Error Handling section for the current fence-and-retry behavior. -### Rejoin reconciliation (generalizes RDHUP truncation, ha.c:3416-3420) +### Rejoin reconciliation (generalizes RDHUP truncation, ha.c:3416-3420 at design time) + +> **(Note: superseded by ha-term-safety, closed 2026-07-17.)** The flow and "Key decisions" below describe this spec's original clamped-truncate-and-restart mechanism, which is no longer live. `ha_rejoin_reconcile` (ha_convergence.c:467-493) is now a documented no-op: the truncate-above-boundary decision, clamp, and once-guard were retired in favor of the raft library's own (term,lsn) log-matching on incoming AppendEntries, which drives `on_truncate_from` to delete exactly the divergent suffix without a self-restart. The function is retained only as the `CtrlHelloReply` hook that records the learned committed boundary. See ha-term-safety's design.md for the current mechanism. ```mermaid sequenceDiagram @@ -328,7 +336,11 @@ extern void (*ha_fatal_hook)(const char *msg); /* invoked instead of abort */ | Intent | Converge a rejoining node whose log runs ahead of the cluster's committed boundary | | Requirements | 2.1–2.4 | -Summary: new `int ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn)` called from the `CtrlHelloReply` intake (cluster.c:283-298) after the existing `last_known_committed_lsn` store. Decision: `local_tip = persist.current_lsn - 1`; if `local_tip > primary_committed_lsn` and the once-guard is clear → cutoff = `max(primary_committed_lsn, local committed_lsn)` (clamp, 2.2) → `persist_truncate_wal_above(cutoff)` → WARN with range → set once-guard → `ha_self_restart`. Otherwise return 0 with no side effects (2.3). The truncation/restart work is deferred to a delaymgr task if the intake runs on iomgr (same rule as demotion). Generalizes the RDHUP truncation (ha.c:3416-3420), which remains in place for the failover path. Once-guard is structural (post-restart tip equals the boundary) plus a defensive `_Atomic int rejoin_reconciled` for the process lifetime (2.1). +**(Note: superseded by ha-term-safety, closed 2026-07-17 — the mechanism below describes this spec's original design/as-shipped behavior; see the current-state summary immediately after.)** + +Original design/as-shipped: new `int ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn)` called from the `CtrlHelloReply` intake (cluster.c:283-298 at design time, now cluster.c:411-419) after the existing `last_known_committed_lsn` store. Decision: `local_tip = persist.current_lsn - 1`; if `local_tip > primary_committed_lsn` and the once-guard is clear → cutoff = `max(primary_committed_lsn, local committed_lsn)` (clamp, 2.2) → `persist_truncate_wal_above(cutoff)` → WARN with range → set once-guard → `ha_self_restart`. Otherwise return 0 with no side effects (2.3). The truncation/restart work is deferred to a delaymgr task if the intake runs on iomgr (same rule as demotion). Generalizes the RDHUP truncation (ha.c:3416-3420 at design time), which the design assumed remains in place for the failover path. Once-guard is structural (post-restart tip equals the boundary) plus a defensive `_Atomic int rejoin_reconciled` for the process lifetime (2.1). + +Current state (ha_convergence.c:467-493): ha-term-safety task 3.2 retired this entire clamped-truncate-and-restart body. `ha_rejoin_reconcile` now ignores `primary_committed_lsn` and unconditionally returns 0 — divergent-suffix reconciliation is owned by the raft library instead: an incoming AppendEntries is routed through `raft_on_append_entries_request`, whose (term,lsn) log-matching drives `on_truncate_from` (`ha_cb_truncate_from_stub`, ha.c:2638-2655) to delete exactly the divergent suffix, never a committed-prefix entry, and never by an LSN watermark. A node whose tip runs ahead keeps its entries until the leader's AE proves the conflict by term; until reconciled the existing promotion-time fence keeps it non-promotable without self-restarting. The function is retained solely as the `CtrlHelloReply` hook (now wired via `cluster_apply_ctrlhello_reply`, ha_convergence.c:512-524) that still records the learned committed boundary. The Testing Strategy section's rejoin-reconcile test descriptions below are similarly stale; current tests (`test_rejoin_no_watermark_truncate_no_restart`, `test_rejoin_repeated_calls_remain_no_op`) pin the no-op contract instead. ### Egress Filter (`dataex.c`, `sched.c`, `job.c`, `persist.c`, `types.h`) diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/design.md b/.kiro/specs/ha-elect-ctx-lifecycle/design.md index 0e3c6651..61259e58 100644 --- a/.kiro/specs/ha-elect-ctx-lifecycle/design.md +++ b/.kiro/specs/ha-elect-ctx-lifecycle/design.md @@ -53,7 +53,7 @@ Two sibling capture functions allocate a fresh `ha_elect_ctx_t` each round and p - `ha_start_election_async` (real election) has **two fast-exit paths that already free the ctx correctly**: no-local-config bail-out and the single-node case (`n_peers == 1`, `raft_start_election` resolves synchronously via `on_become_leader`). Neither path ever creates a vote endpoint, so the ctx is never shared — a direct `free(ctx)` is correct today and stays correct; this design does not touch them. - The **multi-node path** is where the leak lives: `raft_start_election` fires `on_send_vote_request` per peer, each call routing through `raft_cb_send_vote_request`, which `calloc`s a `vote_ep_ctx_t` (`vctx`), sets `vctx->ctx = ectx`, attaches it as `ep->data`, and hands the endpoint to iomgr. `ha_start_prevote_async` → `raft_cb_send_pre_vote_request` is the structurally identical PreVote sibling. -- `vctx` is freed exactly once, generically, inside `endpoint_fini` (`src/lib/netio.c:1094`, `FREEUP(ep->data)`) — reached via `endpoint_unref` (bug-1298 refcount hitting 0) on every teardown path: connect failure, `iomgr_monitor` failure, the flag-OFF loser-path loop (`ha.c:~1553`), the flag-ON deferred-vtask path (`vote_ep_teardown_async` → `vote_ep_teardown_vtask_entry`), and RDHUP/err-handler paths. This is deliberately the **single** place `vctx` memory is reclaimed; the design must not duplicate that free, only observe it. +- `vctx` is freed exactly once, generically, inside `endpoint_fini` (`src/lib/netio.c:1171`, `FREEUP(ep->data)`) — reached via `endpoint_unref` (bug-1298 refcount hitting 0) on every teardown path: connect failure, `iomgr_monitor` failure, the flag-OFF loser-path loop (`ha.c:~2142`), the flag-ON deferred-vtask path (`vote_ep_teardown_async` → `vote_ep_teardown_vtask_entry`), and RDHUP/err-handler paths. This is deliberately the **single** place `vctx` memory is reclaimed; the design must not duplicate that free, only observe it. (Line numbers as of current source; `g_drive_async_peerctrl_dispatch_enabled` now defaults ON per raft-drive-async, so the flag-ON deferred-vtask path is the live default today — both paths still funnel through `endpoint_fini` exactly once, so the mechanism is unaffected.) - The carrier itself is currently **never** freed for the multi-node case: the next round's `atomic_store_explicit(&ha->elect_ctx, mc, ...)` (both the election and PreVote sites) silently discards the previous pointer, with a comment explicitly documenting the leak as deliberate (UAF avoidance). ### Mechanism @@ -84,7 +84,7 @@ free when refcount reaches 0 (pthread_mutex_destroy + sem_destroy + free), regardless of which of the two release kinds above is the last one to fire ``` -Why a bias reference is required (not pure per-endpoint counting): between `atomic_store_explicit(&ha->elect_ctx, ctx, ...)` and the first `raft_cb_send_vote_request` callback actually running, the carrier has zero vote-endpoint references yet but is already live and readable (`ha.c:123-124`, `234-235`, `903-904` read `ha->elect_ctx` without assuming any endpoint exists — e.g. single-vote-outstanding or mid-fan-out windows). A pure "count of attached endpoints" would allow a spurious free at refcount 0 before the round is superseded. The bias reference removes that window: the carrier is only reclaimed once it is *both* superseded *and* drained of endpoint references, in whichever order those two events occur. +Why a bias reference is required (not pure per-endpoint counting): between `atomic_store_explicit(&ha->elect_ctx, ctx, ...)` and the first `raft_cb_send_vote_request` callback actually running, the carrier has zero vote-endpoint references yet but is already live and readable (e.g. `ha.c:1638-1639`, `2008-2009` inside `raft_cb_become_leader` read `ha->elect_ctx` without assuming any endpoint exists — e.g. single-vote-outstanding or mid-fan-out windows). A pure "count of attached endpoints" would allow a spurious free at refcount 0 before the round is superseded. The bias reference removes that window: the carrier is only reclaimed once it is *both* superseded *and* drained of endpoint references, in whichever order those two events occur. Why a generic `endpoint_t.on_fini` hook (not per-call-site decrements at every `endpoint_unref`/teardown call): a `vctx` is only *actually freed* inside `endpoint_fini`, but `endpoint_unref` (which may or may not trigger `endpoint_fini` synchronously, depending on whether a pool vtask still holds a ref) is called from roughly a dozen sites in `ha.c` across connect-failure, iomgr-failure, the loser-path loop, and the bug-1197 deferred-teardown-vtask path. Decrementing at each call site would require knowing, at every one of those dozen sites, whether this call is the one that actually drops the count to zero — which duplicates `endpoint_fini`'s own refcount logic and is exactly the kind of coupling Req 2.3/2.4 (uniform coverage of every free path, "including error and abort paths") is trying to make robust against drift. A single hook fired at the one true free point is correct by construction for every current and future teardown path, mirroring the existing `on_connected` fire-once hook already on `endpoint_t`. @@ -119,7 +119,7 @@ sequenceDiagram ### Modified Files - `src/include/netio.h` — add `void (*on_fini)(struct endpoint_t *ep);` field to `endpoint_t`, adjacent to the existing `on_connected` field, with a doc comment cross-referencing it as the completion-side counterpart. -- `src/lib/netio.c` — in `endpoint_fini`, add `if (ep->data && ep->on_fini) ep->on_fini(ep);` immediately before the existing `FREEUP(ep->data)` line. No other change to the function. +- `src/lib/netio.c` — in `endpoint_fini`, add `if (ep->on_fini) { ep->on_fini(ep); }` immediately before the existing `FREEUP(ep->data)` line (as-built: the hook is invoked unconditionally when set, not gated on `ep->data`; `vote_ep_on_fini` itself null-checks `ep->data`). No other change to the function. - `src/controller/ha.h` — add `_Atomic int refcount;` to `ha_elect_ctx_t` (next to the existing `done_posted` guard field, same struct region). - `src/controller/ha.c`: - New static helper `ha_elect_ctx_release(ha_elect_ctx_t *ctx)` — decrement; on reaching 0, `pthread_mutex_destroy` + `sem_destroy` + `free`. Single choke point for both release kinds (endpoint completion, supersession). @@ -127,7 +127,7 @@ sequenceDiagram - `raft_cb_send_vote_request`: initialize `ectx->refcount` is NOT done here (ctx already exists, allocated earlier) — this site only increments (`atomic_fetch_add_explicit(&ectx->refcount, 1, memory_order_relaxed)`) at the point `ep->data = vctx` is committed, and sets `ep->on_fini`. - `raft_cb_send_pre_vote_request`: identical increment + hook wiring at its `ep->data = vctx` site. - `ha_start_election_async`: at carrier allocation, initialize `ctx->refcount = 1`. At the multi-node publish site, change `atomic_store_explicit(&ha->elect_ctx, ctx, memory_order_release)` to `atomic_exchange_explicit(&ha->elect_ctx, ctx, memory_order_acq_rel)` capturing the return as `old_ctx`; if `old_ctx` non-NULL, call `ha_elect_ctx_release(old_ctx)`. The two existing fast-exit `free(ctx)` paths (no-local-config, single-node) are unchanged — they never share the ctx, so a direct free at refcount 1→destroy is still correct (or, for consistency and to avoid a second free idiom, route them through `ha_elect_ctx_release(ctx)` too — same result, one less code path to reason about). - - `ha_start_prevote_async`: identical `refcount = 1` init at allocation; identical `atomic_exchange_explicit` + `ha_elect_ctx_release(old_ctx)` at its publish site (the "Do NOT free a prior ctx" comment site, `ha.c:~6899`), comment updated to describe the new mechanism instead of documenting the leak as deliberate. + - `ha_start_prevote_async`: identical `refcount = 1` init at allocation; identical `atomic_exchange_explicit` + `ha_elect_ctx_release(old_ctx)` at its publish site (the former "Do NOT free a prior ctx" comment site, now `ha.c:~8042`), comment updated to describe the new mechanism instead of documenting the leak as deliberate. ### No New Files This is a localized extension to two existing files (`ha.c`/`ha.h`) plus a minimal, generic addition to the shared endpoint abstraction (`netio.h`/`netio.c`). No new translation units. diff --git a/.kiro/specs/ha-entity-snapshots/design.md b/.kiro/specs/ha-entity-snapshots/design.md index d5eafc7b..d7b42bd2 100644 --- a/.kiro/specs/ha-entity-snapshots/design.md +++ b/.kiro/specs/ha-entity-snapshots/design.md @@ -22,7 +22,9 @@ provisioned and already replicated independently via `push_config_to_standbys` (`config_repl.c`) — out of this spec's boundary. Only a queue's runtime admin status (open/closed/active/inactive), the one piece `WAL_QUEUE_CTRL` tracks and config replication does not carry, is this spec's concern — restored via the existing `restore_queue_state(name, status)`, -no capnp involved on the queue side at all. +no capnp involved on the queue side at all. (Note: the on-disk wire format was later migrated to a +capnp `QueueAdminState` frame — superseded by config-upgrade-snapshot, see that spec's design.md; +`restore_queue_state`'s own `(name, status)` call contract is unchanged.) ### Goals - A checkpoint written at LSN X contains every live/retained job and queue record as of X. @@ -69,8 +71,10 @@ no capnp involved on the queue side at all. ### Allowed Dependencies - Existing capnp codegen for `Job` (`encode_Job`/`write_Job`/`decode_Job`, already generated from `src/schema/controller.capnp`). Queue entries need no capnp — `restore_queue_state`'s contract - is a plain `(name, status)` pair (see Architecture below). -- `restore_job` (`job.c:96`) and `restore_queue_state` (`queue.c:208`) — existing, unmodified + is a plain `(name, status)` pair (see Architecture below). (Note: config-upgrade-snapshot later + added a capnp `QueueAdminState` frame as the on-disk encoding of that same pair — see Data + Models below.) +- `restore_job` (`job.c:274`) and `restore_queue_state` (`queue.c:208`) — existing, unmodified entity-restore functions already used by WAL replay. - The `collect_jobs`/`collect_queues` enumeration pattern (`dataex.c`) and the `job_should_retain`/`worker_should_retain` retention predicates (`persist.c`). @@ -97,7 +101,7 @@ no capnp involved on the queue side at all. ### Existing Architecture Analysis -- `persist_checkpoint` (persist.c:1763) already sequences correctly for Requirement 3: it calls +- `persist_checkpoint` (persist.c:2820) already sequences correctly for Requirement 3: it calls `write_snapshot`, and only calls `trim_wal_segments` if `write_snapshot` returned 0 (`if (rc != 0) goto done;`, skipping both trim and `rotate_snapshots`). **This means once `write_snapshot` only returns 0 when the entity payload is fully and durably written, the @@ -105,12 +109,12 @@ no capnp involved on the queue side at all. variable is needed in `trim_wal_segments`/`repl_trim_lsn`.** This is a refinement over the initial gap analysis (`research.md`), which assumed a new trim-floor input was needed. - The one retention hazard `write_snapshot`'s success does *not* already guard is - `rotate_snapshots` (persist.c:1672): it keeps only `snap_keep_n` newest snapshot *files* by + `rotate_snapshots` (persist.c:2677): it keeps only `snap_keep_n` newest snapshot *files* by filename sort and deletes the rest, with no awareness of an in-flight transfer. If a checkpoint cadence faster than a slow transfer rotates out the file a standby is still reading from a chunked wire transfer, that transfer breaks. This is the one genuinely new guard this design adds. -- `ha_handle_snapshot_chunk` (ha.c:2529-2609) already has the right shape for Requirement 4: it +- `ha_handle_snapshot_chunk` (ha.c:3635-3794) already has the right shape for Requirement 4: it only advances `ha->applied_lsn` after `persist_load_snapshot` returns success. Once `persist_load_snapshot`/`load_snapshot` correctly gate their own success on entity restoration (Requirement 2), this call site needs no change. @@ -119,7 +123,8 @@ no capnp involved on the queue side at all. per-entity lock** — verified via `restore_queue_state` (queue.c:208), which takes `lock(q, RTTI_LOCK_WRITE)` on the individual `queue_t*` after already releasing the table lock from its own `key_find`, and via `job.c`'s state-transition sites (e.g. lines 1006, 1093, 1137), - which take `lock(j, RTTI_LOCK_WRITE)` on the individual `job_t*`. The table lock only excludes + which take `lock(j, RTTI_LOCK_WRITE)` on the individual `job_t*` (line numbers shift as job.c + grows; verify by grepping `lock(j, RTTI_LOCK_WRITE)` if re-checking this). The table lock only excludes concurrent insert/delete of table entries; it does **not** exclude a concurrent per-entity writer mutating an entity's fields mid-walk. Requirement 1.2's point-in-time consistency therefore requires the entity-collection helper to additionally take each entity's own @@ -135,7 +140,7 @@ flowchart TB subgraph Primary["Primary — checkpoint write path"] PC[persist_checkpoint] --> WS[write_snapshot] WS --> CJ["collect+encode jobs\n(table READ lock + per-job READ lock)"] - WS --> CQ["collect name+status pairs\n(table READ lock + per-queue READ lock,\nno capnp)"] + WS --> CQ["collect name+status pairs\n(table READ lock + per-queue READ lock,\nnow capnp QueueAdminState --\nconfig-upgrade-snapshot)"] CJ --> TMP[tmp file + fdatasync + rename] CQ --> TMP TMP -->|rc==0| TRIM[trim_wal_segments] @@ -175,7 +180,9 @@ flowchart TB entry length-prefixed framing style, so no new capnp schema struct or codegen is required — jobs use the existing `Job` capnp codec, queues use a plain fixed `(name, status)` record since that is the entirety of `restore_queue_state`'s contract); the in-flight-LSN guard in - `rotate_snapshots`. + `rotate_snapshots`. (Note: "no new capnp schema struct" for queues held at this spec's close but + was superseded shortly after by config-upgrade-snapshot, which added a `QueueAdminState` capnp + struct and migrated the queue frame to it — see that spec's design.md.) - Steering compliance: no `fork()` used for the snapshot (per `product.md`'s "No fork() for snapshots" constraint) — consistency comes from the table read-lock, not process duplication. @@ -193,6 +200,9 @@ flowchart TB ### Modified Files - `src/controller/persist.c` — `write_snapshot`: append entity payload after the existing header — job frames (capnp `Job`) and queue frames (plain `(name, status)` records, no capnp). + (Note: queue frames were later migrated to capnp `QueueAdminState`, and host frames plus a + trailing config-bytes section were appended after them — superseded by config-upgrade-snapshot, + see Data Models below and that spec's design.md.) `load_snapshot`: restore entities before advancing `current_lsn`/`checkpoint_lsn`; detect legacy (position-only) files by remaining file size. `rotate_snapshots`: skip deleting any file whose LSN is currently in-flight. New static helpers: a snapshot entity-frame writer/reader @@ -294,6 +304,9 @@ sequenceDiagram - Appends, in order: a format-version marker, a job count and per-job length-prefixed capnp `Job` frames, a queue count and per-queue fixed `(name, status)` frames (no capnp — matches `restore_queue_state`'s exact contract; queue definition is config-owned, out of boundary). + (Note: superseded by config-upgrade-snapshot — queue frames are now capnp `QueueAdminState`, + and host frames plus a config-bytes section were appended after the queue section; see Data + Models below.) - Collects under `RTTI_LOCK_READ` on `c->jobmgr` (released before the queue pass begins); collects under `RTTI_LOCK_READ` on `c->qmgr` for queues. The two table locks are never held simultaneously (no new lock-ordering surface). **Within each table's held READ lock, also takes @@ -446,20 +459,40 @@ int persist_load_snapshot(persist_t* p, const char* path, uint64_t lsn, [uint32_t job_count] [job_count * { uint32_t payload_len; uint8_t capnp_Job_bytes[payload_len] }] [uint32_t queue_count] -[queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }] +[queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }] -- (Note: this raw + -- shape is the + -- layout AS OF + -- this spec's + -- close; see below) ``` +> **Note (superseded by config-upgrade-snapshot, see that spec's design.md):** the physical layout +> above is this spec's as-shipped shape only. config-upgrade-snapshot subsequently (1) migrated the +> queue frame from the raw `{name_len, name, status}` shape above to a length-prefixed capnp +> `QueueAdminState` frame (same shape as the job-frame loop, content unchanged: name + status), +> (2) appended a new host-frames section (`[uint32_t host_count][host_count * {payload_len, +> capnp HostAdminState bytes}]`) directly after the queue section, and (3) appended a trailing +> flat length-prefixed config-bytes blob after that (`[uint32_t cfg_len][cfg_len bytes]`, written +> on every checkpoint). `format_version` was bumped to `2` for this change. Current source: +> `write_snapshot`/`load_snapshot_entities` in `src/controller/persist.c`. + - Job frames are length-prefixed capnp `Job` messages, matching the framing style already used for WAL entries (`wal_entry_hdr_t` + payload) — no new schema/codegen for a one-off container. - Queue frames carry only what `restore_queue_state` can consume: a name and a status word — no capnp at all. A queue's *definition* (attrs, priority, thresholds, ACLs) is config-provisioned and already replicated via `push_config_to_standbys`; encoding it here would duplicate data this - spec does not own and has no restore path for. + spec does not own and has no restore path for. (Note: the wire encoding of this same + name+status pair was later moved to capnp — see the note above.) - A file ending exactly at the legacy header (no bytes remain) is unambiguously the old format — the discriminator is "does more data exist," not a flag bit, so old files need zero reinterpretation. - `format_version` is present for forward evolution (a future layout bumps this value and - `load_snapshot` dispatches on it) without breaking this design's readers. + `load_snapshot` dispatches on it) without breaking this design's readers. (Note: superseded by + config-upgrade-snapshot's actual choice, see that spec's design.md — when it bumped + `format_version` to `2` it made `load_snapshot` hard-reject any non-matching version with + `-EINVAL` rather than dispatch, on the reasoning that a version mismatch must never be silently + masked as an expected legacy shape or real corruption could go undetected. The legacy + (no-`format_version`-present) detection this bullet's neighbor describes is unaffected.) ### Consistency & Integrity - Per-entity encode happens while both the enumerating table's `RTTI_LOCK_READ` AND that entity's diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/design.md b/.kiro/specs/ha-heartbeat-committed-propagation/design.md index 312945b8..3dfc1dd4 100644 --- a/.kiro/specs/ha-heartbeat-committed-propagation/design.md +++ b/.kiro/specs/ha-heartbeat-committed-propagation/design.md @@ -50,11 +50,11 @@ ## Architecture ### Existing Architecture Analysis -- **HA heartbeat = bare `OP_nop`:** `primary_heartbeat_fn` (`ha.c:5256`) sends `xce_nop_req(ep)` with no body; the standby `OP_nop` handler (`cluster.c:300`) replies a `StateReport` and resets liveness (`ha_on_heartbeat_received`). It reads no request body. -- **Unused scaffolding:** `HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3}` and the `heartbeatmsg @25` Request-body member exist but have zero `.c` producers/consumers. -- **Committed knowledge today:** `last_known_committed_lsn` (`_Atomic`, `ha.h:170`) + `last_known_committed_config_tag` (plain, `ha.h:172`) written only at join (`cluster_apply_ctrlhello_reply` — a plain store that ALSO truncates via `ha_rejoin_reconcile`) and an election safe-commit path (`ha.c:925`). +- **HA heartbeat = bare `OP_nop`** (pre-feature baseline, since replaced by this spec — see File Structure Plan): `primary_heartbeat_fn` (now `ha.c:9665`; line drifted from unrelated growth added by later specs elsewhere in the file) sent `xce_nop_req(ep)` with no body; the standby `OP_nop` handler (now `cluster.c:427`, same drift) replied a `StateReport` and reset liveness (`ha_on_heartbeat_received`). It read no request body. +- **Unused scaffolding:** `HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3}` and the `heartbeatmsg @25` Request-body member exist but have zero `.c` producers/consumers. (Verified current: schema unchanged.) +- **Committed knowledge at baseline:** `last_known_committed_lsn` (`_Atomic`, now `ha.h:183`) written at join (`cluster_apply_ctrlhello_reply` — a store that ALSO truncates via `ha_rejoin_reconcile`) and at the election safe-commit path (`ha.c:925` area, still current); `last_known_committed_config_tag` (plain pre-task-1.1, now `ha.h:185`) written only at join. Correction: this bullet originally implied a third writer of `last_known_committed_config_tag` at the election path — verified false; that path writes `last_known_committed_lsn` and the *separate* `committed_config_tag` field (Req 8.1, set on quorum ACK), not `last_known_committed_config_tag`. - **Monotonic precedent:** `commit.c` advances `committed_lsn` with monotonic-max semantics — mirrored here. -- **Vote floor already consumes it:** `max(durable_floor, last_known_committed_lsn)` (`ha.c:2570`). +- **Vote floor already consumes it:** `max(durable_floor, last_known_committed_lsn)`, computed in `ha_vote_self_freshness` (now `ha.c:5310`; citation corrected — durable_floor is supplied by the voter-path caller, e.g. `cluster.c:544`). - **Opcode/body independence:** `OP_hello` already carries different bodies (`ctrlhellomsg` vs `replymsg`), so `OP_nop` + `heartbeatmsg` body is consistent with the existing pattern. ### Architecture Pattern & Boundary Map diff --git a/.kiro/specs/ha-heartbeat-liveness/design.md b/.kiro/specs/ha-heartbeat-liveness/design.md index 6107d121..e46ea42e 100644 --- a/.kiro/specs/ha-heartbeat-liveness/design.md +++ b/.kiro/specs/ha-heartbeat-liveness/design.md @@ -184,7 +184,7 @@ sequenceDiagram | 5.2 | EPOLLRDHUP immediate | `ha_on_primary_ctrl_fd_rdhup` unchanged | poll fn calls same fn on timeout | | 5.3 | Single-node no interference | `primary_heartbeat_fn` skips when no standbys | existing check | | 5.4 | No poll in INIT/ELECTING | `ha_arm_heartbeat_timeout_vtask` called only from STANDBY paths | callsite guard | -| 5.5 | INFO log on arm/reset/fire | `ha_arm_heartbeat_timeout_vtask`, `ha_on_heartbeat_received`, `liveness_poll_fn` | INFO log added to each | +| 5.5 | Log on arm/reset/fire | `ha_arm_heartbeat_timeout_vtask`, `ha_on_heartbeat_received`, `liveness_poll_fn` | as-built: INFO on arm, DEBUG3 on reset (avoids log spam per-message), WARN on fire — not uniformly INFO as originally planned | ## Components and Interfaces @@ -211,7 +211,7 @@ sequenceDiagram **Responsibilities:** - Read `ha->last_activity` (volatile `time_t`) - Compute `elapsed = now - ha->last_activity` -- If `elapsed >= heartbeat_timeout_ms`: call `ha_on_primary_ctrl_fd_rdhup(ha, c)`; do NOT re-arm +- If `elapsed >= heartbeat_timeout_ms`: call `ha_dispatch_rdhup_failover(ha, c)`; do NOT re-arm (Note: as-built this offloads to a vtask that calls `ha_on_primary_ctrl_fd_rdhup` off the delaymgr thread, rather than calling it inline as originally designed — added by raft-drive-async, see that spec's design.md) - Else: call `delay_task_ms(&c->delaymgr, poll_interval_ms, 1, liveness_poll_fn, ctx)` to re-arm - Log INFO on fire; suppress per-poll logging when resetting (only log on state change) @@ -239,20 +239,24 @@ typedef struct { #### `ha_disarm_heartbeat_timeout_vtask` (replaced) -**New implementation:** +**As-built implementation** (corrected from the original design below — freeing here would race a concurrently-running delaymgr task): - Set `s_liveness_ctx->cancelled = 1` (poll fn checks this and skips re-arm + failover) -- Free `s_liveness_ctx`; set to NULL +- Set `s_liveness_ctx` to NULL — does NOT free the ctx here +- The ctx is freed lazily by `liveness_poll_fn` itself, the next time it fires and observes `cancelled == 1` (production builds only; `XCE_TESTING` callers own stack-allocated ctxs) — avoids a use-after-free/double-free race with an in-flight delaymgr thread - No pthread_join needed — delaymgr tasks run to completion and are not cancellable mid-flight; the `cancelled` flag makes the next invocation a no-op #### `ha_on_heartbeat_received` (updated) -**New implementation:** +**As-built implementation** (the DEBUG3 log was added, not omitted as originally planned; a PreVote-cancellation block was appended later — see note): ```c void ha_on_heartbeat_received(ha_state_t *ha) { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); - ha->last_activity = ts.tv_sec; /* atomic on LP64 */ - /* (optional DEBUG3 log omitted to avoid log spam) */ + ha->last_activity = ts.tv_sec; + tlog(TLOG_DEBUG3, "HA: heartbeat received — last_activity updated to %ld", (long)ts.tv_sec); + /* + pre_vote_in_progress cancellation block (Note: added by ha-prevote task 4.2, + * see that spec's design.md — a fresh heartbeat mid-PreVote now abandons the + * round and re-attaches as CTRL_STANDBY) */ } ``` @@ -263,7 +267,7 @@ void ha_on_heartbeat_received(ha_state_t *ha) { **Extension to existing function:** - For each ep in `ha->standby_endpoints`: cast `ep->data` to `standby_ep_ctx_t` - Compute `elapsed = now - sctx->last_push_time` -- If `elapsed >= heartbeat_interval_ms`: call `xce_nop_req(ep)`, write `sctx->last_push_time = now` +- If `elapsed >= heartbeat_interval_ms`: call `xce_ha_heartbeat_req(ep, term, committed_lsn, committed_config_tag)`, write `sctx->last_push_time = now` (Note: superseded by ha-heartbeat-committed-propagation, see that spec's design.md — the plain `xce_nop_req(ep)` this spec originally specified was replaced with the richer heartbeat that also carries the primary's committed position; both send opcode `OPCode_nop`) - Else: skip NOP for this standby - Late-joiner connect path unchanged — `on_standby_connected` writes `last_push_time` on connect @@ -291,9 +295,9 @@ Called from `main.c` after `ha_config_t` is populated, before `ha_init()`. ### cluster.c — Opcode Dispatch Update -Remove `ep->fd == c->ha.primary_ctrl_fd` guard from `OPCode_nop` handler; replace with role check: call `ha_on_heartbeat_received` when `role == CTRL_STANDBY`. This avoids tight coupling on fd value while still restricting the call to standby context. +Remove `ep->fd == c->ha.primary_ctrl_fd` guard from `OPCode_nop` handler; replace with role check: call `ha_on_heartbeat_received` when `role == CTRL_STANDBY` (else `ha_discover_primary` when `role == CTRL_ELECTING`). This avoids tight coupling on fd value while still restricting the call to standby context. (Note: the handler now also gates this on `ha_msg_term_is_stale` — a term-staleness guard added by a later bug-1418-family fix — before checking role; see ha-term-safety's design.md.) -Also add `ha_on_heartbeat_received` call to `OPCode_hello` (ctrlhello) and `OPCode_replEntry` / `OPCode_walCatchupEntry` handlers — any primary-direction message resets the deadline (Req 1.3, 4.1). +Also add `ha_on_heartbeat_received` call to `OPCode_hello` (ctrlhello) and `OPCode_walCatchupEntry` handlers — any primary-direction message resets the deadline (Req 1.3, 4.1). (`OPCode_replEntry` is not dispatched through cluster.c's handler switch — that opcode is used internally in ha.c, not at this call site — so the originally-planned call there does not apply.) ## Data Models @@ -338,9 +342,9 @@ Also add `ha_on_heartbeat_received` call to `OPCode_hello` (ctrlhello) and `OPCo Both paths call the same `ha_on_primary_ctrl_fd_rdhup` function. That function is idempotent on `primary_ctrl_fd == -1` (already correct). The duplicate-fire risk is: - EPOLLRDHUP fires first → `ha_on_primary_ctrl_fd_rdhup` sets `primary_ctrl_fd = -1` and transitions to CTRL_ELECTING. The liveness poll fn then fires → checks `ha->role == CTRL_STANDBY` before proceeding; if role is already CTRL_ELECTING, the poll fn must skip the failover call. -**Fix**: `liveness_poll_fn` checks `atomic_load(&ha->role) == CTRL_STANDBY` before calling `ha_on_primary_ctrl_fd_rdhup`. If role has already transitioned away from CTRL_STANDBY, the poll fn skips the call and does not re-arm. +**Fix**: `liveness_poll_fn` checks `atomic_load(&ha->role) == CTRL_STANDBY` before calling `ha_dispatch_rdhup_failover`. If role has already transitioned away from CTRL_STANDBY, the poll fn skips the call and does not re-arm. -**EPOLLRDHUP cancels poll**: `ha_on_primary_ctrl_fd_rdhup` must call `ha_disarm_heartbeat_timeout_vtask` as one of its first steps, setting `cancelled = 1` on the poll ctx. This prevents the poll fn from firing a second failover if the delaymgr fires it concurrently. +**EPOLLRDHUP cancels poll**: `ha_on_primary_ctrl_fd_rdhup` calls `ha_disarm_heartbeat_timeout_vtask`, setting `cancelled = 1` on the poll ctx, before the failover proceeds. This prevents the poll fn from firing a second failover if the delaymgr fires it concurrently. (Note: as-built this call is no longer literally the first step — two guard blocks, an in-progress-election check and an RDHUP debounce window, run first and may return early without disarming; added by raft-ha-core-safety tasks 5.6/5.10b, see that spec's design.md.) ### `last_activity` Concurrency diff --git a/.kiro/specs/ha-peer-connection-reuse/design.md b/.kiro/specs/ha-peer-connection-reuse/design.md index 03220a2f..ba376e01 100644 --- a/.kiro/specs/ha-peer-connection-reuse/design.md +++ b/.kiro/specs/ha-peer-connection-reuse/design.md @@ -42,7 +42,7 @@ The entire change is confined to `ha.c` and `cluster.c`. iomgr, the capnp wire p - iomgr internals — zero changes to `src/lib/netio.c` or `src/include/netio.h` - Raft library — zero changes to `src/lib/raft/raft.c` - Heartbeat watchdog thread implementation — only the arming callsite changes -- The existing `ha_find_and_connect_to_primary` function — it remains for future use; the standby path no longer calls it after election +- The existing `ha_find_and_connect_to_primary` function — originally scoped to remain for future use; **drifted**: it was removed entirely during implementation (exceeds this design's original intent — see this spec's own research.md "Gap Re-Audit 2026-06-18", finding R4). The standby path no longer calls it, and the function no longer exists in the tree. ### Allowed Dependencies @@ -98,9 +98,12 @@ graph TD end subgraph Promoted_Connection_PRIMARY - F --> J[peer_ctrl_msg_handler role=COMMITTING or PRIMARY] - J --> K[repl_ack_decode: ha_repl_ack_received] + F --> J[peer_ctrl_msg_handler role=PRIMARY] + J --> K[repl_ack_decode_and_dispatch: no-op ack, INFO log] end + %% Note: CTRL_PRIMARY_COMMITTING was removed (superseded by ha-consistency-invariants / + %% ha-single-leader-role, 2026-06-11); ha_repl_ack_received no longer exists -- promotion + %% is now driven by the heartbeat decision loop + commit engine, not ack-counting on this ep. subgraph Standby_Passive H --> L[wait for ctrlhello on listener] @@ -111,9 +114,12 @@ graph TD subgraph Dispatcher D --> O[peer_ctrl_msg_handler] O -- role=ELECTING --> P[vote_reply_decode_and_dispatch] - O -- role=COMMITTING or PRIMARY --> K + O -- role=PRIMARY --> K + O -- role=STANDBY + PreVote round --> R[pre_vote_reply_decode_and_dispatch] O -- other role --> Q[drop packet] end + %% pre_vote_reply_decode_and_dispatch branch added later by ha-prevote (2026-07-18) -- + %% shares this dispatcher but was not part of this spec's original scope. ``` ### Technology Stack @@ -133,8 +139,8 @@ graph TD - `src/controller/ha.c` — All behavioral changes: `peer_ctrl_msg_handler`, `vote_ep_ctx_t` promoted lifetime, `ha_vote_entry_t.ep` tracking, `raft_cb_become_leader` promotion path, `raft_cb_become_follower` passive path + watchdog arm, remove `fetch_config_from_primary` delay task - `src/controller/ha.h` — Add `endpoint_t *ep` to `ha_vote_entry_t`; add `peer_ctrl_msg_handler` forward declaration or make it accessible via a new test-seam hook; update comments on `ha_find_and_connect_to_primary` noting standby post-election path no longer calls it -- `src/controller/cluster.c` — `handle_ctrlhello`: set `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY and ctrlhello arrives for the first time -- `src/controller/testsuite/test_ha_election.c` — Add tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello +- `src/controller/config_repl.c` — `handle_ctrlhello`: set `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY and ctrlhello arrives for the first time (Note: this design originally cited `cluster.c`; `handle_ctrlhello` actually lives in `config_repl.c` — drift already flagged in this spec's own research.md R4, corrected here) +- `src/controller/testsuite/test_ha_connection_reuse.c` — Tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello (Note: implemented under this filename, not `test_ha_election.c` as originally planned — that file is a pre-existing, unrelated quorum-election test suite) --- @@ -159,11 +165,18 @@ sequenceDiagram C->>C: votes[i].ep added to ha->standby_endpoints C->>P: push_config_to_standbys (on promoted ep) C->>P: ha_send_ctrlhello_to_all_standbys (on promoted ep) + C->>C: repl_add_standby registers promoted ep with WAL replicator (ha-commit-engine addition) P->>C: OPCode_replCatchUpDone on same ep - I->>C: peer_ctrl_msg_handler(role=COMMITTING) - C->>C: ha_repl_ack_received → ha_promote_to_primary + I->>C: peer_ctrl_msg_handler(role=PRIMARY) + C->>C: repl_ack_decode_and_dispatch: replCatchUpDone is now a no-op (ack-counting removed) ``` +> Note: the original `role=COMMITTING` step and `ha_repl_ack_received → ha_promote_to_primary` +> ack-counting tail no longer exist — `CTRL_PRIMARY_COMMITTING` and `ha_repl_ack_received` were +> deleted by `ha-consistency-invariants`/`ha-single-leader-role` (2026-06-11); promotion is now +> driven by the heartbeat decision loop + commit engine (`ha-commit-engine`, `ha-repl-hardening`), +> not by counting `replCatchUpDone` acks on the promoted ep. See those specs' design.md. + ### Election + Passive Wait (Loser Path) ```mermaid @@ -207,9 +220,9 @@ sequenceDiagram | 4.2 | Failed outbound vote connection → count peer as denied, no reconnect | Existing `vote_ep_err_handler` unchanged | Error | | 4.3 | New election → fresh outbound connections | `ha_start_election_async` creates new endpoints each time (unchanged) | Election Phase | | 5.1 | Primary pushes config via promoted connection | `push_config_to_standbys` uses `standby_endpoints` (populated from promoted eps) | Winner Path | -| 5.2 | WAL replication via promoted connections | Existing catch-up path uses `standby_endpoints` | Winner Path | +| 5.2 | WAL replication via promoted connections | Existing catch-up path uses `standby_endpoints`; majority-ack commit tracking now also registers the promoted ep with the replicator via `repl_add_standby` (Note: added by `ha-commit-engine`, 2026-06-12 — see that spec's design.md) | Winner Path | | 5.3 | Single-node self-elect without peer connections | Existing raft library single-node path unchanged | Election Phase | -| 5.4 | Committing timeout → degraded PRIMARY | Existing `committing_timeout_fn` unchanged | Winner Path | +| 5.4 | Committing timeout → degraded PRIMARY | `committing_timeout_fn` no longer exists (Note: deleted along with `CTRL_PRIMARY_COMMITTING` by `ha-consistency-invariants`, 2026-06-11; replaced by promotion-fence/own-term-gate machinery — `ha_arm_own_term_gate`, `ha_arm_promotion_fence_timer` — see that spec's design.md) | Winner Path | | 5.5 | No new ASAN errors | Promoted ep lifetime owned by `vote_ep_err_handler` / `raft_cb_become_follower` cleanup | All | --- @@ -222,7 +235,7 @@ sequenceDiagram | `ha_vote_entry_t.ep` field | ha.h | Track per-peer outgoing endpoint for promotion/close | 1.3, 1.4, 4.1 | | `raft_cb_become_leader` (extended) | ha.c | Promote vote eps → standby_endpoints | 1.3, 5.1, 5.2 | | `raft_cb_become_follower` (extended) | ha.c | Close vote eps, arm watchdog, remove fetch_config call | 1.4, 2.1, 2.3, 2.4, 3.3 | -| `handle_ctrlhello` (extended) | cluster.c | Set primary_ctrl_fd from ctrlhello ep | 3.1, 3.2 | +| `handle_ctrlhello` (extended) | config_repl.c (design originally cited cluster.c — corrected) | Set primary_ctrl_fd from ctrlhello ep | 3.1, 3.2 | ### HA Election Layer @@ -239,9 +252,10 @@ sequenceDiagram - Set once in `on_vote_connected`; never reassigned - Reads `ha->role` atomically at dispatch time - Routes `CTRL_ELECTING` to vote reply decode (existing `vote_reply_msg_handler` logic, factored out as `vote_reply_decode_and_dispatch`) -- Routes `CTRL_PRIMARY_COMMITTING` / `CTRL_PRIMARY` to repl-ack decode (`repl_ack_decode_and_dispatch`, calls `ha_repl_ack_received`) +- Routes `CTRL_PRIMARY` to repl-ack decode (`repl_ack_decode_and_dispatch`). (Note: `CTRL_PRIMARY_COMMITTING` no longer exists — removed by `ha-consistency-invariants`/`ha-single-leader-role`, 2026-06-11; `repl_ack_decode_and_dispatch`'s `OPCode_replCatchUpDone` branch is now an INFO-logged no-op and `ha_repl_ack_received` was deleted along with the ack-counting promotion path — see those specs' design.md) +- Also routes `CTRL_STANDBY` with an in-progress PreVote round to `pre_vote_reply_decode_and_dispatch` — added later by `ha-prevote` (2026-07-18), not part of this spec's original scope - Any other role: drop packet (free packet, return 0) -- Runs in iomgr thread — must not block +- Runs in iomgr thread — must not block, for the default (dispatch-flag-OFF) path; `raft-drive-async` (2026-07-18) later added an optional vtask-pool dispatch mode (`ha_peerctrl_dispatch_enabled`) that can run this body off the iomgr thread — see that spec's design.md **Contracts**: State [ x ] @@ -251,20 +265,23 @@ sequenceDiagram /* Signature: matches msg_handler_t = int (*)(void*) */ static int peer_ctrl_msg_handler(void *ep_raw); -/* Internal helpers (static, same file) */ -static int vote_reply_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx); -static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx); +/* Internal helpers (static, same file) — current signatures (corrected; both + * gained an io_packet_t *pkt parameter, and vote_reply_decode_and_dispatch + * dropped its endpoint_t *ep parameter, as part of the raft-drive-async + * pkt-ownership refactor, 2026-07-18) */ +static int vote_reply_decode_and_dispatch(vote_ep_ctx_t *vctx, io_packet_t *pkt); +static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx, io_packet_t *pkt); ``` - State model: branches on `atomic_load_explicit(&ha->role, memory_order_acquire)` - ELECTING path: existing vote-reply capnp decode → `vote_reply_dispatch` → `raft_on_vote_reply` -- COMMITTING/PRIMARY path: decode `OPCode_replCatchUpDone` → `ha_repl_ack_received(ha, c)` +- PRIMARY path: decode `OPCode_replCatchUpDone` — now a no-op INFO log (ack-counting/`ha_repl_ack_received` removed, see note above); the same handler also decodes heartbeat-reply `stateReport` for convergence tracking - Other: free packet, no-op **Implementation Notes** - `vote_reply_decode_and_dispatch` is exactly the body of the existing `vote_reply_msg_handler` (factored out); the XCE_TESTING term-fencing and `xce_vote_reply_hook` seams are preserved -- `repl_ack_decode_and_dispatch` decodes an inbound capnp packet expecting `OPCode_replCatchUpDone` then calls `ha_repl_ack_received`; the opcode check guards against unexpected packets on the promoted endpoint during the COMMITTING phase +- `repl_ack_decode_and_dispatch` decodes an inbound capnp packet; `OPCode_replCatchUpDone` is now a no-op (superseded — see note above), and it separately decodes heartbeat-reply `stateReport` messages for convergence tracking. The original "COMMITTING phase" framing no longer applies since that role was removed. #### raft_cb_become_leader (extended) @@ -280,6 +297,7 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx); - Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)` - If `votes[i].ep != NULL` (connect succeeded): add to `ha->standby_endpoints` under `ha->lock` - Existing `push_config_to_standbys` and `ha_send_ctrlhello_to_all_standbys` calls are unchanged; they iterate the now-populated array +- (Note: `ha-commit-engine`, 2026-06-12, added a further registration in this same loop — each promoted ep is also registered with the WAL replicator via `repl_add_standby(&c->persist.repl, ...)`, which is what majority-ack commit tracking now uses; `standby_endpoints` remains the config-push/ctrlhello target only. See that spec's design.md.) **Implementation Notes** @@ -356,9 +374,9 @@ typedef struct { - Winner: `ep` is transferred to `standby_endpoints`; winner does NOT call `endpoint_fini` on promoted eps - Loser: `ep` is closed via `endpoint_fini` in `raft_cb_become_follower`; set to NULL after close -### vote_ep_ctx_t (unchanged) +### vote_ep_ctx_t -No structural changes. The `promoted` flag is NOT added — role is read from `ha->role` at dispatch time, which is the authoritative state. The ctx carries `ha`, `c`, `peer_idx` which are sufficient for both phases. +The `promoted` flag is NOT added — role is read from `ha->role` at dispatch time, which is the authoritative state, as this design intended. (Note: despite the original "(unchanged)" heading, the struct is no longer literally unchanged since this spec closed — `ha-term-safety` (2026-07-17) added `my_last_log_term`/`my_last_log_index` fields, and `ha-elect-ctx-lifecycle` (2026-07-18) changed its lifetime contract: the endpoint's `on_fini` hook now calls `ha_elect_ctx_release` to decrement a refcount on the owning `ha_elect_ctx_t`, rather than the simple election-cleanup-only lifetime this design describes. See those specs' design.md.) --- @@ -371,15 +389,15 @@ Treat connection failures and packet decode errors identically to the existing v ### Error Categories **Connect failure** (vote_ep_err_handler): -- Existing behavior preserved: `handle_request_vote_reply(..., 0, ...)` counts as denied vote -- Promoted connection disconnect: `vote_ep_err_handler` fires when the standby drops. For COMMITTING/PRIMARY role, remove `ep` from `ha->standby_endpoints` under `ha->lock`; call `ha_repl_ack_received` or degrade gracefully based on `n_peers` +- Existing behavior preserved: a connect-time or post-connect failure counts as a denied vote through the production tally (`vote_reply_dispatch`), not the legacy `handle_request_vote_reply` path (hardened later by `raft-adapter-conformance`) +- Promoted connection disconnect: `vote_ep_err_handler` fires when the standby drops. For `PRIMARY` role (Note: the `COMMITTING` role named here was removed — see the peer_ctrl_msg_handler note above), remove `ep` from `ha->standby_endpoints` under `ha->lock`; current code also calls `repl_remove_standby` to deregister from the WAL replicator (`ha-commit-engine` addition) — `ha_repl_ack_received` no longer exists **Packet decode error in repl_ack_decode_and_dispatch**: - Drop packet, return 0 (iomgr continues monitoring endpoint) - Log WARN with opcode for debugging **Unexpected role in peer_ctrl_msg_handler**: -- Drop packet, return 0 (CTRL_INIT, CTRL_STANDBY on outgoing ep: should not happen normally) +- Drop packet, return 0 (CTRL_INIT: should not happen normally; CTRL_STANDBY on outgoing ep is now an EXPECTED case when a PreVote round is in progress — routed to `pre_vote_reply_decode_and_dispatch` rather than dropped, added by `ha-prevote`, 2026-07-18) - Log WARN ### Monitoring diff --git a/.kiro/specs/ha-prevote/design.md b/.kiro/specs/ha-prevote/design.md index 5f2c33a4..41b788eb 100644 --- a/.kiro/specs/ha-prevote/design.md +++ b/.kiro/specs/ha-prevote/design.md @@ -56,7 +56,7 @@ The election path is split across two layers (preserved by this design): - **raft library** (`src/lib/raft/raft.c`, `raft.h`) — the canonical, TLA+-checkable state machine. Owns term, `voted_for`, role, and the pure vote-decision logic. Communicates outward only through `raft_callbacks_t` (no controller dependency). Today: `raft_start_election` increments+persists the term before any RPC; `raft_on_vote_request` adopts a higher term and steps down a leader in **Step 2 — before** the `on_vote_eligible` gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side. -- **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election. +- **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election. (Note: `handle_request_vote_reply` is now the test-mode-only reply path; production reply collection — including endpoint/connect failures — routes through `vote_reply_dispatch`/`pre_vote_reply_dispatch` as the sole tally driver, per bug-1423/raft-adapter-conformance. `handle_request_vote_reply` is retained only for `XCE_TESTING` callers.) ### Architecture Pattern & Boundary Map @@ -128,7 +128,7 @@ graph TB - `src/schema/controller.capnp` — add two `OPCode` enum entries: `preVoteRequest`, `preVoteReply`. Reuse existing `RequestVoteRequest` / `RequestVoteReply` structs and their `Request.body` union arms (`requestvotemsg` / `requestvotereplymsg`); no new payload structs. - `src/lib/raft/raft.h` — add pre-vote return codes (e.g. `RAFT_PRE_VOTE_WON`, `RAFT_PRE_VOTE_LOST`, `RAFT_PRE_VOTE_PENDING`), an optional `on_send_pre_vote_request` callback in `raft_callbacks_t`, pre-vote tally fields in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`), and declarations for the three new functions. - `src/lib/raft/raft.c` — add `raft_begin_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply` (all pure: no `on_persist`, no `term`/`voted_for`/`role` mutation). `raft_on_pre_vote_request` reuses `on_vote_eligible`. **`raft_on_pre_vote_reply` owns the PreVote tally in `raft_state_t` (mirroring `raft_on_vote_reply`) and returns `WON/LOST/PENDING` — it never fires `on_become_leader`/`on_become_follower`.** -- `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here). +- `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here). (Note: as built, this handle is a module-static `prevote_round_ctx_t * _Atomic s_prevote_round_ctx` in `ha.c`, not a field added to `ha_state_t`/`ha.h` — `ha.h` was not modified.) - `src/controller/ha.c` — add `ha_begin_failover` (common entry), `ha_start_prevote_async` (calls `raft_begin_pre_vote`, arms round timeout), and `pre_vote_reply_dispatch` (calls `raft_on_pre_vote_reply`, mirroring `vote_reply_dispatch → raft_on_vote_reply`); on `RAFT_PRE_VOTE_WON` perform truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST`/round-timeout stay `CTRL_STANDBY`, re-arm liveness, log abandonment. Modify `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn` to call `ha_begin_failover` (PreVote first; **no** truncate/`CTRL_ELECTING` until PreVote wins); modify `ha_on_heartbeat_received` to cancel an in-progress PreVote and re-attach. - `src/controller/cluster.c` — add `OPCode_preVoteRequest` and `OPCode_preVoteReply` dispatch cases mirroring the real-vote cases, calling `raft_on_pre_vote_request` (no term adoption) and routing replies to `pre_vote_reply_dispatch`. @@ -183,6 +183,8 @@ flowchart TD Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Step G is exactly the existing dispatcher freshness floor (`ha_vote_self_freshness`). The only PreVote-specific rule is the **absence of any state mutation** on either grant or deny. +(Note: superseded by `raft-commit-safety`, see that spec's design.md — `raft_on_pre_vote_request` now also runs an internal `raft_log_ok(lastLogTerm, lastLogIndex)` check between step F and step B/C/E, denying stale-log candidates before the eligibility gate is reached; not shown in the flowchart above.) + ## Requirements Traceability | Requirement | Summary | Components | Interfaces / Flows | @@ -233,7 +235,7 @@ Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Ste **Responsibilities & Constraints** - `raft_begin_pre_vote`: compute prospective term = `term + 1` **without** assigning it to `s->term` or persisting; set `pre_vote_in_progress`, `pre_votes_granted = 1` (self); emit `on_send_pre_vote_request(ctx, peer_idx, prospective_term)` per peer. -- `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote). +- `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote). (Note: superseded by `raft-commit-safety`, see that spec's design.md — that spec (closed 2026-06-25, i.e. before this spec's own close) added a `raft_log_ok(lastLogTerm, lastLogIndex)` gate as an internal Step 2 inside `raft_on_pre_vote_request` itself, run **after** the stale-term check and **before** `on_vote_eligible`. The function signature grew two params — `cand_last_log_term`, `cand_last_log_index` — see the corrected Service Interface below. The dispatcher's `ha_vote_self_freshness` floor check in `cluster.c` still also runs, so freshness is now enforced at both layers.) - `raft_on_pre_vote_reply`: **owns the PreVote tally** in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`), mirroring `raft_on_vote_reply`; return `RAFT_PRE_VOTE_WON` at quorum, `RAFT_PRE_VOTE_LOST` when all replied without quorum, else `RAFT_PRE_VOTE_PENDING`. Unlike `raft_on_vote_reply`, it **never** fires `on_become_leader`/`on_become_follower` and makes no role/term change — the controller acts on the returned status. - **Invariant**: across all three functions, `s->term`, `s->voted_for`, `s->voted_for_term`, and `s->role` are unchanged, and `on_persist` is never invoked. @@ -243,11 +245,13 @@ Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Ste ```c /* All pure: no persist, no term/role/voted_for mutation. */ int raft_begin_pre_vote(raft_state_t *s); /* sends pre-votes, self-grant in raft_state_t */ -int raft_on_pre_vote_request(raft_state_t *s, - int from_idx, uint64_t cand_term); /* 1=grant, 0=deny */ +int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term, + uint64_t cand_last_log_term, + uint64_t cand_last_log_index); /* RAFT_GRANT=1 / RAFT_DENY=0 */ int raft_on_pre_vote_reply(raft_state_t *s, int peer_idx, int granted); /* tally in raft_state_t -> WON / LOST / PENDING */ ``` +(Note: superseded by `raft-commit-safety` — `raft_on_pre_vote_request` gained the `cand_last_log_term`/`cand_last_log_index` params and an internal `raft_log_ok` freshness check; see the Responsibilities & Constraints note above.) - Preconditions: `raft_begin_pre_vote` requires `role == RAFT_FOLLOWER`. - Postconditions: on `RAFT_PRE_VOTE_WON` the caller invokes the unchanged `raft_start_election`. - Invariants: persistent state untouched (asserted in `test_raft.c`). diff --git a/.kiro/specs/ha-repl-hardening/design.md b/.kiro/specs/ha-repl-hardening/design.md index c74e8acc..df490e7a 100644 --- a/.kiro/specs/ha-repl-hardening/design.md +++ b/.kiro/specs/ha-repl-hardening/design.md @@ -33,7 +33,7 @@ Enforced at three stages: | Stage | Mechanism | When | |-------|-----------|------| -| **Bootstrap** | Heartbeat reply convergence check | After winning election, before serving client traffic | +| **Bootstrap** | Heartbeat reply convergence check *(Note: superseded by `ha-single-leader-role`, closed 2026-06-11 — `CTRL_PRIMARY_COMMITTING` and the convergence-gated bootstrap were deleted; the elected node now enters `CTRL_PRIMARY` directly and write-confirmation is gated by a `WAL_NOOP` own-term commit through the commit engine, not by standby convergence tracking. See that spec's design.md.)* | After winning election, before serving client traffic | | **Ongoing WAL** | `repl_wait_majority_ack` per entry (existing) | On every job submit, state transition, queue change | | **Ongoing Config** | `push_config_to_standbys` + majority ack (existing) | On every config upgrade | @@ -47,7 +47,7 @@ Enforced at three stages: - WAL catch-up entry application on standby side - Batch WAL entry encoding and transfer - Snapshot creation and transfer -- COMMITTING→serving transition gated on majority convergence +- COMMITTING→serving transition gated on majority convergence *(Note: superseded by `ha-single-leader-role` — the COMMITTING role and this gate were removed entirely; see note under Majority Consistency Invariant above)* ### Out of Boundary @@ -106,11 +106,13 @@ For each standby in standby_endpoints: → transition to serving state ``` +*(Note: step 4 and the COMMITTING role it references were removed by `ha-single-leader-role` (closed 2026-06-11) — see note under Majority Consistency Invariant above. Steps 1-3 (per-standby config/snapshot/WAL decision) remain accurate and are unaffected.)* + **Key insight**: The same decision tree handles all standby states — fresh bootstrap (tag=0, lsn=0), catch-up after disconnect (tag=current, lsn=stale), and converged (tag=current, lsn=current). No separate code paths needed. ### handle_ctrlhello — Reduced to Connection Setup -Before this change, `handle_ctrlhello` (config_repl.c:1359) orchestrated replication: +Before this change, `handle_ctrlhello` (then at config_repl.c:1359, now config_repl.c:2128 post-implementation) orchestrated replication: - Registered standby in replication set - Checked snapshot need - Streamed WAL entries @@ -164,7 +166,7 @@ sequenceDiagram | Standby only reports state, never requests replication | Eliminates standby→primary request messages; standby is purely passive | | Config/snapshot/WAL decided from `(tag, lsn)` alone | Two scalars encode all replication state needed | | handle_ctrlhello reduces to connection setup | Clean separation: connection lifecycle vs replication logic | -| One action per heartbeat cycle per standby | Avoids flooding a standby with config + WAL simultaneously; natural pacing. `stream_wal_entries()` enqueues all batch messages via `xce_request()` (non-blocking) in a single invocation — "one action" means no further decisions for this standby until the next heartbeat reply confirms the new LSN | +| One action per heartbeat cycle per standby | Avoids flooding a standby with config + WAL simultaneously; natural pacing. `ha_stream_wal_to_standby()` (named `stream_wal_entries()` in this design; actual as-built name differs) enqueues all batch messages via `xce_request()` (non-blocking) in a single invocation — "one action" means no further decisions for this standby until the next heartbeat reply confirms the new LSN | ## File Structure Plan @@ -223,16 +225,20 @@ Action ∈ { | Requirements | 1.1, 1.3, 1.4 | | Location | `cluster.c` OPCode_nop handler | -**Cap'n Proto Schema Change**: +**Cap'n Proto Schema Change** (as built — a named struct referenced from the union, not an inline group as originally sketched here): ```capnp # In RequestReply.data union: -stateReport :group { - appliedLsn @15 :UInt64; # standby's last applied WAL LSN - configTag @16 :UInt64; # standby's current config version tag +stateReportMsg @15: StateReport; + +struct StateReport { + appliedLsn @0 :UInt64; # standby's last applied WAL LSN + configTag @1 :UInt64; # standby's current config version tag + term @2 :UInt64; # standby's current raft term (added post-close by bug-1421, + # AE-reject term-discovery — not part of this design) } ``` -**Implementation**: In the `OPCode_nop` handler, replace `data_which = nodata` with `data_which = stateReport`, populate from `ha->applied_lsn` and `myconf->config_version_tag`. +**Implementation**: In the `OPCode_nop` handler, `data_which` is set to `RequestReply_data_stateReportMsg`, populated from `ha->applied_lsn`, `myconf->config_version_tag`, and (post-close addition) `ha->term`. Function: `ha_nop_reply_state_report()` in `ha.c`. ### Component: Per-Standby State Tracking @@ -244,7 +250,7 @@ stateReport :group { **Fields**: ```c -#define MAX_HA_CANDIDATES 8 +/* MAX_HA_CANDIDATES is 4 (not 8), defined in src/include/config.h — not ha.h */ uint64_t standby_reported_tag[MAX_HA_CANDIDATES]; uint64_t standby_reported_lsn[MAX_HA_CANDIDATES]; bool standby_converged[MAX_HA_CANDIDATES]; @@ -265,6 +271,8 @@ bool standby_converged[MAX_HA_CANDIDATES]; Wires the current stub (`OPCode_walCatchupEntry` returns 0) to `apply_wal_entry()`. For batch entries, applies each in order, advances `applied_lsn` after the full batch is durably written. Error entries are logged and skipped. +*(Note: `ha-term-safety` (closed 2026-07-17) materially extended the `WalCatchupEntry` wire message beyond this description — it now carries a full Raft AppendEntries-conformant envelope (`term`, `prevLogIndex`, `prevLogTerm`, `leaderCommit`, `leaderCurrentTerm`, and a real `entries: List(WalCatchupEntry)` for batching) so the standby intake runs through `raft_on_append_entries_request`'s logOk/conflict-truncation gate rather than blind sequential apply. See that spec's design.md and bug-1067/bug-1421.)* + **Durability**: `applied_lsn` is not persisted to disk. On controller restart the standby always reports `lsn=0`; the primary drives full re-bootstrap (config push → WAL streaming → converged) via the unified decision loop. This is correct and expected — the decision loop handles `lsn=0` identically to a fresh standby. ### Component: Snapshot Transfer @@ -283,6 +291,8 @@ The unified decision loop checks two conditions on each heartbeat reply: 2. **Excessive WAL backlog** (heuristic): `committed_lsn - standby_lsn > wal_streaming_threshold` (configurable, default 10M entries). Even within the WAL window, streaming millions of entries would take too long — a snapshot is faster. + *(Note: unverified against current code — `wal_streaming_threshold` and this backlog-size heuristic do not exist anywhere in the source tree. `ha_evaluate_repl_action()` (ha.c) implements only condition 1 above (`standby_lsn + 1 < oldest_wal_lsn`); condition 2 was never built. Not attributable to any later spec — appears to have simply not been implemented.)* + **Design principle**: Prefer WAL retention over snapshot. WAL entries are incremental (O(changes per operation)); a snapshot is O(total state size). Operators should configure `wal_retention_hours` large enough that snapshots are rare. Snapshot transfer is the fallback for long-disconnected standbys. #### Transfer Strategy: Chunked + Resumable @@ -312,14 +322,18 @@ Primary Standby │ ←── heartbeat reply ────────────── │ reports new lsn ``` -**Chunk format** (per `OPCode_snapshotTransfer` message): +**Chunk format** (per `OPCode_snapshotTransfer` message, as built — `struct SnapshotTransfer` in `controller.capnp`): ``` -checkpoint_lsn :UInt64 — LSN this snapshot corresponds to -chunk_index :UInt32 — 0-based chunk number -chunk_offset :UInt64 — byte offset in the full snapshot -total_size :UInt64 — total snapshot size in bytes -is_last :Bool — true for the final chunk -data :Data — chunk payload (64KB max) +snapshot_lsn :UInt64 — LSN this snapshot corresponds to (design.md named it checkpoint_lsn) +payload :Data — chunk payload (64KB max) (design.md named it data) +chunk_index :UInt32 — 0-based chunk number +chunk_offset :UInt64 — byte offset in the full snapshot +total_size :UInt64 — total snapshot size in bytes +is_last :Bool — true for the final chunk +checkpoint_term :UInt64 — (Note: field added by `raft-ha-core-safety`, closed 2026-07-09, + not part of this design — the leader's term for snapshot_lsn, + reconciled into the receiver's per-LSN term index after install + so the snapshot boundary establishes a verifiable (term, lsn) pair) ``` **Failure handling**: Mid-transfer reconnect always restarts snapshot from chunk 0 — the standby discards its partial `snap.part` file on reconnect and the primary re-creates the checkpoint from scratch. Resumption is intentionally not implemented: snapshots are rare (WAL retention is the primary path), and the extra protocol state (`checkpoint_lsn`, `last_chunk_index`) in `stateReport` would complicate the common heartbeat path for an edge case. The existing reconnect model (standby re-sends ctrlhello, primary drives next heartbeat cycle) naturally handles this without schema changes. Req 4.4's "retry by reconnecting" is satisfied by this restart-on-reconnect approach. @@ -347,7 +361,9 @@ This is deferred — the chunked approach handles the common case, and WAL reten Accumulates entries up to batch limit (64KB/256 entries), sends as single message. Standby applies all entries in batch, advances `applied_lsn` after batch complete. -`stream_wal_entries()` enqueues all batch `OPCode_walCatchupEntry` messages via `xce_request()` (non-blocking send-buffer enqueue) in a single handler invocation — it never blocks the vtaskmgr thread. "One action per heartbeat cycle" means the primary makes no further replication decisions for this standby until the next heartbeat reply arrives confirming the standby's updated `applied_lsn`. +`ha_stream_wal_to_standby()` (this design's `stream_wal_entries()`) enqueues all batch `OPCode_walCatchupEntry` messages via `xce_request()` (non-blocking send-buffer enqueue) in a single handler invocation — it never blocks the vtaskmgr thread. "One action per heartbeat cycle" means the primary makes no further replication decisions for this standby until the next heartbeat reply arrives confirming the standby's updated `applied_lsn`. + +*(Note: as built, each `WalCatchupEntry` message also carries the Raft AppendEntries envelope (`term`, `prevLogIndex`, `prevLogTerm`, `leaderCommit`, `leaderCurrentTerm`) added by `ha-term-safety` (closed 2026-07-17) — see the WAL Catch-Up Handler component above.)* ## System Flows @@ -364,6 +380,8 @@ raft_cb_become_leader(): 5. committed_lsn = safe_committed_lsn ``` +*(Note: step 4 is stale — bug-1060 (fixed 2026-06-24, closely tied to `raft-commit-safety`'s Leader Completeness work, closed 2026-06-25) proved a bare `safe_committed_lsn` truncation can cut a new leader's own durable committed entries when a slower peer sits in the granting majority. As built, `raft_cb_become_leader` (ha.c:1615) truncates to `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor) = max(safe_committed_lsn, own durable tip)` via `ha_compute_safe_committed_lsn()` (ha.c:1144) — a new leader never truncates its own durable log. See `src/controller/CLAUDE.md` Leader Completeness section.)* + **Raft TLA+ alignment**: Raft's `BecomeLeader` resets `matchIndex[j] = 0` for all followers and determines `commitIndex` via subsequent `AppendEntries` round-trips. Our optimization uses the vote replies (which carry the Raft-equivalent of each voter's lastLogIndex) to compute the safe commit point with zero additional round-trips. This is safe because any committed entry was on a majority and at least one voter in the winning quorum has it. ### Flow 1: Full Standby Bootstrap (config_tag=0, applied_lsn=0) @@ -391,7 +409,7 @@ Standby reports `(tag=T, lsn=100)`, primary's `oldest_wal_lsn=5000`. Standby pre | 3.1–3.5 | `ha.c`/`cluster.c`: wire `OPCode_walCatchupEntry` → `apply_wal_entry` | | 4.1–4.5 | `config_repl.c`: snapshot creation + `OPCode_snapshotTransfer` | | 5.1–5.5 | `config_repl.c`: batch accumulation in WAL streaming loop | -| 6.1–6.5 | `ha.c`: heartbeat-driven majority check; committing timeout as 60s fallback | +| 6.1–6.5 | `ha.c`: heartbeat-driven majority check; committing timeout as 60s fallback *(Note: COMMITTING and its 60s timeout were removed by `ha-single-leader-role` — see note under Majority Consistency Invariant)* | | 7.1–7.5 | INFO/WARN log statements throughout new code paths | ## Data Models @@ -423,7 +441,7 @@ ha_state_t | Snapshot transfer fails | Standby logs ERROR, reconnects; primary retries next time standby appears in heartbeat loop | | Heartbeat reply lost | Primary re-sends next cycle (3s); no single-reply dependency | | Majority lost (standbys disconnect) | Primary continues serving; logs WARN; `repl_wait_majority_ack` returns timeout for new writes | -| Committing timeout fires (60s) | Primary enters degraded mode; logs WARN with per-standby convergence state | +| Committing timeout fires (60s) | Primary enters degraded mode; logs WARN with per-standby convergence state *(Note: removed by `ha-single-leader-role` along with the COMMITTING role — see note under Majority Consistency Invariant)* | | Config push fails mid-cycle | Standby reports old tag next heartbeat → primary retries config push | ## Testing Strategy @@ -439,7 +457,7 @@ ha_state_t - `handle_ctrlhello` no longer triggers replication — only connection setup ### Integration Tests -- 3-node cluster: primary enters serving via majority convergence (not timeout) +- 3-node cluster: primary enters serving via majority convergence (not timeout) *(Note: superseded by `ha-single-leader-role` — primary now enters CTRL_PRIMARY directly and gates confirmation on a WAL_NOOP own-term commit; see note under Majority Consistency Invariant)* - Fresh standby (tag=0, lsn=0): bootstrap through config → WAL → converged in heartbeat cycles - Standby rejoin within WAL window: WAL streaming only, no config push, no snapshot - Standby rejoin outside WAL window: snapshot transfer → WAL streaming → converged diff --git a/.kiro/specs/ha-wal-replication/design.md b/.kiro/specs/ha-wal-replication/design.md index d935db65..2d31aecb 100644 --- a/.kiro/specs/ha-wal-replication/design.md +++ b/.kiro/specs/ha-wal-replication/design.md @@ -13,8 +13,10 @@ Bridge the WAL replicator and HA connection layer so WAL entries written on the Standby receive (cluster.c): OPCode_walCatchupEntry → handle_wal_catchup_entry - → ha_apply_wal_catchup_entry → apply_wal_entry - → send WalAckReply(lsn) + → ha_apply_wal_catchup_entry → persist_wal_write_replicated (durable + write; entity apply is now deferred to the commit-driven apply pump, + not immediate — see note below) + → send WalAckReply(lsn) [helper: ha_send_wal_ack_reply, ha.c] Primary receive (ha.c): OPCode_reply + WalAckReply → repl_on_ack → signal ack_cond diff --git a/.kiro/steering/api-standards.md b/.kiro/steering/api-standards.md index 101f734e..eb459e2b 100644 --- a/.kiro/steering/api-standards.md +++ b/.kiro/steering/api-standards.md @@ -48,7 +48,7 @@ Every HTTP response uses a consistent envelope via two helpers in `controllers/r ```go // Success (always HTTP 200) ReturnSuccess(ctx, code, msg, data) -// → {code: 0, msg: "success", data: {...}} +// → {code: 200, msg: "success", data: {...}} // Error (HTTP status == code parameter) ReturnError(ctx, code, msg, details) @@ -61,7 +61,7 @@ All codes defined in `common/common.go`: | Code | Constant | Usage | |------|----------|-------| -| 0 | `SUCCESS_CODE` | All success responses | +| 200 | `SUCCESS_CODE` | All success responses | | 400 | `INVALID_PARAMS_CODE` | Malformed/missing input | | 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing/invalid JWT | | 403 | `ERROR_AUTH_CODE` | Valid JWT, insufficient permission | @@ -79,7 +79,7 @@ WhitelistURL → WhitelistIP → RateLimit → ValidateHandler (JWT) → CORS - **WhitelistURL**: filter by allowed paths (applied first for efficiency) - **WhitelistIP**: abort 403 if client IP not in whitelist -- **RateLimit**: 10 req/sec, burst 20 (global config) +- **RateLimit**: 1000 req/sec, burst 2000 (global config, `config/app.yaml`) - **ValidateHandler**: parse and validate JWT from `Authorization: Bearer` header; validate body content-type - **CORS**: origin checks via config - **RBAC**: `RequirePermission("action:scope")` per-route, enforced by `controllers/rbac_check.go` @@ -96,13 +96,13 @@ Defined in `config/rbac.yaml`. Roles support inheritance via `includes:`. All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/controller.capnp`. **Request** (`xce_request_t`): -- `opcode` — OPCode enum (38 values) +- `opcode` — OPCode enum (43 values) - `seq` — request sequence number - `from` — sender peer info (host/key) -- `body` — union of 27 message types +- `body` — union of 29 message types (including the empty `nomsg` placeholder) **Reply** (`xce_reply_t`): -- `opcode` — always `OPCode_reply` or `OPCode_redirect` +- `opcode` — echoes the originating request's opcode by default (`init_reply_by_req()` sets `reply->opcode = req->opcode`); many call sites override it with a type-specific opcode (e.g. `OPCode_preVoteReply`, `OPCode_walAck`, `OPCode_schedAdmin`). Consumers dispatch on this value, so it is not a fixed `OPCode_reply`/`OPCode_redirect` pair — that pairing only applies to the separate wrapper `xce_request_t` that `xce_reply()` builds internally. - `status` — error code (0 = success) - `seq` — echoes request seq - `data` — union of reply payloads (clusterconf, jobid, stateReport, walackreplymsg, etc.) @@ -111,7 +111,7 @@ All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/contro | Group | OPCodes | Purpose | |-------|---------|---------| -| Control | `jobsubmit (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle | +| Control | `jobsub (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle | | Cluster | `hello (2)`, `join (3)`, `clusterconf (17)`, `heartbeat (25)` | Membership and liveness | | Replication | `replConfig (29)`, `replConfigPush (30)`, `walCatchupEntry (37)`, `snapshotTransfer (36)` | Config + WAL replication | | Vote | `requestVote`, reply | Raft election | @@ -137,8 +137,8 @@ int xce_decode(io_packet_t *pkt, xce_request_t *req); ### Adding a New Opcode -1. Add the opcode constant to `src/include/request.h` (OPCode enum) +1. Add the opcode constant to the `OPCode` enum in `src/schema/controller.capnp` 2. Add the message struct to `src/schema/controller.capnp` (Request/Reply union arms) 3. Rebuild capnp schemas (automatic via meson) -4. Wire the handler in the dispatch gate (`ha_dispatch_gate` in `ha.c`) +4. Add a `case OPCode_...:` in the `switch (request.opcode)` dispatch inside `myreq_handler()` in `cluster.c` (this runs after `ha_dispatch_gate`, which only applies a role-based allow/deny filter, not per-opcode handler logic) 5. Add encode/decode paths in `src/lib/request.c` if using raw payloads diff --git a/.kiro/steering/build-deployment.md b/.kiro/steering/build-deployment.md index bd0c29c6..d45c81d0 100644 --- a/.kiro/steering/build-deployment.md +++ b/.kiro/steering/build-deployment.md @@ -2,7 +2,7 @@ ## Philosophy -A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Every build includes debug symbols + ASAN. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management. +A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management. ## Build Commands @@ -16,6 +16,7 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`): /xce-build worker # lib + worker (daemon only, no unit tests) /xce-build apiserver # lib + apiserver + xcecli + Go unit tests /xce-build xcecli # xcecli + Go tests +/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests) ``` `controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it. @@ -23,7 +24,7 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`): Each invocation: 1. `nix develop` into the hermetic shell 2. `meson setup --buildtype= -Db_sanitize=` (fresh clean build) -3. `ninja -C ` (compile all + test executables) +3. `meson compile -C [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything 4. Run registered meson tests (skipped for non-`asan` profiles — see below) **Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use. @@ -41,9 +42,10 @@ src/ ├── controller/meson.build # xce_controller daemon + controller_unit_test ├── worker/meson.build # xcecd daemon ├── scheduler/meson.build # scheduler daemon + test -├── autoscaler/meson.build # autoscaler daemon +├── autoscaler/meson.build # libscaler static lib + provider tools (no daemon binary) ├── frontend/apiserver/meson.build # Go apiserver via tools/go_build -└── frontend/xcecli/meson.build # Go xcecli via tools/go_build +├── frontend/xcecli/meson.build # Go xcecli via tools/go_build +└── xceadmin/meson.build # xceadmin config-build tool (used by /xce-deploy-ha) ``` **Dependency flow**: `lib → {controller, worker, scheduler, apiserver}` @@ -51,8 +53,7 @@ src/ **Test registration**: ```python # controller/meson.build -test('controller', - controller_unit_test, +test('controller test', controller_unit_test, suite: 'controller', timeout: 300, env: ['ASAN_OPTIONS=detect_leaks=0']) @@ -71,7 +72,7 @@ nix build .#xcecore_debug # default package: debug + ASAN nix build .#xcecli # CLI only ``` -**AppImage**: The flake produces an AppImage wrapping `xce` (shell entry point) with bundled daemons, haproxy, glow, and less. The entry script reads from `/tools/appimage/entry.sh`. +**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced separately with `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` (see README.md, "3.5 build AppImage package"). ## Local HA Cluster Deployment @@ -86,7 +87,7 @@ This creates under `/tmp/xce-deploy/`: certs/ # TLS certs for all peers controller/ ctrl-1/ - conf/{current, cluster.bin} # primary's config + conf/{current, cluster.bin.} # primary's config (tag = sha256sum prefix) data/ # WAL + checkpoint storage ctrl-2/ conf/ # receives config via OPCode_replConfigPush @@ -135,7 +136,7 @@ Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` f | Controller daemon | `nix build .#xcecore` | `xce_controller` binary | | Worker daemon | same | `xcecd` binary | | Scheduler daemon | same | `xce_scheduler` binary | -| REST API server | same | `apiserver` binary | +| REST API server | same | `xce_apiserver` binary | | CLI tool | `nix build .#xcecli` | `xcecli` binary | -| AppImage | `nix build .#xcecore` | `xce-x86_64.AppImage` | +| AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) | | Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries | diff --git a/.kiro/steering/error-handling.md b/.kiro/steering/error-handling.md index 183479da..a858d00f 100644 --- a/.kiro/steering/error-handling.md +++ b/.kiro/steering/error-handling.md @@ -6,7 +6,7 @@ Fail fast with a consistent, machine-readable response. Handle errors at the con ## Go API Response Pattern -All HTTP responses go through two helpers in `controllers/response.go`: +All HTTP responses go through two helpers in `controllers/response.go` (a small number of legacy handlers, e.g. `FilesController` list/dir endpoints in `controllers/files.go`, still call `ctx.JSON()` directly — treat those as bugs to fix opportunistically, not as sanctioned exceptions): ```go // Success (always HTTP 200) @@ -18,7 +18,7 @@ ctx.Abort() // always Abort() after ReturnError in middleware return // always return immediately after ReturnError in handlers ``` -**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`. +**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`. `ctx.Abort()` should always follow in middleware since gin does not stop the handler chain on its own — `middleware/whitelist.go`'s `WhitelistIPMiddleware` currently has one path that returns without calling `ctx.Abort()` (invalid client IP); that is a gap to close, not an accepted variant. ## HTTP Status Code Mapping @@ -78,17 +78,23 @@ if (rc < 0) { ## Input Validation Pattern -Validate at the controller boundary using a `preCheck{Resource}Request()` helper before touching services: +Validate at the controller boundary in two steps: bind the request body with `utils.BindJSONStrict`, then pass the already-bound struct pointer to a `preCheck{Resource}Request()` helper before touching services: ```go -func (c JobController) SubmitJob(ctx *gin.Context) { - req, err := preCheckSubmitJobRequest(ctx) +func (j JobController) CreateJob(ctx *gin.Context) { + var req models.JobSubmitRequest + if err := utils.BindJSONStrict(ctx, &req); err != nil { + ReturnError(ctx, common.ERROR_CODE, err.Error(), nil) + return + } + + parsedIndices, err := preCheckJobRequest(&req) if err != nil { - ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil) + ReturnError(ctx, common.ERROR_CODE, err.Error(), nil) return } // ... service call } ``` -Keep `preCheck*` functions in the same file as the handler. They bind, validate, and return the typed request struct or an error. +Keep `preCheck*` functions in the same file as the handler. They take the already-bound request struct pointer (never `ctx`) and return `error`, or `(extra, error)` when the caller also needs derived data — e.g. `preCheckJobRequest(req *models.JobSubmitRequest) ([]int, error)`, `preCheckHostControlRequest(req *models.HostControlRequest) (bool, error)`. They never bind the request themselves and never return the request struct — binding is always the separate `utils.BindJSONStrict` step shown above. Note that handlers choose the response code for bind/precheck failures per-endpoint; `common.ERROR_CODE` (500) and `common.INVALID_PARAMS_CODE` (400) both appear in practice depending on the handler. diff --git a/.kiro/steering/ha-async-io.md b/.kiro/steering/ha-async-io.md index 88e1bc64..f61779aa 100644 --- a/.kiro/steering/ha-async-io.md +++ b/.kiro/steering/ha-async-io.md @@ -15,7 +15,7 @@ Three components in `src/lib/` plus a dedicated slow-lane executor: | Component | Thread model | May do | Must NOT do | |-----------|-------------|--------|-------------| | **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) | -| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool | +| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1743-1745) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool | | **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts | | **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) | @@ -23,10 +23,10 @@ Three components in `src/lib/` plus a dedicated slow-lane executor: No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline: 1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask. 2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover. -3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, commit.c:195-203) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here. +3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, `wait_cap`/`wait_admit_locked`, commit.c:241-271) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here. 4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr. -> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`) — tracked as the `raft-drive-async` / P2 work in roadmap.md. +> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`). **All three are now fixed** — spec `raft-drive-async` (IMPLEMENTED, validate-impl GO 2026-07-18, 21/21 tasks) moved each off its liveness thread: snapshot transfer now runs on a dedicated slow-lane worker (`snapshot_lane_fn`, `ha.c:4021`, kept off the iomgr thread — see the "DEFAULT ON" note at `ha.c:5568`); `push_config_transport_only` now spawns a dedicated pthread per standby (`pthread_create`, `config_repl.c:1589`) instead of busy-polling+joining on iomgr; and `socket_connect` (`netio.c:302`, doc comment `netio.c:287`) is now a bounded, non-blocking TCP connect via `poll()` with `connect_timeout_ms`, so it no longer pins the delaymgr heartbeat thread. See bug-1447 below for a same-day crash this spec's own validation soak surfaced and fixed. Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking. @@ -87,9 +87,9 @@ The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commi | Role | Allowed | |------|---------| -| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) | -| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes | -| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer | +| **CTRL_PRIMARY** | All opcodes, subject to two fences checked before the quorum gate (`ha_dispatch_gate`, ha.c:4525): `config_serve_fenced()` (ha-term-safety task 4.3) and `upgrade_materialization_fenced()` (config-upgrade-snapshot task 6.2) each refuse state-changing ops while set, admitting only replication/read-only/status opcodes. Once unfenced: client ops are subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` (plus read-only/status/replication opcodes) is still admitted | +| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, heartbeat, redirect, steppingDown, requestVote, preVoteRequest, probeConfigTag, upgradeConfigAck, ctrlstatus, + all replication opcodes (replEntry, replAck, replSnapshot, replCatchUpDone, replConfig, replConfigPush, walAck, snapshotTransfer, walCatchupEntry) | +| **CTRL_INIT / ELECTING** | requestVote, preVoteRequest, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer, ctrlstatus | This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind. diff --git a/.kiro/steering/product.md b/.kiro/steering/product.md index f611563c..e24d420f 100644 --- a/.kiro/steering/product.md +++ b/.kiro/steering/product.md @@ -6,9 +6,9 @@ XCE is an HPC (High-Performance Computing) cluster management system providing j 1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API. 2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering. -3. **Cluster and host management** — node status monitoring (online/offline/unavailable), cluster topology tracking, and host control operations. -4. **Multi-tenant RBAC + pluggable auth** — domain-based `action:scope` permission model (e.g., `job:view:self`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting. -5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with LSF-style priority election for standby failover and live replication over TCP (Cap'n Proto serialization). +3. **Cluster and host management** — node status monitoring (OK, UNAVAIL, LOCKEDU, LOCKEDW, BUSY, CLOSED, CLOSED_BY_ADMIN), cluster topology tracking, and host control operations. +4. **Multi-tenant RBAC + pluggable auth** — two-part `action:operation` permission IDs (e.g., `job:view`) with a separately-keyed domain scope per role (e.g., `type: self`, `type: self_ug`, `type: all`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting. +5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with Raft-style quorum election (RequestVote, term-based voting, freshness gates) for standby failover and live replication over TCP (Cap'n Proto serialization). ## Target Use Cases @@ -55,9 +55,11 @@ raft-sim-conformance — all now GO. That sweep surfaced one real bug (bug-1447: same-batch endpoint use-after-free in `io_worker`'s epoll dispatch loop, see `ha-async-io.md`), root-caused and fixed the same day. -Remaining backlog (requirements-only or design-pending, not yet implemented): -admin-config-tool (spec finalized), scheduler-framework, config-upgrade-snapshot -(blocked on entity-bearing snapshots). See `roadmap.md` for dependency order +Remaining backlog (requirements-only, design-pending, or partially implemented): +admin-config-tool (spec finalized), config-upgrade-snapshot (tasks generated, +unblocked, implementation nearly complete). scheduler-framework was decomposed +into scheduler-core (implementation-complete), scheduler-allocation, and +scheduler-preemption (both active). See `roadmap.md` for dependency order and status. Key design decisions already locked: diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 16b04b8d..a95609c5 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -6,10 +6,10 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin ## In Flight -- **scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`. Dependencies: none hard (builds on existing plugin framework + select expression evaluator). Next: `/kiro-spec-requirements scheduler-framework`. +- ~~**scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`.~~ **DECOMPOSED 2026-07-27** into three child specs (`.kiro/specs/scheduler-framework/spec.json` `child_specs`): **scheduler-core** — core/driver separation, pluggable job picker interface, and simulation mode; **IMPLEMENTATION-COMPLETE**, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups), 45/45 sub-tasks done. **scheduler-allocation** — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. **scheduler-preemption** — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. The parent `scheduler-framework` spec is retained only for discovery-context provenance and is no longer in flight itself. Next: `/kiro-spec-design scheduler-allocation` and `/kiro-spec-design scheduler-preemption`. - ~~**ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.~~ **COMPLETE 2026-07-13** — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight. - ~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it~~ **task 5.4 CLOSED 2026-07-18** via `/kiro-validate-impl`'s live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight. -- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:568-569`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this. +- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:598-599`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this. ## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21) @@ -21,9 +21,11 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin > **ha-term-safety** (`.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. **IMPLEMENTED, validate-impl GO 2026-07-17** (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix). -## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`) +> **raft-adapter-conformance** (`.kiro/specs/raft-adapter-conformance/`) — NEW 2026-07-29, sibling audit to the library-side conformance work above: a systematic, action-by-action pass over the controller's Raft **adapter** (the wire-level send/receive/reply code in `ha.c`/`repl.c`/`config_repl.c`) against `raft.tla`'s actions — distinct from `raft-sim-conformance`'s in-process library harness. Motivated by a live-soak finding that the controller's `WalAckReply` constructor hardcoded success (no failing reply ever reached the primary) plus a standby catch-up streaming-range self-deadlock, both already fixed (bug-1479) before this spec was seeded. This pass found and fixed two more instances of the same defect class: configuration-push rejects previously sent no reply at all (bug-1480), and a vote/PreVote endpoint failure arriving *after* connect routed through a legacy tally path instead of the production one (extending bug-1423's connect-time fix to that remaining timing window). Also records, as a single explicit decision, which `AppendEntries` mechanism is production-authoritative (`repl_broadcast` + `ha_stream_wal_to_standby`; the per-peer index-driven `raft_send_append_entries` stays dormant by documented design, not oversight). Requirements/design/tasks generated and approved, all 15 leaf tasks (1–7.3) checked; `src/controller/CLAUDE.md`'s Protocol Verification Checklist marks the corresponding invariants "post `raft-adapter-conformance`, landed 2026-07-29" — `spec.json` phase (`tasks-generated`) has not been flipped to reflect this, same administrative-staleness pattern seen elsewhere in this doc (cf. the former libxce-failover entry above). Dependencies: raft-log-replication, raft-commit-safety, ha-term-safety, raft-sim-conformance (adjacent/informational — this spec fixes the adapter layer those validate underneath). -**Goal:** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. Today `persist_job_hook_fun` (an `AFTER_EVENT` hook) performs the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutates in-memory state BEFORE that commit (`job.c:558` before `job.c:593`; `persist.c:2532`/`persist.c:2198`/`ha.c:3080`). A failed/timed-out commit does not cleanly revert the optimistic mutation — the structural root of the bug-795/828/851/1060 divergence family. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decompose via Path E (2 new specs + touchpoints). +## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`); IMPLEMENTED — see `raft-apply-decoupling` below + +**Goal (historical — achieved):** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. At the time this family was opened, `persist_job_hook_fun` (an `AFTER_EVENT` hook) performed the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutated in-memory state BEFORE that commit — the structural root of the bug-795/828/851/1060 divergence family. A failed/timed-out commit did not cleanly revert the optimistic mutation. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decomposed via Path E (2 new specs + touchpoints), delivered as `raft-apply-decoupling` (**IMPLEMENTED 2026-07-07**, see below). Current code reflects the fix: `persist_job_hook_fun` (`persist.c` ~6509-6553) now explicitly returns early for `WAL_MODE_WAIT` entries, with a comment noting job-submit durability is invoked directly by `handle_jobsubmit` — `persist_submit_job_payloads_and_commit()`, called strictly before any job object is constructed — never from this hook; `create_job()` is now defined at `job.c:48`. ### Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`) @@ -60,34 +62,34 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han ## Future Specs / Backlog (dependency order) -- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **HARD dependency: `entity-bearing-snapshots`** (today snapshots are a 16-byte LSN sentinel; the entity payload does not exist) AND `ha-term-safety` (the config-change marker it anchors on). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Promote to design once `entity-bearing-snapshots` lands. +- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.** - [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands. - [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only). -- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) remains blocked on `entity-bearing-snapshots` (pre-existing). Next: `/kiro-spec-requirements admin-config-tool`. +- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) was blocked on `entity-bearing-snapshots`; that blocker is now cleared (see Future Specs/Backlog above — `ha-entity-snapshots` implemented, `config-upgrade-snapshot` `blocked_on: []`). Next: `/kiro-spec-requirements admin-config-tool`. - [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state). - [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready. - [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships. - [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item. - [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization). - [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants). -- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream (recorded 2026-06-21): `config-upgrade-snapshot` HARD-depends on this** — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857). **This is also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) — recorded 2026-06-25:** WAL streaming alone cannot serve a standby whose needed range was trimmed (bug-795/828/1147), and a new node always starts empty, so safe aggressive WAL trimming + new-node join *requires* InstallSnapshot (snapshot-at-boundary + tail WAL). Until this lands, trim must stay cursor-bounded (strategy-A stopgap, see Hardening) and the system cannot safely admit a standby that has fallen below the trim floor. Scope to add when promoted: the standby-side snapshot-install / apply path (request snapshot when its cursor is below the leader's earliest retained LSN, install, resume from snapshot tail). +- [x] entity-bearing snapshots ("Task 11" format) — **PROMOTED to spec `ha-entity-snapshots` (created 2026-07-01); IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks 1.1-8.1; 962 controller tests green including all 9 `test_persist_snapshot_entities.c` cases; live `--force-snapshot` soak: committed_divergence=0, committed_job_loss=0, confirmed real "HA: snapshot" install log line). `write_snapshot()` (`persist.c` ~2304-2372) now collects and serializes job/queue/host entity frames (`collect_job_frames`/`collect_queue_frames`/`collect_host_frames`) ahead of the header — no longer a bare 16-byte LSN sentinel. Original backlog note retained for provenance: real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs was already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream: `config-upgrade-snapshot`'s former HARD dependency on this is now cleared** (`blocked_on: []`) — see that item above. **Also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0)** is now live: the standby-side snapshot-install/apply path is wired (`ha.c` ~3112-3147 wire-receive path task 6.1; `persist.c` ~4277-4285 local boot recovery task 8.1), with a decision path that requests a snapshot transfer when a standby's cursor falls below the leader's oldest retained WAL LSN (`ha.c:4413`). The bug-1147 WAL-trim cursor-bound guard (see Hardening below) is implemented in production, no longer just a stopgap. ## Hardening / Direct-Implementation Items Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work. - [ ] **investigate: job_ctrl direct-mutate path (admin suspend/resume) — migrate to commit-then-apply?** — **recorded 2026-07-13, surfaced during job-array-single-wal-record's `persist_job_hook_fun` investigation.** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes) — confirmed the ONLY remaining live call site that still mutates state directly, THEN fires the `AFTER_EVENT` hook, with `persist_job_hook_fun` doing the real WAL append + async commit from inside the hook (not through `apply_committed_entry()`). This is the same mutate-before-commit shape as the bug-795/828/851/1060 divergence family raft-apply-decoupling was built to eliminate for job submit + worker facts, narrowly scoped to admin suspend/resume. Not yet known to have caused a live incident — rare, synchronous, admin-only traffic. **Investigate before committing to a fix**: (a) actual blast radius if a commit fails/times out post-mutation (does it ever get observed/notified before commit — violates criterion 2.4-style provisional-state-export concerns?); (b) live-apply vs replay-apply asymmetry for this one WAL entry type — does replay actually diverge from live in any reachable scenario, or is it benign in practice; (c) cost of migrating vs. accepted-residual status quo. Promote to a spec (likely a small `raft-apply-decoupling` follow-up, not standalone) only if the investigation finds real risk, not proactively. -- [ ] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — `trim_wal_segments` (`persist.c:1374`) bounds deletion ONLY by the open segment (`if (seg_start_lsn < active_seg_lsn) unlink()`) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** in-flight snapshot boundary. With no working snapshot backfill (`write_snapshot` emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is **strategy-A retention discipline that should exist regardless**. Stopgap fix: compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. **Folded into `raft-drive-async` task 3.2** (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs `entity-bearing snapshots` (strategy B, backlog). Related: bug-795, bug-828, bug-1060. +- [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060. - [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_ - [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`. -- [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` **and `OPCode_schedAdmin`** (added 2026-08-02, `scheduler-core`) with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.** +- [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.** - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.** - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_ - [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_ - [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1. - [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump `** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare `** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_ - [ ] xce_test shell command support — add `!` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log. -- [→] scheduler framework enhancement — **PROMOTED to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/scheduler-framework/brief.md`). Original backlog seed superseded by the full brief with iterator-driven allocation architecture, configurable job picker, combinatorial parallel job allocation, and session-level unsatisfiable resreq cache. Scope narrowed from the original seed: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). Next: `/kiro-spec-requirements scheduler-framework`. +- [x] scheduler framework enhancement — **PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27** into `scheduler-core` (**IMPLEMENTATION-COMPLETE**), `scheduler-allocation`, and `scheduler-preemption` — see **In Flight** above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here. - [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass. - [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`. diff --git a/.kiro/steering/security.md b/.kiro/steering/security.md index 891c80f2..6e70adc3 100644 --- a/.kiro/steering/security.md +++ b/.kiro/steering/security.md @@ -20,7 +20,7 @@ hostsGroup.POST("/control", RequirePermission("host:control"), handler) ## Permission String Format -`"action:scope"` — e.g., `"job:submit"`, `"host:view"`, `"queue:manage"`. +`"action:scope"` — e.g., `"queue:submit"`, `"host:view"`, `"queue:control"`. Scopes: `all` (unrestricted), `self` (own resources), `cluster`, `hostgroup`, `self_ug`, `self_hg`. @@ -39,7 +39,7 @@ Token secret is loaded from `config/auth.yaml` at startup. It must never appear ## Auth Backends -Backend selection is controlled by `config/auth.yaml`. Supported: `configfile`, `ldap`, `oauth2`, `pam`. All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware. +Backend selection is controlled by `config/auth.yaml`'s `provider:` field, matched literally against a `switch` in `backend/auth/init.go` — an unrecognized value (e.g. `configfile` without the underscore) fails apiserver startup with "unsupported auth provider". Valid identifiers: `config_file`, `pam`, `ldap`, `oauth2`. Only `config_file` (dev mode, users defined in the YAML) and `pam` (Linux + CGO only) are functional today — `ldap` and `oauth2` are unconditional stubs whose `Authenticate()` always returns "not yet implemented". All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware. ## Input Validation @@ -47,14 +47,14 @@ Validate at the API boundary inside `preCheck{Resource}Request()` helpers — be ## Secrets -- JWT secret key: `config/auth.yaml` (file-system protected, chmod 0600) +- JWT secret key: `config/auth.yaml` — should be file-system protected (e.g. chmod 0600); this is not currently enforced by the code or deploy tooling, so treat it as an operator responsibility, not a guarantee - LDAP bind credentials: `config/auth.yaml` - **Never log** tokens, passwords, secret keys, or full request bodies containing credentials - Validate all required config at startup; fail fast with a clear error if missing ## Rate Limiting and IP Whitelist -Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/ip_whitelist.go`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds. +Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/whitelist.go`, which implements both `WhitelistURLMiddleware()` and `WhitelistIPMiddleware()`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds. ## Transport @@ -64,6 +64,6 @@ XCE uses **server-side TLS only** — encryption without peer certificate verifi **Clients talking to the API server** (xcecli, or any HTTP client of `apiserver`) must support **both `http://` and `https://`** — the scheme is a per-deployment/user choice (`APIServerInfo.URL`), not something the client hardcodes. When the scheme is `https://`, set `InsecureSkipVerify: true` on the TLS client config (see `xcecli/common/request/request.go`'s `NewAPIClient`) rather than attempting cert validation — the API server's cert is typically self-signed/auto-generated and the client has no CA to validate it against. -**Clients talking to an XCE controller directly** (worker, scheduler, and any other libxce-based client via `cluster.c`'s `do_connect()`) need **only server-side certs — no client-side certs are required or supported**. `xce_cluster_config_t` has no `ca_cert_path`/client-cert fields; do not add any. TLS verification mode is `SSL_VERIFY_NONE` throughout — this matches the API-server posture above, just at the C/libxce layer instead of Go/HTTP. +**Clients talking to an XCE controller directly** (worker, scheduler, and any other libxce-based client) need **only server-side certs — no client-side certs are required or supported**. The live connection path is `xcecd_join_cluster()` (`src/worker/src/controller_connector.c`) calling `xcecli_start`/`xcecli_join` (`src/lib/client.c`), which drives `endpoint_connect()` in `src/lib/netio.c` over an `SSL_CTX` built by `create_ssl_context(NULL, NULL)` (`src/lib/xcectx.c`) — note that `cluster.c`'s `do_connect()`/`xce_connect_cluster()`/`xce_reconnect_init`/`xce_reconnect_step` are an unused stub path (explicitly documented as "to be wired in a later task") with no callers outside `cluster.c` and its own testsuite; don't point at them as the live mechanism. `xce_cluster_config_t` has no `ca_cert_path`/client-cert fields; do not add any. TLS verification mode is `SSL_VERIFY_NONE` throughout — this matches the API-server posture above, just at the C/libxce layer instead of Go/HTTP. **Known follow-up (not yet scoped):** a HAProxy reverse proxy in front of `apiserver` is planned to terminate HTTPS, likely with a CA-trusted cert (Let's Encrypt candidate, unconfirmed), and whether `apiserver`'s own `RunTLS`/`httpsConfig.Enabled` listener stays in the picture is also undecided. Once both are settled, revisit xcecli's blanket `InsecureSkipVerify: true`: if the client-facing endpoint ends up CA-trusted, default to real verification (Go's default `tls.Config{}` validates against the system trust store with no extra plumbing) and make skip-verify an explicit opt-in for self-signed deployments only — matching the automated security review's suggested fix. Don't implement this speculatively; the right shape depends on which of those two decisions lands where. diff --git a/.kiro/steering/structure.md b/.kiro/steering/structure.md index 0d61e7be..535534b5 100644 --- a/.kiro/steering/structure.md +++ b/.kiro/steering/structure.md @@ -2,7 +2,7 @@ ## Organization Philosophy -**Layered monorepo, component-per-directory.** Each major subsystem lives under `src//` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/xce/`. +**Layered monorepo, component-per-directory.** Each major subsystem lives under `src//` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/`. ``` src/ @@ -14,14 +14,16 @@ src/ ├── scheduler/ C — job scheduling engine (plugin architecture) ├── autoscaler/ C — autoscaling engine (provider plugins) ├── frontend/ Go — apiserver (Gin REST) + xcecli (Cobra CLI) -├── include/xce/ C — public shared headers consumed across all daemons +├── xceadmin/ Python — admin CLI tool +├── include/ C — public shared headers consumed across all daemons +│ └── xce/ C — subset of public headers under the xce/ prefix ├── schema/ Cap'n Proto schema files (c.capnp, controller.capnp) └── docs/ Project-wide documentation (markdown) ``` ## C Daemon Layout Pattern -Each C daemon (`controller/`, `worker/`, `scheduler/`, `autoscaler/`) follows: +`worker/` and `scheduler/` follow: ``` src// @@ -32,7 +34,10 @@ src// testsuite/ — test programs (test_*.c) ``` -**Exception**: `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`). All other C daemons use `src//src/`. +**Exceptions**: +- `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`), and does have its own `testsuite/`. +- `worker/` has no `testsuite/` — no `test_*.c` files exist anywhere in the component. +- `autoscaler/` does not follow this pattern at all: it has no CLAUDE.md, and its daemon code lives two levels deeper in `src/autoscaler/scaler/` (flat `.c` files plus its own `include/` and `testsuite/`), alongside `src/autoscaler/providers/`. Plugin-based components add their own subdirectory: - `scheduler/` — plugins in `src/plugins/` @@ -62,7 +67,7 @@ Plugin-based components add their own subdirectory: ### C - **Files**: `snake_case.c` / `snake_case.h` — one domain per file -- **Public headers**: `src/include/xce/.h` — the libxce API boundary +- **Public headers**: `src/include/.h` (e.g. `persist.h`, `netio.h`, `resreq.h`) — the libxce API boundary; a small subset (`client_failover.h`, `cluster.h`, `cluster_def.h`, `errnox.h`, `types.h`) lives under `src/include/xce/` and is included as `xce/.h` - **Component headers**: `src//include/.h` - **Test files**: `test_.c` (e.g., `test_ha_cluster.c`, `test_persist_wal.c`) - **Functions**: `snake_case` with component prefix — `ha_*`, `cluster_*`, `xcecd_*`, `xce_*` @@ -74,8 +79,8 @@ Plugin-based components add their own subdirectory: - **Files**: one file per resource — `jobs.go`, `queues.go`, `hosts.go` - **Tests**: `*_test.go` co-located; integration tests as `integration_test.go`, `_control_test.go` -- **Controller structs**: `{Resource}Controller` -- **Handler methods**: `{Action}{Resource}` (e.g., `SubmitJob`, `ListJobs`) +- **Controller structs**: `{Resource}Controller` (e.g., `JobController`, `ClusterController`, `QueueController`); one exception — `hosts.go` uses `HostHandler` +- **Handler methods**: `{Action}{Resource}` (e.g., `CreateJob`, `GetJobsInfo`, `ControlJobByID`) - **Permission strings**: `"action:scope"` (e.g., `"job:view:self"`) - **JSON tags**: `snake_case` diff --git a/.kiro/steering/tech.md b/.kiro/steering/tech.md index c0884717..b66a35aa 100644 --- a/.kiro/steering/tech.md +++ b/.kiro/steering/tech.md @@ -8,9 +8,9 @@ xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C da ↑↑↑ all linked against libxce (src/lib/) ``` -Request path inside apiserver: `Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store` +Request path inside apiserver: `Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store` -All controller state is in-memory, seeded at startup from YAML config. No persistent database in the API tier. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level. +All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level. ## Core Technologies @@ -52,18 +52,20 @@ nix build .#xcecli # CLI only ### Controller Pattern (Go) ```go -// Struct: {Resource}Controller -type JobController struct { ... } +// Struct: {Resource}Controller (typically an empty struct{}) +type JobController struct{} // Handler: {Action}{Resource} -func (c JobController) SubmitJob(ctx *gin.Context) { +func (j JobController) CreateJob(ctx *gin.Context) { // Validate input - req, err := preCheckSubmitJobRequest(ctx) - if err != nil { ReturnError(ctx, code, err, nil); return } + var req models.JobSubmitRequest + if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return } + parsedIndices, err := preCheckJobRequest(&req) + if err != nil { ReturnError(ctx, code, err.Error(), nil); return } - // Business logic - result, err := c.svc.Submit(req) - if err != nil { ReturnError(ctx, code, err, nil); return } + // Business logic — package-level services function, not a controller field + err = services.SubmitJob(&newJob, arraySize) + if err != nil { ReturnError(ctx, code, err.Error(), nil); return } ReturnSuccess(ctx, code, "submitted", result) } @@ -86,15 +88,15 @@ Early return on every error. Log with zap fields before returning. ### Auth and RBAC -- Auth backend set in `config/auth.yaml`. Supports: configfile, LDAP, OAuth2, PAM. -- Permissions as `"action:scope"` strings: `"job:view:self"`, `"queue:submit:all"`. +- Auth backend set in `config/auth.yaml` (`provider: config_file | pam | ldap | oauth2`). Only `config_file` and `pam` (Linux + CGO) are functional; `ldap` and `oauth2` are documented as future/stub providers whose `Authenticate()` always returns a "not yet implemented" error. +- Permissions are a plain action string (e.g. `"job:view"`, `"queue:submit"`) paired with a separate structured `domain` list (e.g. `{type: self}`, `{type: all}`) in `rbac.yaml` — not a combined `"action:scope"` string. `controllers.RequirePermission()` checks the action string only; domain/scope is evaluated separately. - RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware. ## Key Technical Decisions 1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both. -2. **In-memory API state** — no database in Go tier; state loaded from YAML at startup via `store.LoadBuiltinData()`. +2. **In-memory API state** — no database in Go tier; `store.InitStores()` allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (`backend.RegisterToBackend()` + `backend.StartNotificationListener()` in `cmd/main.go`). Only RBAC (`config/rbac.yaml`) and auth (`config/auth.yaml`) are actually loaded from YAML at startup. 3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell. 4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin. 5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output. -6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in the `src/lib/raft/` shared module (`raft.c`/`raft.h`), with the single-writer constraint avoiding Raft log replication complexity. +6. **Single-writer HA with full Raft log replication** — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, `AppendEntries` request/reply, per-peer `next_index[]`/`match_index[]` progress vectors, `(term,index)` log matching/truncation, commit-index advance) implemented in the `src/lib/raft/` shared module (`raft.c`/`raft.h`) and validated by the `raft-log-replication`, `raft-commit-safety`, `raft-ha-core-safety`, and `ha-term-safety` specs. diff --git a/.kiro/steering/testing.md b/.kiro/steering/testing.md index cb47f4b3..0a2fed27 100644 --- a/.kiro/steering/testing.md +++ b/.kiro/steering/testing.md @@ -48,10 +48,12 @@ func TestPermissions(t *testing.T) { - `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir) - `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars) -## C Tests (controller / worker / scheduler / lib) +## C Tests (controller / scheduler / lib) **Location**: `testsuite/` directory inside each C component. +**Note**: `src/worker` has no `testsuite/` directory and no meson `test()` registrations today — it currently carries no C test coverage of its own. + **Structure**: one `.c` test program per unit: ``` @@ -83,7 +85,7 @@ extern void reset_ha_state(ha_state_t *s); - Go: 80% minimum on business logic (controllers, services, store) - C: every new function in a daemon must have at least one test case in `testsuite/` -- Integration: all public REST endpoints must have at least one integration test case in `regression/` +- Integration: all public REST endpoints must have at least one integration test case in `src/frontend/tests/regression/` ## What Not to Mock diff --git a/CLAUDE.md b/CLAUDE.md index b645dc6a..d493058b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,7 +29,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s ## Key Rules (from steering — summarized for quick reference) - **TDD mandatory**: red → green → refactor; test in `testsuite/` before C changes, `_test.go` before Go changes -- **Async I/O**: never block in vtask handlers; delaymgr for blocking ops. `xce_request`/`xce_reply` are always non-blocking +- **Async I/O**: delaymgr is trigger-only and must never block (no `connect()`, no file I/O) — it drives heartbeats/election timeouts, so blocking there causes spurious failover. Long/unbounded blocking goes on the slow lane; bounded blocking in vtask handlers is permitted only under an admission reservation (`COMMIT_WAIT_RESERVED`). `xce_request`/`xce_reply` are always non-blocking. See `ha-async-io.md` - **Connections are long-lived**: closing signals RDHUP failover. Inbound connections are vote-only; outbound carry all state push - **Middleware chain** (non-negotiable): WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC - **Error pattern**: `ReturnError(code, msg, nil); return` — early return on every error @@ -100,5 +100,5 @@ Kiro-style Spec-Driven Development on an agentic SDLC ## Steering Configuration - Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative - Core: `product.md`, `tech.md`, `structure.md` -- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md` +- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`, `config-design.md`, `persistence.md`, `roadmap.md` - Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering` diff --git a/docs/superpowers/plans/2026-05-07-p0-single-controller-persistence.md b/docs/superpowers/plans/2026-05-07-p0-single-controller-persistence.md index 0e937c4f..ef673f48 100644 --- a/docs/superpowers/plans/2026-05-07-p0-single-controller-persistence.md +++ b/docs/superpowers/plans/2026-05-07-p0-single-controller-persistence.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This planned a single-controller WAL (40-byte header, fsync pthread) plus a dual-zarray copy-on-write freeze/thaw checkpoint mechanism, with HA/replication deferred to the companion P1–P3 plans (fan-out TCP, single-ACK). What actually shipped: the WAL/persist.c core survived and evolved, but checkpoint consistency uses table-level + per-entity RTTI_LOCK_READ/WRITE nesting, not the COW freeze/thaw path described here (confirmed by `.kiro/specs/ha-entity-snapshots`, which explicitly notes COW entity enumeration was a documented non-goal). Replication/HA is quorum-based Raft, not the fan-out/single-ACK design in the P1–P3 plans — see the closed specs `raft-library`, `raft-ha-core-safety`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`, `ha-single-leader-role`, `ha-commit-engine`, `ha-commit-durability`, `ha-commit-barrier-rollout`. + # P0: Single-Controller Persistence — Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/superpowers/plans/2026-05-07-p1-ha-foundation.md b/docs/superpowers/plans/2026-05-07-p1-ha-foundation.md index 0d2ece6b..18060766 100644 --- a/docs/superpowers/plans/2026-05-07-p1-ha-foundation.md +++ b/docs/superpowers/plans/2026-05-07-p1-ha-foundation.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This planned an LSF-style priority election (blocking `am_i_primary()` probes + dedicated watchdog pthread for self-promotion) on top of the pre-Kiro persistence-and-ha-design.md. What actually shipped is quorum-based Raft: `am_i_primary()` was deleted and replaced by `ha_start_election_async` (see closed specs raft-library, election-iomgr-refactor, raft-ha-core-safety, ha-prevote, raft-log-replication, raft-commit-safety, ha-term-safety, ha-single-leader-role), with real voting/term logic in `src/lib/raft/raft.c`. A few surface artifacts (`ha_init`, `ha_promote_to_primary`, `my_priority` as peer index, `E_XCE_REDIRECT`/`E_XCE_NOMASTER`, join.c redirect) survived in similar form but were rearchitected around Raft. + # P1: HA Foundation — Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` or `superpowers:executing-plans` to implement this plan task-by-task. diff --git a/docs/superpowers/plans/2026-05-07-p2-replication.md b/docs/superpowers/plans/2026-05-07-p2-replication.md index 50a7c406..8e6e9708 100644 --- a/docs/superpowers/plans/2026-05-07-p2-replication.md +++ b/docs/superpowers/plans/2026-05-07-p2-replication.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This planned a custom fan-out TCP replicator (`repl_wait_ack()` unblocking on any single standby's `OP_replAck`, plain WAL-copy `OP_replEntry` streaming, no Raft term concept). What actually shipped is majority-quorum Raft: `repl_wait_majority_ack()` delegating to a commit engine, and `repl_broadcast()`/`repl_send_append_entries()` carrying Raft terms through `raft_on_append_entries_request`. See closed specs `ha-commit-engine`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`, `ha-commit-barrier-rollout`, `ha-single-leader-role`. + # P2: Replication (Fan-Out TCP) — TDD Implementation Plan **Branch**: `persistent_ha` diff --git a/docs/superpowers/plans/2026-05-07-p3-hardening.md b/docs/superpowers/plans/2026-05-07-p3-hardening.md index a431d24c..6b11f12a 100644 --- a/docs/superpowers/plans/2026-05-07-p3-hardening.md +++ b/docs/superpowers/plans/2026-05-07-p3-hardening.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This planned hardening/test coverage for the pre-Raft HA design: single-primary promote/demote via plain term-increment (no election/voting) and single-ACK sync replication (`persist_wal_sync_append` + `repl_wait_ack`, no majority quorum). The real HA implementation that shipped is quorum-based Raft with prevote and majority-ack commit — see closed specs `raft-library`, `ha-prevote`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`, `ha-single-leader-role`, `ha-commit-engine`, and `ha-commit-durability`. (Some of this doc's infrastructure ideas, e.g. `ctrl_metrics.c`'s metric names and the promote/demote/redirect function names, did land and still exist in `src/controller/ha.c`, but `ha_promote_to_primary` is now explicitly the "NON-RAFT" legacy/bootstrap road, not the primary election mechanism.) + # P3: Integration & Hardening — TDD Implementation Plan **Branch**: `persistent_ha` diff --git a/docs/superpowers/plans/2026-05-07-persistence-ha-overview.md b/docs/superpowers/plans/2026-05-07-persistence-ha-overview.md index 68aab5d2..148f2f62 100644 --- a/docs/superpowers/plans/2026-05-07-persistence-ha-overview.md +++ b/docs/superpowers/plans/2026-05-07-persistence-ha-overview.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This phase map planned dual-zarray COW checkpointing, LSF-style priority-list election, and single-standby-ACK durability for HA. What actually shipped instead: capnp-frame checkpoints (`ha-entity-snapshots`), full quorum-based Raft consensus (`raft-library`, `raft-ha-core-safety`, `ha-prevote`, `election-iomgr-refactor`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`, `ha-single-leader-role`), and majority-quorum commit (`ha-commit-engine`, `ha-commit-durability`, `ha-commit-barrier-rollout`) — all closed `.kiro/specs/`. See the SUPERSEDED notes in `src/docs/persistence-and-ha-design.md` (Parts 3, 5, and the durability section) for details. + # XCE Persistence & HA — Implementation Overview > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/superpowers/plans/2026-05-13-config-replication.md b/docs/superpowers/plans/2026-05-13-config-replication.md index 18299fbd..8f37defc 100644 --- a/docs/superpowers/plans/2026-05-13-config-replication.md +++ b/docs/superpowers/plans/2026-05-13-config-replication.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This plan proposed a priority-based `am_i_primary()` election (non-blocking connect/poll probing with xorshift jitter) and a standalone single-ACK `OP_replConfigPush` fan-out (500ms timeout, no WAL involvement) for config replication. What actually shipped: primary election is quorum-based Raft (`ha_start_election_async`, replacing the now-deleted `am_i_primary`/`am_i_primary_quorum`/`probe_ctx_t` — see closed specs `election-iomgr-refactor`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`), and config changes now travel as `WAL_CONFIG_MARKER` entries inside the Raft/WAL commit path (`ha-term-safety` task 4.1, `ha-commit-engine`) rather than an out-of-band push. The HEAD-file/SHA-256 versioned storage layer (`conf_save_config`, `conf_read_current_filename`, `config_repl.c`) did ship largely as described here and is still current. + # Config Replication Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/superpowers/plans/2026-05-20-config-repl-wire-protocol.md b/docs/superpowers/plans/2026-05-20-config-repl-wire-protocol.md index 902de550..9a505361 100644 --- a/docs/superpowers/plans/2026-05-20-config-repl-wire-protocol.md +++ b/docs/superpowers/plans/2026-05-20-config-repl-wire-protocol.md @@ -1,3 +1,5 @@ +> **ARCHIVED — superseded.** This planned raw-byte capnp shims, a `conf_save_config()` path fix, and a 500ms-timeout push fan-out for the original single-ACK, non-quorum config-push/pull design in `config-replication-design.md` Phase 1. Verified against current source: the path fix and `wait_all_with_timeout()`/`handle_config_request()→xce_replconfig_reply_raw()` shipped close to as-designed, but `do_one_config_fetch()` was replaced by a differently-named probe path and, more importantly, the whole commit model was superseded — success now requires a WAL_CONFIG_MARKER reaching the majority-quorum committed prefix via `config_commit_change()`, per the closed `config-upgrade-snapshot`, `ha-term-safety`, and `ha-commit-barrier-rollout` specs. + # Config Replication — Wire Protocol Completion + Bug Fixes > **SDD Plan** — Subagent-Driven Development diff --git a/reports/aug03-aug09-2026.md b/reports/aug03-aug09-2026.md new file mode 100644 index 00000000..99229b2f --- /dev/null +++ b/reports/aug03-aug09-2026.md @@ -0,0 +1,86 @@ +# XCE Project Status — Aug 03 – Aug 07, 2026 + +## TL;DR + +**scheduler-core: fully implemented.** All 13 task groups landed across the week — picker plugin contract, core extraction, normal-mode driver wiring, replay wire format, snapshot save/load, recording lifecycle, admin opcode dispatch, simulation mode, interactive REPL, and HA soak regression. One controller bug fixed (bug-1505: WalCatchupEntry SIGSEGV). Aug 08 (today) was documentation cleanup: deployment guide, CLAUDE.md sync, and a broad audit of 33 closed specs. The scheduler-framework feature branch is now implementation-complete. + +## Commits by day + +### Aug 03 (Sun) — Group 1 (picker contract) + Group 2 (builtin_fcfs) + Group 4 (core extraction) + Group 5 (driver wiring) + +10 commits, closing out Groups 1–5: + +- **Group 1 — Picker Plugin Contract** (tasks 1.1–1.3): defined `job_elect_fn` plugin contract, registered picker plugin category, added picker configuration fields +- **Group 2 — builtin_fcfs** (tasks 2.1–2.6): lifecycle, state-change hooks, session/post-allocation hooks, wired as registered default picker, scheduling-loop ordering guard regression tests, data-sharing invariant audit +- **Group 4 — Core Extraction** (tasks 4.1–4.6): extracted `xce_sched_core_t` from monolithic `scheduler_t`, pinned jbuf capacity + kvmgr determinism audit, event batch push interface (CONFIG_PUSH decomposition approved round 2), session-start signal + decision callback, clock virtualization + session timing, core contract test (full push-go-drain-end cycle) +- **Group 5 — Driver Wiring** (tasks 5.1–5.2): normal-mode event translation path, decision publishing + mode-selection wired into `main.c` + +### Aug 04 (Mon) — Group 6 (HA soak gate) + Group 7 (replay wire format) + Group 8 (snapshot) + +9 commits: + +- **Group 6** (task 6): Phase 2 HA soak regression gate closed +- **Group 7 — Replay Wire Format** (tasks 7.1–7.5): replay stream envelope schema, event payload types (JobStateChangeEvent), decision payload types + snapshot schema (SnapshotData, ReplayDecision), admin opcode schema, round-trip coverage gap (1087/1087 tests pass) +- **Group 8 — Snapshot** (tasks 8.1–8.2): snapshot save/load with cross-thread concurrency fix, `-EBUSY` guard against active session + +### Aug 05 (Tue) — Group 9 (recording lifecycle) + Group 10 (admin dispatch + xceadmin CLI) + +9 commits: + +- **Group 9 — Recording Lifecycle** (tasks 9.1–9.5): directory provisioning + path validation, record start/stop lifecycle (O_TRUNC→ftruncate hardlink fix, CWE-59 patched), periodic snapshot checkpointing, scheduler-side admin request handler, integration tests (26/26 pass) +- **Group 10 — Admin Dispatch + xceadmin CLI** (tasks 10.1–10.3): controller-side `OPCode_schedAdmin` dispatch (fmt-noise + leak fix), dispatch test suite (timeout/RPC-status/error paths), xceadmin recording client functions + +### Aug 06 (Wed) — Group 10 (cont.) + Group 11 (simulation mode) + +9 commits: + +- **Group 10 cont.** (tasks 10.4–10.6): xceadmin scheduler dump command, CLI wiring (record_start/stop/status), mock-server E2E tests +- **Group 11 — Simulation Mode** (tasks 11.1–11.3): simulation-mode startup, virtual clock advancement + event loop, simulation isolation enforcement + +### Aug 07 (Thu) — Group 11 (cont.) + Group 12 (REPL) + Group 13 (validation) + +12 commits: + +- **Group 11 cont.** (tasks 11.4–11.6): simulation decision verification, what-if branching + output recording, normal↔sim decision equivalence tests (5/5 pass, 33/33 regression) +- **Group 12 — REPL** (tasks 12.1–12.4): execution control (readline→libedit GPL→BSD), breakpoints + state inspection, snapshot capture + simulation parameter commands, step/continue equivalence to batch (37/37 tests pass) +- **fix(controller):** stop SIGSEGV in WalCatchupEntry capnp encode (**bug-1505**) +- **Group 13:** full cross-phase HA soak regression (15 rounds sequential pass) +- **Validation:** spec marked implementation-complete; feature-level validation findings recorded + +### Aug 08 (Fri, today) — Documentation Cleanup + +5 commits (documentation only, not implementation): + +- `docs:` add scheduler recording/simulation/REPL to deployment guide +- `docs(scheduler-core):` sync CLAUDE.md for core/driver/picker/sim/REPL architecture +- `docs(scheduler-core):` commit accumulated brief/requirements/roadmap edits +- `docs(scheduler-core):` backfill `_Verified` notes for tasks 2.1–2.6 +- `docs(scheduler-core):` commit design.md/research.md, sync design.md with as-built code + +Plus a broad documentation audit (not yet committed): 33 closed `.kiro/specs` design.md files audited for drift against production source, with confirmed corrections across 8+ HA/Raft spec design docs. A separate stream of work (S1275–S1278) covered 28 free-standing docs → 126 audit findings, and TLS encryption claims in deployment.md were verified against `libxce` source (server-side TLS confirmed, `do_connect()` is an uncalled stub). + +## Bugs opened/fixed this window + +| Bug | Status | Summary | +|---|---|---| +| bug-1505 | **Fixed** (2026-08-07, `1fd467d0`) | `WalCatchupEntry` capnp encode SIGSEGV — `Data` fields in capnp schema caused null-deref; fixed by migrating to `List(UInt8)` (same class as bug-1504 in SnapshotData) | + +## Spec status changes + +- **scheduler-core** — **implementation-complete** (2026-08-07). All 13 task groups (1.1–13) delivered; spec.json phase → `implemented`. 55 commits, 112 files changed, ~42K insertions. + - Architecture: picker plugin system (job_elect_fn contract), builtin_fcfs, extracted xce_sched_core_t, normal-mode driver, capnp replay wire format, snapshot persistence, recording lifecycle, xceadmin CLI, simulation mode (normal↔sim equivalence), interactive REPL (BSD libedit). + +## Stats + +- **55 commits** in 5 days +- **112 files changed**, 42,005 insertions, 803 deletions +- **1 bug fixed** (bug-1505) +- **13 task groups** completed (Groups 1–13) +- **1 spec** moved to implemented (scheduler-core) +- **1,087+ roundtrip tests**, **26 integration tests**, **37 REPL tests**, **15-round HA soak** all passing + +## Open threads heading into next session + +1. scheduler-core is implementation-complete — next step is either PR review + merge to master, or starting scheduler-allocation/scheduler-preemption design. +2. Documentation audit (33 specs) is in progress — design.md corrections applied to ~8 HA/Raft specs; remaining ~24 specs need verification. +3. Broad documentation audit (S1278) still running — 28 docs with 126 findings, Tier 3 workflow targeting 32 closed spec design.md files. diff --git a/src/controller/CLAUDE.md b/src/controller/CLAUDE.md index 9afa17f1..e053e905 100644 --- a/src/controller/CLAUDE.md +++ b/src/controller/CLAUDE.md @@ -13,9 +13,18 @@ For full build system details, see `.kiro/steering/build-deployment.md`. Core source files handle distinct domains: - `job.c` — Job lifecycle management - `queue.c` — Queue state and control -- `cluster.c` — Cluster topology +- `cluster.c` — Cluster topology, request dispatch - `dataex.c` — Data exchange with other daemons +HA / replication / persistence subsystem (see HA Protocol section below for the election/replication/commit design; functions from these files are what the TLA+ mapping and safety invariants below actually cite): +- `ha.c` — HA state machine, election glue, promotion/fencing logic +- `ha_convergence.c` — post-election convergence and recovery +- `persist.c` — WAL persistence, durability, recovery +- `repl.c` — replication broadcast, majority-ack, WAL streaming +- `config_repl.c` — configuration replication and catch-up + +`src/controller/` has ~27 `.c` files total (`ls src/controller/*.c`); the remainder (`commit.c`, `ctrlstatus.c`, `join.c`, `worker.c`, `resources.c`, `task.c`, and others) cover commit-index advancement, status reporting, node join, and worker/resource/task bookkeeping. + Links against `libxce` from `src/lib/`. ## Conventions @@ -45,12 +54,12 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft | TLA+ Variable | XCE Code | Location | |--------------|----------|----------| -| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` | -| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` | -| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` | +| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:157` | +| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:158` | +| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:179` | | `log[i]` | WAL segments in `persist->wal_path` | `persist.c` | | `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` | -| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:141` | +| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:160` | | `nextIndex[i,j]` | `standby.join_applied_lsn` (replicator) | `repl.h` | | `matchIndex[i,j]` | `standby.acked_lsn` (replicator) | `repl.h` | @@ -58,23 +67,23 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft | TLA+ Action | XCE Code | Location | |------------|----------|----------| -| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:57` | -| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` | -| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` | -| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` | +| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:105` | +| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:178,433` | +| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:1615` | +| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:589` | | `AppendEntries(i,j)` — production send (LIVE) | `repl_broadcast` (per-entry fan-out on append, NACK-driven retry via `ha_raft_on_repl_nack`) + `ha_stream_wal_to_standby` (catch-up-range streaming) — the sole production `AppendEntries`-equivalent send/retry/backoff mechanism (raft-adapter-conformance Req 3.3) | `repl.c:295`, `config_repl.c:2351` | | `AppendEntries(i,j)` — nextIndex-driven sender (DORMANT, documented) | `raft_send_append_entries` is fully protocol-conformant but has ZERO production callers (only `raft_sim.c` and an `XCE_TESTING` test helper); its registered `on_send_append_entries` callback `ha_cb_send_append_entries_stub` is reachable only through it. Kept as an intentionally-dormant reference implementation (retire/wire-in both rejected, see `research.md`), guarded against silent regression by `ha_send_append_entries_stub_call_count` staying 0 while the production path fires (`test_ha_core_safety.c` task 4.2) | `src/lib/raft/raft.c:595`, `ha.c:2590` (stub), `ha.c:2588` (guard counter) | | `AppendEntries(i,j)` reply — WAL catch-up reject | Every WAL catch-up rejection sends an explicit `WalAckReply{success=0}` via `ha_send_wal_ack_reply_ex` back to the primary — never a silent drop (bug-1479; reply-construction boundary regression-locked by raft-adapter-conformance task 5.1) | `ha.c:3155` (fn), `ha.c:3190` (accept call site), `ha.c:3326` (reject call site) | -| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:750-780` (primary-side accept/reject branch + term-adoption) | +| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:1030-1055` (primary-side accept/reject branch + term-adoption) | | `RequestVote`/`PreVote(i,j)` reply — endpoint-failure routing, all timings | A vote/PreVote endpoint failure is always counted through the production tally (`vote_reply_dispatch` / `pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path — true both at connect-time (bug-1423/1424, regression-locked) and post-connect, i.e. a failure after connect but before a reply arrives (raft-adapter-conformance Req 4.2/4.3, closing that one remaining failure-timing gap). `vote_ep_err_handler` (real-vote) and `pre_vote_ep_err_handler` (PreVote) are separate handlers sharing one `CTRL_PRIMARY`-cleanup helper | `ha.c:7667` (`vote_ep_err_handler`), `ha.c:7705` (`pre_vote_ep_err_handler`), `ha.c:7627` (shared cleanup helper), registered at `ha.c:692` / `ha.c:831` | | `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` | -| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` | +| `UpdateTerm(i,j,m)` | Term CAS before granting/casting a vote — now lives in the Raft library, not the adapter | `raft.c:178-270` (`raft_on_vote_request`) via `raft_cb_persist`, `ha.c:2257` | > Full gap-analysis citations, options considered, and design rationale for the `raft-adapter-conformance` rows above (including the dormant-sender decision and the vote/PreVote handler-split decision): `.kiro/specs/raft-adapter-conformance/research.md` and `design.md`. This table stays the terse, current pointer; the evidentiary trail lives there. ### Safety Invariants Verified -**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term. +**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request`'s duplicate-vote check, `raft.c:231-237`). A candidate needs majority votes; each server votes at most once per term. **Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock). @@ -101,7 +110,7 @@ The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogL ### Protocol Verification Checklist When modifying HA protocol code (election, replication, heartbeat), verify: -- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:1784-1824`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:477` → `raft_on_vote_request`. Corrected 2026-07-18 — the prior `ha.c:1218-1230` citation had drifted onto unrelated post-election code.) +- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:2257`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:582` → `raft_on_vote_request`. Re-corrected 2026-08-08 — the `ha.c:1784-1824` and `cluster.c:477` citations from the prior 2026-07-18 correction had themselves drifted onto unrelated promotion/recovery-gate code.) - [ ] `currentTerm` is monotonic — never decreases - [ ] `commitIndex` never decreases (monotonic) - [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack) @@ -121,8 +130,6 @@ When modifying HA protocol code (election, replication, heartbeat), verify: Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference. -## Key Files - ## See Also - Shared library: `src/lib/CLAUDE.md` - Root guidelines: `CLAUDE.md` (repo root) diff --git a/src/docs/FUTURE_WORK.md b/src/docs/FUTURE_WORK.md index 6083a701..38a7bf9a 100644 --- a/src/docs/FUTURE_WORK.md +++ b/src/docs/FUTURE_WORK.md @@ -3,5 +3,5 @@ > **Moved (2026-06-10)**: All items merged into the single consolidated future-work document at > [`.kiro/steering/roadmap.md`](../../.kiro/steering/roadmap.md). > -> Section mapping: §1+§8 → admin-config-tool (backlog); §2 → superseded by ha-consistency-invariants Req 5; +> Section mapping: §1+§8 → admin-config-tool (nearly complete — see `.kiro/specs/admin-config-tool/`); §2 → superseded by ha-consistency-invariants Req 5; > §3, §4, §5, §6, §7 → Hardening / Direct-Implementation Items. diff --git a/src/docs/apiserver-backend-tasks.md b/src/docs/apiserver-backend-tasks.md index f5887434..de395a44 100644 --- a/src/docs/apiserver-backend-tasks.md +++ b/src/docs/apiserver-backend-tasks.md @@ -11,7 +11,7 @@ - [x] **#1 P0-1a: Add HostConfig + Cluster fields to models** - `models/clusters.go` — Added `HeartbeatInterval`, `Vcpus`, `InactiveHeartbeatCount` to `ClusterInfo` - - `models/hosts.go` — Changed `HostStateStr(0)` from `"OK"` to `"UNKNOWN"` + - `models/hosts.go` — This task changed `HostStateStr(0)` from `"OK"` to `"UNKNOWN"`; commit `d7afd72e` ("fix front end issues") later reverted that branch, so `HostStateStr(0)` currently returns `"OK"` again - `models/clusters_test.go` — New: verifies new fields serialize + existing fields unchanged - `models/hosts_test.go` — New: verifies `HostStateStr` for zero, flags, non-flag, combined - All 15 tests passing @@ -51,7 +51,7 @@ - `include/xceapi.h` — Added `xce_hostctrl()` declaration - `include/request.h` — Added `xce_hostctrl_req()` declaration - `lib/client.c` — Implemented `xcecli_hostctrl()` - - `lib/xceapi.c` — Implemented `xce_hostctrl()` with `cbuf_get()` wait + - `lib/xceapi.c` — Implemented `xce_hostctrl()` with `cbuf_get()` wait (a later, unrelated repo-wide change converted this to `cbuf_timedget()` with a 10s deadline) - `lib/request.c` — Implemented `xce_hostctrl_req()` (validates HOST_OPEN/HOST_CLOSE only, rejects HOST_TAG/UNTAG) - `lib/helper.c` — Added `Request_body_hostctrlmsg` case to `xce_request_body()` + `xce_request_msg_name()` - `controller/cluster.c` — Added `OPCode_hostcontrol` case → `do_hostcontrol_op()` @@ -77,7 +77,7 @@ - All tests passing (`controllers` package) - [x] **#9 P0-2d: Update stub.go with new function stubs** - - `backend/stub.go` — Added `StartHeartbeat(_ context.Context){}`, `SetHeartbeatInterval(_ int){}`, `EvalSelectForHost(_, _ string) error` (returns `errNoCGO`), `Cleanup(){}`, added `"context"` to imports + - `backend/stub.go` — Added `StartHeartbeat(_ context.Context){}`, `SetHeartbeatInterval(_ int){}`, `EvalSelectForHost(_, _ string) error` (returns `errNoCGO`), `Cleanup(){}`, added `"context"` to imports (`SetHeartbeatInterval` was later pulled out of `stub.go`/`clib.go` entirely into the shared, no-build-tag `backend/heartbeat.go`) - `backend/stub_funcs_test.go` — 4 TDD tests: `TestStartHeartbeat_StubNoOp`, `TestSetHeartbeatInterval_StubNoOp` (3 edge values), `TestEvalSelectForHost_StubErrNoCGO`, `TestCleanup_StubNoOp` - All backend tests passing @@ -106,7 +106,7 @@ ## P1-3: Heartbeat Keep-Alive - [x] **#13 P1-3: Heartbeat keep-alive implementation** - - `backend/clib.go` — Added `heartbeatIntervalSecs int32` (default 30), `SetHeartbeatInterval()`, `heartbeatFn` var, `StartHeartbeat(ctx context.Context)` goroutine with ticker + - `backend/clib.go` — Added `heartbeatIntervalSecs int32` (default 30), `SetHeartbeatInterval()`, `StartHeartbeat(ctx context.Context)` goroutine with ticker (within the same P1-3 commits, `heartbeatIntervalSecs`/`SetHeartbeatInterval()`/the tick loop were extracted into the shared, no-build-tag `backend/heartbeat.go` for CGO-free testability; `clib.go` now passes an inline closure to `runHeartbeat` rather than a named `heartbeatFn` var) - `backend/handler.go` — `OnClusterUpdated` calls `SetHeartbeatInterval(cluster.HeartbeatInterval)` after store update (zero-preserve: skips if interval==0) - `cmd/main.go` — Calls `backend.StartHeartbeat(ctx)` after `backend.RegisterToBackend(...)` - `backend/handler_test.go` — Tests for heartbeat interval update, zero-preserve, timer leak, test drain, store consistency @@ -123,7 +123,7 @@ - Input safety — comment limit 512 chars; select expression limit 4094 chars (1024 reduction deferred to P1-2b) - **CGO fix applied:** extracted `static` val helpers from `clib.go` preamble into `val_helpers.h` shared header; `convert.go` now includes it; added explicit `float64()` cast for `C.double` return -- [x] **#15 Integration test suite** — see [`apiserver-backend-integration-test-plan.md`](apiserver-backend-integration-test-plan.md) +- [x] **#15 Integration test suite** — plan doc `apiserver-backend-integration-test-plan.md` was later removed from `src/docs/` in an unrelated cleanup commit; see the test files below for current coverage - `backend/integration_test.go` — §1 (config push), §3 (resource values), §4 (heartbeat), §5 (cross-feature): 17 tests - `controllers/integration_test.go` — §2 (host control), §6 (RBAC), §8 (input safety): 17 tests + 1 skip (§8.1 pending P1-2b) - `store/concurrency_test.go` — §7 (thread safety): 3 concurrent-access tests diff --git a/src/docs/archive/cluster-config-split-analysis.md b/src/docs/archive/cluster-config-split-analysis.md index 03118c01..ee31a3bd 100644 --- a/src/docs/archive/cluster-config-split-analysis.md +++ b/src/docs/archive/cluster-config-split-analysis.md @@ -1,3 +1,5 @@ +> **ARCHIVED 2026-08-08.** This analysis's central subject — the ClusterConfig struct and its proposed 3-way Group1/PeersConfig/ControllerConfig split — no longer describes the schema. The closed `clusterconfig-schema-split` spec implemented a different, simpler design; `MasterCandidate`, `candidates`, and `ClusterConfig.port` (which this doc analyzes as fields to remove/split) are already gone from the schema. Kept for historical context only — do not treat any field/line reference below as current. + # ClusterConfig Schema Split Analysis > Analysis date: 2026-06-08. Branch: persistent_ha. diff --git a/src/docs/archive/job_peek.md b/src/docs/archive/job_peek.md index db628c04..24b68974 100644 --- a/src/docs/archive/job_peek.md +++ b/src/docs/archive/job_peek.md @@ -1,3 +1,5 @@ +> **ARCHIVED 2026-08-08.** This implementation plan is stale throughout — the C-layer fetchfile pipeline it describes as "not started" is fully implemented, `output.go` already uses the "Plan B" (CGO/FetchFile) path this doc proposes migrating to, and opcode/struct layouts have drifted from what shipped. Note: the shipped `handle_fetchfile()` worker handler has no path-traversal or job-ownership validation — a real gap this doc does not flag. Kept for historical context only. + # 项目实施方案：xcecli jobs peek ## 背景 diff --git a/src/docs/config-replication-design.md b/src/docs/config-replication-design.md index 93538329..4b9497fd 100644 --- a/src/docs/config-replication-design.md +++ b/src/docs/config-replication-design.md @@ -142,7 +142,8 @@ The full design is chosen for clusters where config integrity verification, chan auditability, and the push mechanism for online config updates are operational requirements. **WAL-based config replication (considered and rejected)**: XCE already has a WAL -replication path (`src/controller/repl.c`, ~340 lines) that streams job/queue/resource +replication path (`src/controller/repl.c`, 881 lines as of this audit — grown from the +~340 lines at design time) that streams job/queue/resource mutations from primary to standbys. Extending it to carry serialised `ClusterConfig` blobs would reuse the existing replication infrastructure. This approach is rejected for three reasons: (1) **Semantic mismatch** — WAL replication is continuous and @@ -187,8 +188,12 @@ and `validate_host()` are complete; see `FUTURE_WORK.md`): - ~170 LOC The `OP_upgradeConfig` handler is gated behind a compile-time flag -`CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled — see §5.3 safety gate). The -gate MUST NOT be set to 1 in any production build until BOTH prerequisites are met: +`CONF_ENABLE_UPGRADE_CONFIG`. **As designed**, the default is 0 (disabled — see +§5.3 safety gate) and the gate MUST NOT be set to 1 in any production build +until BOTH prerequisites are met. **As shipped**, `src/controller/config_repl.h` +hardcodes the flag to `1` (enabled) with no Meson build option to override it — +see the drift note in §5.3. Both prerequisites below remain unmet in the +current codebase: 1. **`validate_host()`** is implemented with a real authorisation check (not a stub that returns success for all peers). Minimum viable check: verify the peer's @@ -206,6 +211,11 @@ active, Phase 2 infrastructure compiled but inert). The gate is removed from the codebase when both prerequisites are merged and integration-tested. Phase 1 alone eliminates manual provisioning and config drift — the core problems stated in §1. +**This commitment is not currently met**: the codebase already ships with +`CONF_ENABLE_UPGRADE_CONFIG` hardcoded to `1` while `validate_host()` remains +the unconditional-success stub — the exact unsafe combination this gate was +designed to prevent. See the drift note in §5.3. + --- ## 3. Config Storage Layout @@ -221,7 +231,7 @@ would require a separate atomic-rename technique anyway. **Local filesystem only — NFS is explicitly unsupported.** The controller's working directory (including `conf/`, `data/wal/`, and `data/snap/`) MUST reside on a local -filesystem. `persist_init()` (persist.c:217-222) detects and rejects NFS mounts at +filesystem. `persist_init()` (persist.c:839-843) detects and rejects NFS mounts at startup with a FATAL error. Config replication assumes local `rename(2)` atomicity and `fsync(2)` durability guarantees; neither holds on NFS. No effort is made to support network or distributed filesystems for config storage. Cross-filesystem @@ -615,6 +625,18 @@ half-implemented admin endpoint from being reachable in production before its authorisation check is complete. The compile-time gate is removed once `validate_host()` has a production implementation (tracked in `FUTURE_WORK.md`). +**Drift from shipped code**: The compile-time default described above was the +design intent and is not what the current codebase does. As implemented, +`src/controller/config_repl.h` defines `CONF_ENABLE_UPGRADE_CONFIG` to `1` +unconditionally (comment: "enabled: standbys apply pushed config"), and +`src/controller/meson.build` has no `-Denable-upgrade-config` option or any +override of the flag — the compile-time default cannot be reverted to 0 +without editing the header. Meanwhile `validate_host()` (`src/controller/join.c`) +is still the literal stub `return 1;` for every peer. `OP_upgradeConfig` is +therefore reachable in any build of the current codebase without the +authorisation check this gate was meant to require. This should be treated as +an open safety gap, not a documentation nit. + ### 5.4 Primary handler The primary handles `OP_replConfig` via a new case in `cluster.c:myreq_handler()` (the @@ -1320,6 +1342,17 @@ load_primary_config() INFO("Starting as standby cluster_start() ``` +> **Implementation note — startup ordering changed**: The diagram above +> reflects this design's original proposal (election before +> `iomgr_init`/`endpoint_init`). The shipped implementation moves the listener +> and `iomgr_init`/`vtaskmgr_init` **before** election instead, closing a +> 2–3 second window during which the port was unreachable to election probes. +> See `.kiro/specs/ha-config-bootstrap/design.md` +> (`Net[iomgr_init + cluster_open_listener + vtaskmgr_init]` runs before +> `Quorum[am_i_primary_quorum]`) — this ordering change accompanies the +> replacement of `am_i_primary()` with Raft-style quorum election (§9 +> archival note). + ### 7.1 Primary config load — fail-fast behaviour `load_primary_config()` replaces `get_default_conf()` on the primary path: @@ -1385,6 +1418,10 @@ From the current `cluster_init()` (`cluster.c`): ### 8.2 What stays in phase 2 (after config) +> **Implementation note**: As shipped, `iomgr_init`/`endpoint_init` (and +> `vtaskmgr_init`) run **before** election, not after — see the ordering note +> in §7. The table below reflects this design's original phase-2 placement. + | Line(s) | Code | Phase | |---|---|---| | new (before iomgr) | Stop heartbeat listener: set `atomic_store(&c->heartbeat_listener_running, 0)` and `pthread_join(c->heartbeat_listener_tid, NULL)` — the join must complete before `iomgr_init` to free the listening port and prevent `EADDRINUSE` | 2 | @@ -1460,288 +1497,26 @@ the primary's startup log and tune `max_attempts` × `max_delay_ms` accordingly. ## 9. am_i_primary() Implementation -The current `am_i_primary()` stub (`ha.c:213`) returns `c->ha.my_priority == 0`, -which is never correct for a multi-node cluster — it never probes whether a -higher-priority candidate is actually alive. - -### 9.0 Simultaneous-boot symmetry breaking - -When all candidates start simultaneously, nobody is primary yet and nobody responds -to election probes. Without symmetry breaking, every candidate would expire its -deadline and self-declare primary — a multi-way split-brain. - -The mechanism has two parts: - -**1. Dedicated heartbeat listener (all self-declared primaries)** - -After Phase 1, **every candidate that `am_i_primary()` returns true for** launches a -minimal heartbeat listener thread before entering Phase 2. This is not limited to -priority-0 — if a lower-priority candidate wins the jitter race and self-declares -primary, its heartbeat listener is essential for the higher-priority true primary -to detect the split-brain during its own election probe. The listener is started -in a dedicated pthread immediately after `am_i_primary()` returns: - -```c -if (am_i_primary()) { - /* Start a minimal SSL listener that responds to OP_heartbeat probes. - * This lets lower-priority candidates detect us while Phase 2 initializes. */ - start_heartbeat_listener(c); /* dedicated pthread — non-blocking poll loop */ - load_primary_config(c); /* FATAL if missing */ - ... -} -``` - -The heartbeat listener is fully specified in the parent HA design -(`persistence-and-ha-design.md` Part 5). This section only describes the -integration point relevant to config replication: the listener must be active -before the standby's `fetch_config_from_primary()` connects, so the standby can -reach a primary that is still in Phase 2 initialisation. Summary of the -listener's properties (see parent design for full specification): - -- Binds the port from `--peers` using `c->ha.ssl_ctx` -- Non-blocking poll loop; accepts `OP_heartbeat` only; rejects other opcodes -- Does NOT depend on iomgr, vtaskmgr, persist, or any Phase 2 component -- Exits when the main thread clears `atomic_int heartbeat_listener_running` - (set to 0 before `iomgr_init` in Phase 2). The main thread calls - `pthread_join(heartbeat_listener_tid, NULL)` before `endpoint_init()` to - prevent EADDRINUSE. -- Registered in `cluster_t` (`c->heartbeat_listener_tid`, - `c->heartbeat_listener_running`) so `cluster_fini()` can join the thread - on any FATAL exit path before Phase 2 - -**2. Priority-based base delay + randomized jitter (all candidates)** - -Each candidate's election timeout has two components: a deterministic -**priority-based base delay** (higher priority = shorter delay) plus a small -random jitter for tie-breaking: - -```c -/* Priority-based base delay: priority-0 waits 0ms, priority-1 waits - * election_priority_step_ms, priority-2 waits 2× that, etc. - * This ensures the highest-priority node nearly always wins without probing. */ -long to_ms = c->ha.config.election_timeout_ms; -to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms; - -/* Small random jitter for tie-breaking between same-step candidates */ -long jitter_ms = (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms); -to_ms += jitter_ms; -``` - -With `election_timeout_ms = 2000`, `election_priority_step_ms = 500`, and -`election_jitter_max_ms = 200`, the effective deadlines are: -- Priority-0: 2000ms–2199ms (self-declares primary first) -- Priority-1: 2500ms–2699ms (probes priority-0 first; if alive, becomes standby) -- Priority-2: 3000ms–3199ms (probes priority-0 and priority-1 first) - -The priority-based base delay all but eliminates simultaneous split-brain during -normal boots: priority-0 declares primary and starts its heartbeat listener -~500ms before priority-1's deadline, so priority-1's probe finds a live primary. -The priority-based step and jitter together make simultaneous split-brain -vanishingly unlikely for N=3 in systemd-managed deployments: - -- Priority-0 fires at t=0–199ms (election_timeout_ms + jitter only) -- Priority-1 fires at t=500–699ms (step + jitter, probes priority-0 first) -- Priority-2 fires at t=1000–1199ms (2×step + jitter) - -Adjacent-priority candidates collide only if the lower-priority node starts so -late that its jitter-extended deadline precedes the higher-priority node's first -heartbeat response, which requires the 500ms step to be less than 2×J_max -(400ms) — false for the configured values. For three nodes to all self-declare -primary, all three must start within the same ~200ms jitter window, which is -bounded by OS scheduler precision and systemd startup ordering. - -The `election_priority_step_ms` is added to `ha_config_t` alongside the existing -`election_timeout_ms` and `election_jitter_max_ms`. The jitter-only fallback -(priority-0 with no step) retains the original behaviour when `election_priority_step_ms = 0`. - -The window is bounded: in the worst case (all jitter values 0ms apart), all -candidates declare primary. Once both reach Phase 2, the HA watchdog resolves -the split via priority comparison in heartbeat exchange (see parent HA design). -This is bounded by the time for the slower candidate to complete Phase 2 -(~4–13s per §8.5) plus one heartbeat interval (~500ms). - -**Split-brain window during simultaneous boot**: The heartbeat listener starts -immediately after `am_i_primary()` returns, but it is a minimal pre-iomgr -listener that only handles `OP_heartbeat`. **All candidates that self-declare -primary launch a heartbeat listener**, not just priority-0 — this is critical: -if a lower-priority candidate wins the jitter race and self-declares primary, -its heartbeat listener provides the response that resolves the split-brain -when the higher-priority (true) primary probes it. - -During simultaneous boot, all candidates start with `ha.term = 0` -(`atomic_store_explicit(&ha->term, 0ULL, ...)` in `ha_init()`, ha.c:34). -Term-based priority comparison cannot break symmetry at this stage — symmetry -breaking during simultaneous boot relies entirely on the jitter mechanism -(part 2 below). Term-based step-down becomes effective only after the first -successful election, when the winning primary increments its term. The jitter -window is therefore the primary defence against multi-way split-brain during -cold start; the heartbeat listener provides the detection channel but term -comparison is not yet discriminating. - -If a heartbeat listener fails to bind its port (`EADDRINUSE`), the candidate -logs a WARN indicating a likely split-brain condition (another candidate has -already bound the port) and continues initialization. This is expected during -simultaneous boot with overlapping jitter windows — the candidate that bound -the port first keeps it; the other continues to Phase 2 where the HA watchdog's -heartbeat exchange will resolve the split-brain. - -If a lower-priority candidate also -self-declares primary and enters Phase 2, the HA watchdog heartbeat exchange -will resolve the split-brain once both candidates reach Phase 2 — but this -requires the HA watchdog to compare candidate priority (from `--peers` ordering) -in addition to term, since both nodes have term=0 during simultaneous boot. -**This design implements** `ha_check_heartbeat_nocluster()` (ha.c:101-126) -with a priority comparison at term=0 so that a lower-priority self-declared primary -steps down when it receives a heartbeat from a higher-priority node with the -same term. Without this enhancement, the current elapsed-time-only check cannot -distinguish a legitimate primary from a split-brain peer during cold start. - -**CRITICAL DEPENDENCY — implemented by this design**: The `ha_check_heartbeat_nocluster()` -enhancement (priority comparison at term=0) is a **hard prerequisite** for correct -simultaneous-boot behavior. Without it, a lower-priority self-declared primary will -not step down when it receives a heartbeat from a higher-priority peer with the same -term — the split-brain is unbounded, not 4–13s. This enhancement is implemented in -this design (`ha.c` — see §12 files changed) alongside the election probing. The -parent HA design (`persistence-and-ha-design.md` Part 5) specifies the heartbeat -exchange requirements; this design provides the priority-comparison implementation -which the parent design references. Test coverage: -`test_am_i_primary_simultaneous_boot` in §14.1 validates priority-based step-down -at term=0 during simultaneous boot. - -**Implementation scope**: This design adds a `sender_priority` field to the -`OP_heartbeat` request in `controller.capnp`. The heartbeat listener (§9.0) -populates this field from `c->ha.my_priority`. `ha_check_heartbeat_nocluster()` -(ha.c) compares `sender_priority` against `ha->my_priority` when `term == 0`: -if the sender has higher priority (lower index in `--peers`), the local node -steps down to standby. The data source is the heartbeat message, not `ha_state_t`, -so both the schema change and the handler change are part of this design. - -The effective split-brain window is therefore the time for the -secondary-primary to complete Phase 2 and start processing heartbeats: ~4–13s -per §8.5. During this window both self-declared primaries have `iomgr_init` and -`endpoint_init` complete (per the Phase 2 ordering in §7), so both are accepting -client connections before `ha_promote` resolves the split-brain. Conflicting -client operations (job submissions, queue modifications) submitted to both -primaries during this window may diverge. This falls within the parent design's -acknowledged `split_brain_risk_accepted` model, with the caveat that the -window is longer than a single heartbeat interval during cold start. - -**Data divergence during split-brain**: The losing primary's WAL is discarded when it -steps down to standby — any client operations it accepted during the split-brain -window are lost. This design does not add a reconciliation protocol; the scope is -bounded to the existing HA model. Operators who cannot tolerate any data loss during -cold start should not submit client operations until the primary's `INFO("Promoted -to primary")` log message confirms Phase 2 is complete. The typical cold-start -window occurs only during initial cluster boot or full-cluster restart — not during -normal single-node failover where an existing primary with a term > 0 prevents the -split-brain scenario from occurring. - -**PRNG**: The jitter calculation uses an xorshift64 PRNG seeded from -`/dev/urandom`. The PRNG state is stored in `ha_state_t.jitter_seed` (one per -node, initialised during Phase 1). The xorshift64 implementation should be a -shared inline function in `ha.h` (or a small `prng.h` utility header) so both -the election probe (§9.1) and the heartbeat listener (parent HA design) use the -same generator without code duplication. The seed is initialised from -`/dev/urandom` (read 8 bytes) during Phase 1. -If `/dev/urandom` is unavailable, fall back to -`clock_gettime(CLOCK_MONOTONIC, &ts)` nanoseconds XOR'd with -`((uint64_t)priority << 56) ^ ((uint64_t)getpid() << 16)` with a `WARN` log. -The shift mixing avoids the correlation problem where two systemd-started -processes in the same nanosecond produce jitter values differing only by -their priority index (a small integer). The `time(NULL) ^ getpid()` fallback -is rejected because it produces nearly identical values for processes started -simultaneously by systemd (same second, sequential PIDs) — exactly the scenario -where symmetry breaking matters most. - -### 9.1 Algorithm (design doc §5.4) - -```c -int am_i_primary(cluster_t *c) -{ - /* Priority 0: unconditionally primary — highest-priority node does not probe. - * Safe because all controllers receive the same --peers list, so ordering is - * deterministic and priority-0 is unique across the cluster. */ - if (c->ha.my_priority == 0) return 1; - - /* Launch one pthread per candidate with priority < my_priority. - * Each thread makes a non-blocking connect + poll loop to the candidate, - * sends OP_heartbeat. If ANY thread gets a live ACK before the shared - * election_timeout_ms deadline → return 0 (standby). - * If the deadline expires with no ACK → return 1 (primary). */ - - atomic_int election_done = 0; - probe_ctx_t probes[MAX_HA_CANDIDATES]; - int n_probes = 0; - - for (int i = 0; i < c->ha.my_priority; i++) { - probes[n_probes++] = (probe_ctx_t){ - .candidate = &c->myconf->peers_cfg.candidates[i], - .ssl_ctx = c->ha.ssl_ctx, /* stashed in Phase 1 */ - .timeout_ms = c->ha.config.probe_connect_timeout_ms, - .election_done = &election_done, - .got_ack = 0, /* set to 1 by probe_thread on success */ - }; - pthread_create(&probes[n_probes-1].thread, NULL, - probe_thread, &probes[n_probes-1]); - } - - /* Wait for first ACK or deadline (with jitter for simultaneous-boot, see §9.0) */ - long to_ms = c->ha.config.election_timeout_ms; - to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms; - to_ms += (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms); - struct timespec deadline; - clock_gettime(CLOCK_MONOTONIC, &deadline); - deadline.tv_sec += to_ms / 1000; - deadline.tv_nsec += (to_ms % 1000) * 1000000L; - if (deadline.tv_nsec >= 1000000000L) { - deadline.tv_sec++; - deadline.tv_nsec -= 1000000000L; - } - int got_ack = wait_any_probe(probes, n_probes, &deadline); - - /* Signal all threads to stop (no pthread_cancel — unsafe through OpenSSL) */ - atomic_store(&election_done, 1); - for (int i = 0; i < n_probes; i++) - pthread_join(probes[i].thread, NULL); - - return got_ack ? 0 : 1; -} -``` - -### 9.2 Thread coordination - -`pthread_cancel()` is **NOT** used (cancellation through OpenSSL shutdown handlers -can deadlock). Each probe thread uses **non-blocking `connect()` + `poll()`**: - -1. `socket()` + `fcntl(O_NONBLOCK)` + `connect()` → expect `EINPROGRESS` -2. `poll()` with timeout `probe_connect_timeout_ms` (default 500ms) -3. Between poll attempts, check `atomic_load(&election_done)` — if set, exit cleanly -4. If `poll()` returns -1 and `errno != EINTR`, treat as probe failure — log at TRACE - level and exit thread with `*got_ack = 0` -5. After connect, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` polling; - on hard SSL failures (certificate expired, protocol mismatch, peer disconnect), - treat as probe failure — log at TRACE level and exit thread -6. If `send()` or `recv()` fails on the SSL socket, treat as probe failure -7. Send `OP_heartbeat`, receive ACK, set `*got_ack` via shared pointer - -The per-attempt poll timeout bounds the worst-case exit delay to one poll interval -per thread (~500ms). TCP connect timeout (kernel SYN retry) is not reached because -the thread checks `election_done` between each poll iteration and exits early. -Probe failures are expected when peers are down — only TRACE-level logging is -appropriate; ERROR logs would generate noise during normal operation. - -### 9.3 Election outcome logging - -``` -/* Higher-priority candidate alive: */ -INFO("HA election: candidate %s:%u responded — starting as standby (my_priority=%d)", - winner_host, winner_port, my_priority) - -/* No higher-priority candidate alive: */ -INFO("HA election: no higher-priority candidate responded within %ums — " - "starting as primary (my_priority=%d)", - election_timeout_ms, my_priority) -``` +> **SUPERSEDED.** This section (originally: the pthread-based priority +> election — `am_i_primary()`, `probe_thread()`, `wait_any_probe()` — plus the +> pre-iomgr `start_heartbeat_listener()`/`stop_heartbeat_listener()` heartbeat +> listener and the `ha_check_heartbeat_nocluster()` priority-based step-down +> used together for simultaneous-boot split-brain avoidance) described a +> design that was later replaced. Every function named above has been deleted +> from the codebase — see the deletion comments in `src/controller/ha.c` +> (`am_i_primary() deleted — replaced by ha_start_election_async +> (election-iomgr-refactor)`; `start_heartbeat_listener() / stop_heartbeat_listener() +> deleted — never called`; `ha_check_heartbeat_nocluster() / ha_watchdog_thread() / +> ha_start_watchdog() / ha_stop_watchdog() deleted — watchdog pthread replaced +> by ha_arm_heartbeat_timeout_vtask`) and `src/controller/ha.h`. Election is +> now a Raft-style quorum protocol (`ha_start_election_async()` → +> `raft_start_election()`) driving a 4-state FSM (`CTRL_INIT` / +> `CTRL_ELECTING` / `CTRL_PRIMARY` / `CTRL_STANDBY`), with `EPOLLRDHUP`-based +> failover detection replacing the heartbeat-timeout watchdog pthread. See +> "src/docs/config-replication-design.md" findings and the closed specs in +> `.kiro/specs/` — `ha-config-bootstrap`, `election-iomgr-refactor`, +> `raft-library`, `raft-ha-core-safety`, `ha-prevote` — for what shipped +> instead. --- @@ -2098,16 +1873,16 @@ See `FUTURE_WORK.md`. | File | Change | Complexity | |---|---|---| -| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @20 :UInt64` to `ClusterConfig` (§4.4); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low | +| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @12 :UInt64` to `ClusterConfig` (§4.4; implemented at field number @12, not the @20 originally planned at design time); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low | | `src/controller/main.c` | Update `conf_file` resolution: read `conf/current` pointer file, then open named versioned file (replaces flat `conf/current_config` path); remove stack-allocated `char conf_file[]` and the `myconf.conffile = conf_file` assignment — `conffile` is now heap-allocated via `asprintf()` in `load_primary_config()` or `fetch_config_from_primary()` and freed in `myconf_fini()`; add `--conf-stale-timeout` and `--conf-keep-n` CLI flag parsing; add `--peers` validation (WARN on empty host or port 0) | Trivial | | `src/include/myconf.h` | Add `config_version_tag` (uint64_t), `config_sha256[32]`, `conf_keep_n`, `config_generation` (uint64_t, §4.4), `conf_stale_timeout` (time_t, default 86400), and `config_stale` (bool) fields to `myconf_t` | Trivial | | `src/include/xce/cluster_def.h` | Add `restart_requested` (atomic_int, for config-change restart), `heartbeat_listener_tid`, `heartbeat_listener_running`, and `lost_and_found_job_count` (atomic_int, incremented during `persist_recover()` when jobs are routed to the synthetic queue, §6.5) fields to `cluster_t` | Trivial | | `src/controller/myconf.c` | Replace `get_default_conf()` with `load_primary_config()` (fail-fast + SHA-256 compute on primary path); config save/trim/validation functions moved to `config_repl.c`; add `myconf_fini()` to free `conffile` (now heap-allocated via `asprintf()` per §3.2), `cluster_config`, `config_version_tag`, and `config_sha256` | Low | | `src/controller/cluster.c` | Split `cluster_init()` into phase 1 and phase 2; gate config load on role; move `iomgr_init` and `endpoint_init` to phase 2; add `create_lost_and_found()` call before `persist_recover`; add `OP_replConfig`, `OP_replConfigPush`, and `OP_upgradeConfig` cases to `myreq_handler()` opcode dispatch; `cluster_run()` event loop checks `c->restart_requested` and exits cleanly when set; `cluster_fini()` joins heartbeat listener thread on any exit path and frees `standby_endpoints` | Medium | | `src/controller/myapp.c` | Restructure `myapp_run()`: call phase 1, then `am_i_primary()`, then config load/fetch, then phase 2. When `cluster_run()` returns cleanly (restart_requested=1), `myapp_run()` returns 0 to main() which exits; the systemd service manager restarts the process. Call `myconf_fini()` in the cleanup path before `cluster_fini()` | Medium | -| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0, §5.3) | Medium | +| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0 as designed; ships hardcoded to 1, §5.3 drift note) | Medium | | `src/include/byteorder.h` | **New file**: `be64toh()` fallback macro for non-glibc or pre-2.25 targets (§6.4 portability note). The Meson build checks for `be64toh()` at configure time; if unavailable, this header provides a portable implementation via `__builtin_bswap64()` or `le64toh()`/`ntohll()` | Trivial | -| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources | Trivial | +| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources. **Not implemented as designed**: both macros are defined via `#ifndef`/`#define` fallback directly in `config_repl.h` instead (with the enable flag hardcoded to `1`) — `meson.build` has no `-Denable-upgrade-config` option and no override of either macro. | Trivial | | `src/controller/ha.c` | Replace `am_i_primary()` stub with real election probing; add `probe_thread()`, `wait_any_probe()`; add `start_heartbeat_listener()` / `stop_heartbeat_listener()`; add `xorshift64_next()` (or use shared `ha.h` inline); enhance `ha_check_heartbeat_nocluster()` with priority comparison for simultaneous-boot symmetry breaking (§9.0) | High | | `src/controller/ha.h` | Add `ssl_ctx` field to `ha_state_t` (use `void*` to avoid pulling in `` — cast to `SSL_CTX*` at use sites in election probe and heartbeat listener); add `jitter_seed` (uint64_t, for xorshift64 PRNG) and `xorshift64_next()` inline function to `ha_state_t`; add `zarray_t *standby_endpoints` to `ha_state_t` for tracking connected standby controllers; add `probe_ctx_t` struct; add `election_jitter_max_ms` and `election_priority_step_ms` to `ha_config_t` with defaults `HA_DEFAULT_ELECTION_JITTER_MAX_MS 200` and `HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500` (§9.0 priority-based base delay); update "no new TLS" comment to note SSL transport is sufficient for trusted-LAN model; update `am_i_primary()` declaration; declare `ha_is_primary()` for runtime role check (§10.12) | Low | | `src/controller/join.c` | Update `handle_controller_join()`: remove `-ENOSYS` (controller-controller sessions are valid — config exchange is dispatched from `myreq_handler()`); register connected standby endpoints in `c->ha.standby_endpoints` after successful join handshake; handle endpoint removal on disconnect | Low | @@ -2123,7 +1898,8 @@ Note: the existing `src/controller/repl.c` (WAL replication — standby slot man ACK tracking, catch-up) is a separate module with a similar name but different domain. `repl.c` uses `replicator_t.standbys[]` for per-connection WAL ACK tracking; `config_repl.c` uses `ha_state_t.standby_endpoints` for config push fan-out. -See `src/controller/repl.c` (340 lines) for the existing WAL replication module. +See `src/controller/repl.c` (881 lines as of this audit, up from 340 at design time) +for the existing WAL replication module. **Total estimated LOC**: ~510 net new lines across all files (updated from ~480 to account for compile-time gate, `sanitise_cn()`, `config_contains_reserved_name()`, diff --git a/src/docs/config-replication-impl-status.md b/src/docs/config-replication-impl-status.md index ee89f128..43d93369 100644 --- a/src/docs/config-replication-impl-status.md +++ b/src/docs/config-replication-impl-status.md @@ -29,28 +29,40 @@ of 2026-05-22 (commits `2ed7e34`–`5053c8d`). | Raw capnp shims (`request.c`) | ✅ Done | `26393e9` | `xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` — bypass `xce_data_t` for raw bytes | | `handle_config_request()` (primary handler) | ✅ Done | `4ef2089` | Tag compare, SHA-256 double-check; sends real bytes via `xce_replconfig_reply_raw()` | | `fetch_config_from_primary()` + `do_one_config_fetch()` | ✅ Done | `2aae45f` | Full SSL roundtrip: non-blocking connect+poll, `xce_replconfig_req_ssl()`, SHA-256 verify, `conf_save_config()`, reload; 20-attempt exponential backoff | -| `handle_controller_join()` | ✅ Done | `8f4db30` | Registers endpoint in `standby_endpoints` under lock | +| `handle_controller_join()` | ⚠️ Reversed | `8f4db30` → `40f510fb` | Originally registered the endpoint in `standby_endpoints` under lock, but this was later identified as a live ownership-model violation (controller-controller peering is primary-initiated outbound only, per `ha-async-io.md`'s Connection Ownership). `40f510fb` (post-dates this doc) rewrote it to unconditionally reject inbound `role=HostRole_controller` joins with `-E_XCE_BAD_ROLE`; it no longer touches `standby_endpoints` | | WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records | | `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired | | API server rejects `lost_and_found` job submissions | ✅ Done | `bc92166` | `preCheckJobRequest()` returns "internal recovery queue" error before store lookup; 3 tests in `jobs_test.go` | --- -## Phase 2 (Remote config push) — COMPILED, GATED +## Phase 2 (Remote config push) — COMPILED, ENABLED BY DEFAULT -Phase 2 infrastructure ships compiled but inert. The `CONF_ENABLE_UPGRADE_CONFIG` -compile-time gate (default: 0) prevents `OP_upgradeConfig` from being reachable -in production builds until both prerequisites are met (see §5.3 and §2 of the -design doc). +> **⚠ Doc drift note (2026-08-08):** the paragraph and prerequisites list below +> describe Phase 2 as gated off pending two prerequisites. Neither prerequisite +> is met, but `CONF_ENABLE_UPGRADE_CONFIG` has defaulted to **1** (enabled) +> since commit `fd93997f` (2026-05-31) — 9 days after this doc's own "updated" +> date — and has never been reverted. `OP_upgradeConfig` is a **live, reachable +> path in current production builds**. See `src/controller/config_repl.h:27-29` +> and dispatch at `src/controller/cluster.c:772-807`. This is a real, +> outstanding risk, not a future gate — track via `.kiro/steering/roadmap.md`. + +Phase 2 infrastructure ships compiled and **enabled**. The +`CONF_ENABLE_UPGRADE_CONFIG` compile-time gate (default: **1** as of +`fd93997f`, 2026-05-31; originally 0) was intended to keep `OP_upgradeConfig` +unreachable in production builds until both prerequisites below are met (see +§5.3 and §2 of the design doc), but the gate now defaults to enabled with +neither prerequisite satisfied. | Component | Status | Notes | |---|---|---| -| `handle_upgrade_config()` | ✅ Wired, gated | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` | +| `handle_upgrade_config()` | ✅ Wired, enabled by default | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Reachable by default since `CONF_ENABLE_UPGRADE_CONFIG` defaults to 1 (`fd93997f`, 2026-05-31) | | `push_config_to_standbys()` | ✅ Timeout enforced | `f3babec` — `wait_all_with_timeout()` with `CONF_PUSH_TIMEOUT_MS` (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect | | `create_lost_and_found()` | ✅ Done | `91e66b8` | Inserts synthetic queue via `create_queue()` and host via `makeWorker()` before `persist_recover()`; gated with `#ifndef XCE_TESTING` for unit test binary | -**Prerequisites before enabling** (`CONF_ENABLE_UPGRADE_CONFIG=1`): -1. `validate_host()` implemented with real CN allowlist check (see `FUTURE_WORK.md §6`) +**Prerequisites — NOT met, despite the gate defaulting to enabled** +(`CONF_ENABLE_UPGRADE_CONFIG=1` in `src/controller/config_repl.h:27-29`): +1. ❌ `validate_host()` still an unconditional-accept stub — `src/controller/join.c:13-16` returns `1` for every host/key pair, i.e. no real CN allowlist check (see `FUTURE_WORK.md §6`, confirmed still open per `.kiro/steering/roadmap.md:82-83`) 2. Admin config tool exists and tested end-to-end (see `FUTURE_WORK.md §1`) --- @@ -59,14 +71,21 @@ design doc). | Suite | Tests | Covers | |---|---|---| -| `test_config_repl` | 24 | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips | +| `test_config_repl` | 101 (was 24 as of this doc's date) | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips — grown substantially since 2026-05-22 by `ha-repl-hardening`, `ha-term-safety`, `raft-adapter-conformance`, `config-upgrade-snapshot`, and assorted bug-fix coverage (e.g. bug-1475, bug-1476) | | `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire | | `test_ha_role` | (existing) | Role transitions, watchdog, redirect | | `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default | | `test_ha_cluster` | 5 | Cluster HA integration | | `test_p1_integration` | 5 | Phase 1 init structure | | All other suites | ~41 | 0 failures | -| **Total** | **84** | **0 failures, 0 ignored** | +| **Total** | **84 at doc date; stale — `test_config_repl` alone now exceeds this** | **0 failures, 0 ignored (as of 2026-05-22)** | + +> **Note (2026-08-08):** this table was accurate as of the doc's 2026-05-22 +> date but has not been kept in sync with ~2.5 months of subsequent test +> growth. `test_config_repl` alone now has 101 `RUN_TEST(...)` registrations +> (verified via `src/controller/testsuite/test_config_repl.c`), so the suite +> total is understated. Re-run and re-tally before relying on this table for +> current coverage figures. --- @@ -74,10 +93,10 @@ design doc). | Item | Severity | Tracked | |---|---|---| -| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority-step delay only | LOW | `FUTURE_WORK.md §7` | +| ~~Heartbeat listener (`start_heartbeat_listener`) is a no-op stub~~ — **RESOLVED/OBSOLETE**: `start_heartbeat_listener()`/`stop_heartbeat_listener()` have been deleted from the codebase entirely (never called, per `election-iomgr-refactor`; see `src/controller/ha.h:798`, `src/controller/ha.c:2916`). The priority-step-delay election model this limitation described has been superseded by real Raft-based election/prevote machinery (`raft-library`, `ha-prevote`, `election-iomgr-refactor` specs, all CLOSED) | — | superseded | | Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` | -| `OP_replConfigPush` standby-side handler not implemented — pushed config accepted only via pull on reconnect | LOW | `FUTURE_WORK.md` | -| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` | LOW | `FUTURE_WORK.md` | +| ~~`OP_replConfigPush` standby-side handler not implemented~~ — **FALSE, now implemented**: `handle_repl_config_push()` (`src/controller/config_repl.c:1308`) is fully implemented — validates the request, checks SHA-256 (line 1341), calls `conf_save_config()` (line 1357), activates the config, and sends an explicit `ReplConfigPushReply`. Dispatched from `OPCode_replConfigPush` in `src/controller/cluster.c:754-757`. Added by commit `3d076ccd` ("fix(ha): fix config replication to standbys; add replConfigPush handler"), which post-dates this doc | — | resolved | +| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` — **unverifiable as written**: `resolve_queue_or_lost_and_found` no longer appears anywhere in `src/controller/*.c`/`*.h`. `apply_wal_entry()` (`src/controller/persist.c:3281`) and the `WAL_CONFIG_MARKER` apply path (`persist.c:3851-3866`, added by `config-upgrade-snapshot` task 5) now treat lost-and-found entity re-homing as a deliberately separate, later step from apply/activation — suggesting this routing design was reworked or renamed rather than simply left unfinished. Needs re-investigation, not a straight carry-forward | LOW | `FUTURE_WORK.md` | --- diff --git a/src/docs/deployment.md b/src/docs/deployment.md index ab6f726b..bf6ac087 100644 --- a/src/docs/deployment.md +++ b/src/docs/deployment.md @@ -7,11 +7,11 @@ Covers building and running all four daemons (controller, worker, scheduler, api ``` xceadmin (Python CLI) --config push (TLS)--> controller (C, HA cluster, 1..N nodes) ^ ^ - plaintext join | | plaintext join + TLS join | | TLS join | | worker (C) scheduler (C) ^ - CGO-linked libxce client (TCP, HA-aware) + CGO-linked libxce client (TCP, TLS, HA-aware) | apiserver (Go, REST) <-- xcecli (Go CLI, REST) ``` @@ -23,16 +23,23 @@ xceadmin (Python CLI) --config push (TLS)--> controller (C, HA cluster, 1..N n - **xcecli** — Go CLI, talks to `apiserver` over REST, never touches the controller directly. - **xceadmin** — Python CLI, the only tool that compiles and pushes *cluster configuration* (`cluster.bin`) to the controller. Separate concern from the four daemons above. -Only controller↔controller HA replication traffic is TLS. Worker→controller, scheduler→controller, and apiserver→controller traffic is **plaintext**. +**Every** connection into the controller is TLS-encrypted — controller↔controller HA replication, and every `libxce`-client connection (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller). The controller runs a single `iomgr` bound to one server `SSL_CTX` (`myapp.c`); `endpoint_accept()`/`endpoint_connect()` in `src/lib/netio.c` have no plaintext code path at all — every accepted or outgoing socket is wrapped in `SSL_new()` before use. What's actually missing is **client authentication and identity verification**, not encryption: TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase (server and client contexts alike, no client certificates), so any of these connections buys you encryption-in-transit, not authentication of who's on the other end. ## Building +Two different Nix entry points, for two different jobs — don't conflate them: + +- **`nix develop --command` — single-component dev builds.** `/xce-build` is not a separate build system; it's `nix develop ./src --command bash src/tools/xce-build.sh ` under the hood, i.e. the hermetic dev shell driving meson/ninja for exactly one component. This is the reproducible way to build just `controller`, just `worker`, etc. +- **`nix build` — whole-stack production packaging.** A packaged Nix derivation, all-or-nothing (see Production Gaps item 6) — it always produces every daemon in one `result/`, never a single one. + ```bash -# Dev loop, per component (debug + ASAN, runs tests) +# Dev loop, per component (debug + ASAN, runs tests) — wraps `nix develop ./src --command ...` +/xce-build lib # libxce only /xce-build controller /xce-build worker /xce-build scheduler -/xce-build apiserver +/xce-build apiserver # also builds xcecli in the same pass (same meson step) +/xce-build xcecli # xcecli alone, if you don't need apiserver /xce-build xceadmin /xce-build # all components @@ -41,10 +48,10 @@ Only controller↔controller HA replication traffic is TLS. Worker→controller, # debug -> build.dbg/ -O0 -g3, no sanitizer, gdb-friendly # release -> build.release/ -O3, no sanitizer, build-only -# Production (Nix) +# Production packaging (Nix, whole stack only) nix build .#xcecore # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation nix build .#xcecore_debug # same, debug + ASAN (this flake's default package) -nix build .#xcecli # xcecli only (separate Go-only derivation) +nix build .#xcecli # xcecli, as its own separate Nix *package* derivation (Go-only, no C toolchain) — `/xce-build xcecli` above gets you the same binary without going through `nix build` at all ``` `nix build .#xcecore` is all-or-nothing — there is no per-daemon Nix target. It produces one `result/bin/` containing `xce_controller`, `xcecd` (+ `xcecd_worker`, `xcecd_virtual`), `xce_scheduler`, `xce_apiserver`, the `xceadmin` venv wrapper, and `xce` (the AppImage dispatcher script, see below). `nix build .#xcecli` is the one genuinely separate target. @@ -96,8 +103,8 @@ xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991 ### Known limitations (as of this writing) -- **`config push` against a real controller fails with a client-side decode crash**, live-verified against a freshly deployed single-node controller: the controller's `OPCode_upgradeConfig` dispatch is fine — `CONF_ENABLE_UPGRADE_CONFIG` defaults on, the earlier dispatch-gate polarity bug (bug-1468) was already fixed 2026-07-25, and the controller log shows the push genuinely committed, activated, and hash-verified server-side. The bug is in `xceadmin` itself: `xce_reply()` (`src/lib/request.c`) wraps every reply — including the upgradeConfig reply — in an outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope, but `discovery.py`'s `_redirect_target()` decodes the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope. The resulting pointer-offset misalignment throws `capnp.lib.capnp.KjException: ... non-list pointer where text was expected` from `push_config()`. This affects every opcode reply routed through `follow_redirects`, not just upgradeConfig. Fix belongs in `xceadmin`, not the controller. -- **`config push --dry-run`'s "OK" result is not meaningful evidence of anything**: `probe_config()`'s docstring itself notes `OPCode_probeConfigTag` has no dispatch case in `cluster.c`, so the "accepted" response is a client-side inference, not a real wire signal from the controller. Live-verified: dry-run returns exit 0 with no corresponding entry in the controller log at all. +- **~~`config push` against a real controller fails with a client-side decode crash~~** — **Fixed 2026-07-30.** `decode_reply()` (`src/xceadmin/src/xceadmin/discovery.py:46-77`) now unwraps the outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope before decoding the inner `RequestReply`. Both `_redirect_target()` and `push_config()` go through `decode_reply()`; the old bare-`RequestReply` decode path is gone. `decode_reply()` validates `body.which() == "replymsg"` and raises `ReplyDecodeError` on anything else. Live-verified against a real deployed controller. +- **~~`config push --dry-run`'s "OK" result is not meaningful evidence of anything~~** — **Fixed 2026-08-09 (bug-1506/1507).** The controller gained a real `OPCode_probeConfigTag` handler (`config_repl.c:handle_probe_config_tag`) that returns the current `config_version_tag` through the standard `Request`/`RequestReply` envelope. `probe_config()` now wraps the probe in a `Request` envelope and goes through `follow_redirects` → `decode_reply` (same as `push_config`), so `accepted=True` means the controller actually replied with a valid tag, and `--dry-run` reports the real generation number from the cluster. - No `config diff`, no rollback, no version history command. `xceadmin` today has exactly two subcommands: `config build` and `config push [--dry-run]`. - No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification. @@ -117,7 +124,7 @@ xce_controller \ - Omit `--peers` for a single-node, non-HA controller. - `--workdir` holds per-node `conf/` (cluster config) and `data/` (WAL + checkpoint) subdirectories. -- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, used exclusively for controller↔controller HA traffic. Generate a self-signed pair for local/dev use: +- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, and the controller loads it into the single server `SSL_CTX` used for **every** inbound connection it accepts: controller↔controller HA traffic *and* every worker/scheduler/apiserver/xceadmin client join. Generate a self-signed pair for local/dev use: ```bash openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \ -keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce" @@ -163,7 +170,7 @@ xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var log_dir: "/var/log/xce" ``` - `--peers "[name=]host:port,..."` enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join. -- No TLS on this connection — worker↔controller traffic is plaintext. +- Worker↔controller traffic is TLS-encrypted (via the shared `libxce` client, `xcecli_ctx_init()` → `xce_ctx_init()`), same posture as everything else in the stack: no client certificate, `SSL_VERIFY_NONE`. ## Deploying the Scheduler @@ -174,7 +181,7 @@ xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.lo - No config file at all — pure CLI (`argp`). - **Gotcha**: the default controller target is hard-coded to `localhost:9990`. There is no `-m`/`-p` equivalent — the *only* way to point the scheduler at any other host or port, even a single non-HA controller, is `--peers`. Forgetting `--peers` silently targets `localhost:9990` regardless of where your controller actually is. - `-v` is repeatable for more verbose logging; default log file is `/tmp/xce_scheduler.log`. -- No TLS — plaintext to the controller, same as the worker. +- TLS-encrypted to the controller, same as the worker (same shared `libxce` client path). - Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary through the picker plugin interface (`builtin_fcfs` is the registered default), not dynamically loaded. ### Session recording (live cluster) @@ -241,7 +248,7 @@ Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fas ### Relationship to the controller -apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. No TLS on this connection. +apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. This connection is TLS-encrypted, same as every other `libxce` client (no client certificate, `SSL_VERIFY_NONE`) — do not confuse it with apiserver's own separate HTTPS termination for its REST API, covered next. ### TLS @@ -277,18 +284,18 @@ xceadmin config push cluster.bin --controller localhost:9991 Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features: 1. **No process supervision is shipped.** No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit *sketch* exists in `src/docs/config-replication-design.md` but has a wrong binary name (`xcecontroller` vs the real `xce_controller`) and specifies `Type=notify`, which the controller does not support (it never calls `sd_notify`) — using it as written will hang at startup and time out. If you adapt it, switch to `Type=simple` and fix the binary name. -2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**, including an `admin`/`admin` account and a named-user `cluster_admin` account. Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead. -3. **Only controller↔controller traffic is encrypted.** Worker→controller, scheduler→controller, and apiserver→controller are all plaintext, with no path to enable TLS on them today. -4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. -5. **`xceadmin config push` doesn't work against a real cluster yet** (see xceadmin's Known Limitations above) — the controller side is fine; `xceadmin`'s `discovery.py` fails to unwrap the `Request` envelope around every reply, crashing on decode. -6. **`nix build .#xcecore` is all-or-nothing** — there's no way to build just one daemon via Nix; use `/xce-build ` for per-component builds. +2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**: four accounts total — `admin`/`admin` and `op7x2k9`/`CHANGE_ME_dev_only` (both role `cluster_admin`), plus `user1`/`user1` and `user2`/`user2` (role `basic_user`). Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead. +3. **All controller-bound traffic is encrypted but not authenticated.** Controller↔controller HA replication and every `libxce` client (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller) run over TLS, but `SSL_VERIFY_NONE` is used throughout — no client certificates, no hostname/identity verification. This buys confidentiality and tamper-resistance in transit, not authentication of the peer; anyone who can reach the port and complete a TLS handshake is treated as trusted the moment the transport-level join succeeds. +4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. (This is apiserver's REST-facing HTTPS, separate from its TLS-encrypted `libxce` connection to the controller in item 3.) +5. **`xceadmin config push` against a real cluster** — the decode-crash bug is fixed (2026-07-30, `decode_reply()` unwraps the `Request` envelope), but `--dry-run` still has no real controller-side handler (see Known Limitations item 2). +6. **`nix build .#xcecore` (the packaged derivation) is all-or-nothing** — there's no per-daemon Nix package target. For a single component, use `/xce-build `, which itself runs inside the Nix dev shell (`nix develop ./src --command ...`) rather than through `nix build`. ## Quick Reference | Daemon | Binary | Config | Controller-pointing flag | TLS | |---|---|---|---|---| -| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, controller↔controller only | -| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | No | -| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | No | -| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | Optional, self-terminated | +| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, all inbound (peers + clients), no client cert | +| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | Yes, no client cert | +| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | Yes, no client cert | +| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | To controller: yes, no client cert. Own REST API: optional, self-terminated | | xceadmin | `xceadmin` (Python) | none — takes YAML source dir | `--controller host:port` | Yes, no client cert | diff --git a/src/docs/p1-2-host-select.md b/src/docs/p1-2-host-select.md index 1b3954ef..bedaf31e 100644 --- a/src/docs/p1-2-host-select.md +++ b/src/docs/p1-2-host-select.md @@ -33,7 +33,8 @@ The full feature requires two layers: **Files implemented:** - `include/resreq.h` — expression-tree types, enums, `extern` declarations. - `lib/xce_select.peg` — PEG grammar for select boolean expressions. -- `lib/xce_select.c` — `parse_select()`, `eval_select_simple()`, CSE tree. +- `lib/resreq.c` — `parse_select()`. +- `lib/xce_select.c` — `eval_select_simple()`, CSE tree. - `lib/meson.build` — `xce_select.peg` in build. ### #12 P1-2b: Go EvalSelectForHost + FilterHostByResReqSelect (remaining) diff --git a/src/docs/persistence-and-ha-design.md b/src/docs/persistence-and-ha-design.md index f4dc2b38..0865f213 100644 --- a/src/docs/persistence-and-ha-design.md +++ b/src/docs/persistence-and-ha-design.md @@ -277,7 +277,7 @@ typedef struct { metric for each auto-terminated job. */ uint32_t finished_retain_s;/* retain finished jobs for N sec (default 3600) */ uint32_t worker_retain_s; /* retain unavailable workers for N sec (default 86400) */ - uint32_t sync_timeout_ms; /* sync replication ACK timeout (default 50) */ + uint32_t sync_timeout_ms; /* sync replication ACK timeout (default 500) */ } persist_config_t; ``` @@ -326,7 +326,7 @@ The controller only handles connections from three peer types: End users never connect directly to the controller — the API server provides traffic control, rate limiting, and request validation. This has important implications: -- **Sync replication**: Only triggered by job submit from API servers. At 1–3 API server connections, vcpu pool exhaustion from blocking `repl_wait_ack()` is not a concern — the wait is bounded by `sync_timeout_ms` (default 50ms) and only affects the vcpu handling that specific API server's request. +- **Sync replication**: Only triggered by job submit from API servers. At 1–3 API server connections, vcpu pool exhaustion from blocking `repl_wait_ack()` is not a concern — the wait is bounded by `sync_timeout_ms` (default 500ms) and only affects the vcpu handling that specific API server's request. - **Connection count**: Workers (100s–1000s) + scheduler (1–2) + API servers (1–3). The scheduler and API server connections are long-lived and stable. - **All connections use SSL**: The controller's iomgr enforces SSL on all connections (see `netio.c`). Replication channels must also use this pipeline, not raw TCP. @@ -334,684 +334,7 @@ End users never connect directly to the controller — the API server provides t ## Part 3: Checkpoint — Dual zarray COW -### 3.1 Why Not fork()? - -The original design considered `fork()` + COW for checkpoint snapshots. This is problematic for a multithreaded controller: - -- After `fork()`, only the calling thread survives. If another thread held a mutex (e.g., the WAL writer held `persist.lock`), the child inherits a **locked mutex with no owner** — instant deadlock. -- kvmgr's internal `my_rwlock_t` is not fork-safe. The child must not take any lock, but serializing kvmgr state likely requires traversal that touches locked structures. -- Linux memory overcommit: if the parent modifies many pages during the child's serialization, the OS may OOM-kill the child (or parent). - -### 3.2 Why Not Online Compaction or Snapshot-and-Continue? - -Two alternatives to the dual-zarray COW approach were considered: - -**Online compaction** (e.g., log-structured merge, shadow paging with page-level COW): requires per-page dirty tracking or a WAL-driven snapshot mechanism. XCE's kvmgr uses pointer-based structs (`job_t*`, `worker_t*`) — page-level COW would require every hot pointer to be remapped across page boundaries, adding complexity the codebase does not support today. - -**Snapshot-and-continue** (serializing while holding a read lock the whole time): viable but blocks mutations for the duration of serialization. For 10K jobs at ~1KB each, serialization takes ~10–100ms. Blocking all job submits for 100ms at checkpoint time is worse than the brief freeze/thaw window (≤100μs for 10K entries — see Part 3.8). - -**Decision: dual-zarray application-level COW.** The "stop-the-world" cost is only at freeze (O(1) pointer swap, ~ns) and thaw (O(shadow_caps) pointer copies, ~100μs). Serialization of the shadow arrays happens with no locks held, so mutations continue unblocked. This gives checkpoint-without-blocking at the cost of ~2× peak memory during COW. At XCE's scale (10K jobs × ~1KB = ~10 MB), this is negligible. - -### 3.3 Design: Application-Level COW via Dual zarray - -XCE's kvmgr uses a two-level structure: `kvtab` (hash table for key→index lookup) + `data` (xzarray — dense pointer array for iteration and index-based access). This structure makes application-level COW practical. - -**Key property**: The hash table maps keys to **stable indices** in the zarray. COW only needs to happen at the zarray level. The kvtab never needs to be duplicated. - -``` -kvmgr structure: - kvtab: key ──hash──▶ index (stable, never changes for a given key) - data: xzarray of { inuse, data_ptr } indexed by index - -COW operation: - 1. Swap: data → shadow (frozen, immutable) - 2. New data = empty array with same capacity, all inuse=0 - 3. During COW: reads check data first, fall through to shadow - 4. After serialization: merge shadow back into data, apply GC -``` - -### 3.4 xzarray COW Extensions - -**`xzarray_t` is file-private in `zarray.c`**: the struct definition is internal to the RTTI type system (line 353–369), and external code only accesses it through the `typeinfo_t xzarray_type` interface. Rather than expose the struct, COW state is stored in `xzarray_private_t` (the RTTI private section, allocated by `makeInstance()` alongside each xzarray instance): - -```c -/* In zarray.c — extend xzarray_private_t */ -typedef struct { - mylock_t lock; - xzarray_item_t* shadow; /* Frozen snapshot (NULL when not in COW) */ - size_t shadow_caps; /* Shadow array capacity */ -} xzarray_private_t; -``` - -New freeze/thaw operations are exposed by extending the RTTI `intf_varray_t` interface. The rtti library is maintained in-house (`/nix/store/...-rtti-0.0.1/`), so adding new function pointers is a controlled change — version bumped to 0.1.0 (see Appendix A.9 for the full rtti enhancement proposal). - -Add two new function pointer types and slots to the `intf_varray_t` interface in `rtti/typeinfo.h`: - -```c -/* retain_fn_t is defined in rtti/typeinfo.h (see Appendix A.9) */ -typedef int (*vfreeze_fn_t)(meta_obj_t* obj); -typedef int (*vthaw_fn_t)(meta_obj_t* obj, retain_fn_t should_retain, void* arg); - -typedef struct { - vstore_fn_t vstore; - vget_fn_t vget; - vget_vec_fn_t vget_vec; - vput_fn_t vput; - vput_vec_fn_t vput_vec; - vclear_fn_t vclear; - vfreeze_fn_t vfreeze; /* NEW — freeze array for COW snapshot */ - vthaw_fn_t vthaw; /* NEW — thaw + merge + GC after snapshot */ -} intf_varray_t; -``` - -These map to `xzarray_freeze()` and `xzarray_thaw()` implementations in `zarray.c`, which access shadow state through `RTTI_OBJ_TO_PRIV()`. - -```c -static int xzarray_freeze(meta_obj_t* obj) { - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - - priv->shadow = a->data; - priv->shadow_caps = a->caps; - - a->data = calloc(a->caps, sizeof(xzarray_item_t)); - if (a->data == NULL) { - /* Rollback: restore original data from shadow. - * a->count still holds the pre-freeze value (not modified above), - * so no restoration needed for count. */ - a->data = priv->shadow; /* restore active pointer */ - a->pos = 0; - priv->shadow = NULL; - priv->shadow_caps = 0; - return -ENOMEM; - } - a->count = 0; - a->pos = priv->shadow_caps; /* start past shadow to prevent vstore index collisions */ - return 0; -} -``` - -When `shadow != NULL` (i.e., `priv->shadow != NULL`), the array is in COW mode. The existing `xzarray_item_t` is unchanged: - -```c -typedef struct { - _Atomic _Bool inuse; /* C11 atomic flag — true when slot holds data. - * Changed from unsigned int :1 bitfield because - * C11 atomic operations require an addressable - * object, and bitfields are not addressable. - * _Atomic _Bool is lock-free on all supported - * platforms. */ - void* data; -} xzarray_item_t; -``` - -### 3.5 Modified zarray Operations During COW - -**Critical constraint**: During COW mode, `vclear` is never called (see Part 4). This means `inuse=0` in the active array always means "slot not yet touched" — there is no tombstone ambiguity. - -**Runtime enforcement**: `xzarray_clear()` must assert `priv->shadow == NULL` at entry. If `vclear` is invoked while COW is active, the array's invariant is violated (a hole in the active array would cause `vget` to fall through to stale shadow data). The assertion converts a silent data corruption bug into a loud crash during development: - -```c -static void xzarray_clear(meta_obj_t* obj) { - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - assert(priv->shadow == NULL && "xzarray_clear called during COW — deferred cleanup required"); - /* ... existing clear logic ... */ -} -``` - -#### vget — Two-Level Lookup - -```c -static int xzarray_get(meta_obj_t* obj, int idx, void** data) { - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - - /* COW mode: check active first, fall through to shadow */ - if (priv->shadow != NULL) { - if (idx < a->caps && atomic_load_explicit(&a->data[idx].inuse, memory_order_acquire)) { - *data = a->data[idx].data; - return 0; - } - if (idx < priv->shadow_caps && atomic_load_explicit(&priv->shadow[idx].inuse, memory_order_acquire)) { - *data = priv->shadow[idx].data; - return 0; - } - return -ENOENT; - } - - /* Normal path: unchanged */ - /* ... existing code ... */ -} -``` - -#### vput — Write-Through to Active at Same Index - -```c -/* vput during COW: write to active at the same index. - The kvtab still maps key→same index, so the next vget - finds the updated value in active first. */ -static int xzarray_put_vec(meta_obj_t* obj, int count, int* idxs, void** data) { - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - - /* Ensure active has capacity */ - int maxid = 0; - for (int i = 0; i < count; i++) - if (idxs[i] > maxid) maxid = idxs[i]; - xzarray_ensure_items(a, maxid + 1); - - for (int i = 0; i < count; i++) { - int idx = idxs[i]; - if (atomic_load_explicit(&a->data[idx].inuse, memory_order_acquire) && a->killer) - a->killer(a->data[idx].data); - a->data[idx].data = data[i]; - atomic_store_explicit(&a->data[idx].inuse, true, memory_order_release); - } - return 0; -} -``` - -Since indices are stable (hash table maps key→same index always), a `vput(42, new_data)` writes to `active[42]`. The hash table still says key→42. Reads check `active[42]` first and find the updated value. The shadow's `shadow[42]` is silently masked. - -**Lock protocol design**: The rtti `intf_lock_t` provides a thin abstraction — `lock()` / `unlock()` function pointers that each type implements. Whether vget acquires a read lock internally is an implementation decision, not an rtti contract. The current design uses the kvmgr `my_rwlock_t` as the outer coordination lock: callers acquire the kvmgr READ lock before vget and release it after they are done with the returned pointer. The xzarray `mylock_t` mutex is used for individual array mutations (vstore, vput, vclear). This two-level locking (kvmgr rwlock for container-level coordination, xzarray mutex for array-level mutations) is the established pattern — see `handle_jobsubmit()` in `job.c:455-525` for the reference implementation. vget does not acquire the xzarray mutex because the caller already holds the kvmgr READ lock, which prevents concurrent freeze/thaw from modifying the array structure. - -**Memory ordering during COW serialization (CRITICAL — must be implemented)**: During the no-locks-held serialization phase (Part 3.8), `vput()` writes to `active[idx]` while the serialization thread reads `shadow[]` (which is immutable). These two accesses are to different memory regions, so there is no data race on the shadow array. However, concurrent `vput()` and `vget()` on the active array require careful memory ordering. `vget()` reads `active[idx].inuse` then `active[idx].data` as separate fields — a `vput()` that writes `.data` before setting `.inuse = 1` could cause `vget()` to see `inuse = 1` with a stale `.data` pointer if the compiler reorders the stores or the CPU reorders their visibility. - -**Why x86-64 alone is NOT sufficient**: The x86 memory model prevents the CPU from reordering stores to the same location, but `.data` and `.inuse` are **different fields**. The C compiler is free to reorder stores to different fields of a struct regardless of target architecture — the `volatile` qualifier on the struct would prevent this but imposes performance costs on all accesses. Without `volatile`, the compiler may emit the store to `.inuse` before the store to `.data`, and a concurrent `vget()` would see `inuse = 1` with an uninitialized or stale `.data` pointer — a use-after-free. - -**Required implementation**: Use C11 atomics for the `.inuse` field to enforce release-acquire ordering between `vput()` (writer) and `vget()` (reader): -- Change `inuse` from `unsigned int :1` bitfield to `_Atomic _Bool` (C11 atomics cannot operate on bitfields — the address-of operator is undefined for bitfields, and `_Atomic _Bool` is lock-free on all supported platforms). -- `vput()` writes `.data` first, then stores `.inuse` using `atomic_store_explicit(&a->data[idx].inuse, true, memory_order_release)`. -- `vget()` reads `.inuse` using `atomic_load_explicit(&a->data[idx].inuse, memory_order_acquire)`, and only accesses `.data` if `.inuse == true`. -- The release-acquire pair ensures `.data` is visible to `vget()` when it sees `.inuse == true`. - -This is required on ALL architectures, not just ARM — compiler reordering is sufficient to cause the race even on x86-64. The kvmgr READ lock in `vget()` callers provides happens-before between `vput()` and `vget()` on the **same vcpu thread** (serialized by lock/unlock), but does not prevent the compiler from reordering stores within a single `vput()` call. The atomic store on `.inuse` is the only reliable barrier against both compiler and CPU reordering. - -**Concurrent `vstore` race under READ lock (CRITICAL)**: During COW mode, multiple vcpu threads may hold the kvmgr READ lock simultaneously and call `vstore()`. The kvmgr READ lock does NOT serialize these calls — it allows concurrent readers. Two concurrent `vstore()` calls can both see `inuse==0` at the same `idx` during the scan phase, and the second silently overwrites the first's `.data` — silent data loss. **Fix**: During COW mode, the entire `xzarray_store()` find-slot-and-write sequence is serialized via the inner xzarray mutex (`mylock_lock(&priv->lock)`). The mutex is acquired at the top of the COW-mode path and released on all return paths. This is the ONLY place in the COW protocol where the inner mutex is required — normal-mode vput and vstore rely on the kvmgr WRITE lock for serialization (unchanged). See the full `xzarray_store()` implementation below (R-22 fix). - -#### vstore — Must Skip Shadow Indices - -**Design invariant**: After `xzarray_freeze()`, the new active array is empty (`count=0`, `pos=shadow_caps`). The `vstore` scan during COW mode must use a **linear scan bound to `[shadow_caps, a->caps)`** — it must NOT wrap around to index 0 when reaching `a->caps`. If all slots in `[shadow_caps, a->caps)` are occupied, `vstore` grows the array via `xzarray_ensure_items()` and assigns from the new capacity. This prevents index reuse below `shadow_caps`, which would cause silent data corruption when `vget` falls through to the stale shadow entry. - -**Why circular scan is unsafe during COW**: The existing `xzarray_store()` at `zarray.c:456-490` wraps from `idx >= a->caps` to `idx = 0`. With `pos = shadow_caps`, after enough `vstore` calls fill `[shadow_caps, a->caps)`, the next store wraps to index 0 — but index `i < shadow_caps` may have an `inuse=0` slot which `vstore` reclaims. The kvtab now maps `new_key → i`, while `shadow[i]` still holds the old entity's pointer. A `vget(i)` finds `active[i].inuse == 0` (recently assigned but not yet visible to a concurrent reader) or `active[i]` holds the new entity while `shadow[i]` holds stale data — either case produces wrong results. - -**Fix**: Add a COW-mode flag check in `xzarray_store()`: when `priv->shadow != NULL`, scan only the range `[pos, a->caps)` without wrapping. If no free slot is found, grow the array: - -After thaw, `pos` is reset to the normal low-water scan (or left at the current high-water mark — the existing vstore scan-forward logic handles this correctly). - -#### vstore — Linear Scan During COW (No Wrap) - -```c -static int xzarray_store(meta_obj_t* obj, void* data) { - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - int idx; - - if (priv->shadow != NULL) { - /* COW mode: linear scan from pos without wrapping. - * Must not reuse indices < shadow_caps — those belong to shadow. - * inuse must be read with atomic_load_explicit(acquire) to pair - * with vput's atomic_store_explicit(release) — both vstore and - * vput hold the kvmgr READ lock concurrently, so the atomic - * ordering (not the lock) provides inter-reader visibility. - * - * CRITICAL: The entire find-slot-and-write sequence is serialized - * via the inner xzarray mutex (priv->lock). Without this, two - * concurrent vstore callers can both see inuse==0 at the same idx - * during the scan, and the second silently overwrites the first's - * .data — a silent data loss bug. The mutex is only held during - * COW mode; normal-mode vput/vstore rely on the kvmgr WRITE lock - * for serialization (unchanged). */ - mylock_lock(&priv->lock); - for (size_t idx = a->pos; idx < a->caps; idx++) { - if (atomic_load_explicit(&a->data[idx].inuse, - memory_order_acquire) == 0) { - a->data[idx].data = data; - atomic_store_explicit(&a->data[idx].inuse, true, memory_order_release); - a->count++; - a->pos = idx + 1; - mylock_unlock(&priv->lock); - return idx; - } - } - /* No free slot in [shadow_caps, a->caps) — pre-grown range is full. - * Do NOT call xzarray_ensure_items() (realloc) here: realloc moves - * a->data, invalidating pointers held by concurrent vget() callers - * that only hold the kvmgr READ lock (not the inner xzarray mutex). - * The 1.5× pre-grow at freeze time is designed to make this path - * rare (< 100ms COW window at normal submit rates). Return -ENOMEM - * so the caller logs WARN and the client retries after thaw. */ - mylock_unlock(&priv->lock); - return -ENOMEM; - } - - /* Normal path: unchanged circular scan */ - /* ... existing xzarray_store() code ... */ -} -``` - -#### Iteration — Merge Walk of Both Arrays - -```c -static int xzarray_iter_next(meta_obj_t* obj, void* ctx, void** data) { - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - size_t* pos = (size_t*)ctx; /* iterator position — caller-allocated cursor */ - - if (priv->shadow != NULL) { - /* COW mode: walk both arrays, active takes precedence */ - size_t max_caps = (a->caps > priv->shadow_caps) ? a->caps : priv->shadow_caps; - for (size_t i = *pos; i < max_caps; i++) { - if (i < a->caps && atomic_load_explicit(&a->data[i].inuse, memory_order_acquire)) { - *data = a->data[i].data; /* active wins — shadow[i] intentionally skipped */ - *pos = i + 1; - return RTTI_ITER_OK; - } - if (i < priv->shadow_caps && atomic_load_explicit(&priv->shadow[i].inuse, memory_order_acquire)) { - *data = priv->shadow[i].data; /* shadow only (no active entry at i) */ - *pos = i + 1; - return RTTI_ITER_OK; - } - } - return RTTI_ITER_NO_DATA; - } - - /* Normal path: unchanged */ - /* ... existing code ... */ -} -``` - -**Serialization must iterate shadow only**: During the no-locks-held serialization phase (Part 3.8), the iterator must only walk the shadow array (`priv->shadow[0..shadow_caps-1]`), since shadow is immutable and active[] may be concurrently modified by vput. The `xzarray_item_t` struct has a bitfield (`inuse : 1`) and a pointer (`data`) — these are not atomically written, so reading active[] without locks is a data race. New entries written to active[] during serialization are covered by the WAL and will be included in the next checkpoint's shadow. The serialization code should use a shadow-only iterator (bound: `0..shadow_caps-1`), not the merge-walk iterator above. The merge-walk iterator is for normal-mode queries where callers hold the kvmgr READ lock. **The kvmgr READ lock acquisition provides the necessary memory ordering to see vput's release-ordered stores to `active[].inuse`. Iterating without the READ lock is a data race.** - -**COW invariant — new keys may hash below shadow_caps**: New entities added during COW mode (via vstore) are allocated at indices `>= shadow_caps` (the linear scan starts at `pos = shadow_caps`). However, a `vput` for a new hash table key can map to any index, including indices below `shadow_caps`. This is safe because: (1) vput writes to `active[i]` at the same index, (2) vget checks active first and finds the new entry, (3) thaw merges shadow and detects `active[i].inuse == 1`, taking the active-wins branch and freeing the stale shadow entry. The COW vstore linear scan only prevents index *reuse* from the free list below `shadow_caps` — it does not prevent new hash-to-index mappings in that range via vput. - -### 3.6 Freeze — Start COW Phase - -```c -static int xzarray_freeze(meta_obj_t* obj) { - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - - /* Current data becomes shadow (frozen, immutable) */ - priv->shadow = a->data; - priv->shadow_caps = a->caps; - - /* Allocate new active with 1.5× capacity (pre-grown to avoid realloc during COW). - * - * WHY OVER-ALLOCATE: During COW, vstore() scans [shadow_caps, a->caps) for free - * slots. If a->caps == shadow_caps (typical case right after freeze), that range - * is empty and any vstore() would need to call xzarray_ensure_items() → realloc(). - * realloc() moves a->data, invalidating any pointer that a concurrent vget() caller - * holds into the array (vcpu threads can hold READ lock simultaneously, so the - * growth lock — the inner mutex — does not block concurrent vget() access). - * - * By pre-allocating 1.5× shadow_caps slots, the range [shadow_caps, 1.5×shadow_caps) - * is immediately available for vstore(). For a typical COW window (< 100ms), this - * range is almost always sufficient (new entity arrival rate is bounded by the - * job submit rate). If the pre-grown range fills up, vstore() returns -ENOMEM and - * the caller logs a WARN — the submit is rejected transiently and the client retries - * after the COW window closes (< 100ms). This avoids the realloc-during-COW race. - * - * CALLER PRE-ALLOCATION (P3 optimization for large deployments): For clusters - * with >10K entities, this calloc() can allocate 10–25 MB while the kvmgr WRITE - * lock is held, blocking all readers for the duration of the kernel TLB operation. - * To keep the write lock hold time at ~nanoseconds (pointer-swap only), a P3 - * improvement adds `xzarray_freeze_with_preallocated(obj, new_active, cow_caps)` — - * a NON-RTTI direct C function (NOT routed through vfreeze_fn_t, which has the - * single-argument signature `int (*)(meta_obj_t*)` and cannot accept extra params). - * The pre-allocation pattern: - * - * size_t cow_caps = a->caps + a->caps/2 + a->delta; // read caps without lock - * xzarray_item_t* new_active = calloc(cow_caps, sizeof(xzarray_item_t)); - * if (!new_active) return -ENOMEM; - * rwlock_wlock(kvmgr); - * // Direct call — bypasses RTTI dispatch intentionally: - * xzarray_freeze_with_preallocated(obj, new_active, cow_caps); - * rwlock_wunlock(kvmgr); - * - * If a->caps grew between the caps read and the lock acquisition, the function - * falls back to inline calloc() and frees the pre-allocated buffer. - * For P0, the inline calloc() path inside xzarray_freeze() is correct and - * acceptable; the pre-allocation optimization is deferred to P3. */ - size_t cow_caps = priv->shadow_caps + (priv->shadow_caps / 2) + a->delta; - a->data = calloc(cow_caps, sizeof(xzarray_item_t)); - if (a->data == NULL) { - /* Rollback: restore original data pointer from shadow. - * a->count still holds the pre-freeze value (not modified above), - * so no restoration needed for count. */ - a->data = priv->shadow; - a->pos = 0; - priv->shadow = NULL; - priv->shadow_caps = 0; - return -ENOMEM; - } - a->caps = cow_caps; /* active array is larger than shadow — update caps */ - a->count = 0; /* calloc zeroes the allocation; count reflects zero live entries */ - a->pos = priv->shadow_caps; /* start past shadow to prevent vstore index collisions */ - return 0; -} -``` - -This is a **pointer swap** — O(1), no data copied. The write lock is held only for the duration of the swap (~nanoseconds). The pre-grown active array (`cow_caps = 1.5 × shadow_caps + delta`) eliminates realloc during the COW window, avoiding the concurrent-vget dangling-pointer race. - -### 3.7 Thaw — End COW Phase with Merge and GC - -```c -/* xzarray_thaw — merge shadow back into active, apply optional GC. - * should_retain: callback to decide entity survival. NULL = keep all, - * no GC (used when aborting a checkpoint — merge shadow back unchanged). - * - * TWO-PASS DESIGN (required for correct GC): - * Pass 1: Merge all shadow entries into active[] (pointer copies only, - * no retention decisions, no frees). This ensures all entities - * are visible in active[] before any GC check runs. - * Pass 2: Run should_retain() on merged active[] entries, freeing those - * that fail retention. This is correct because job_has_dependents() - * can now see ALL entities (including ex-shadow ones) in active[]. - * - * Single-pass designs where retention checks happen during merge are - * INCORRECT: when should_retain(shadow[i]) calls job_has_dependents(), - * dependents at shadow[j] (j > i) have not yet been merged and are - * invisible in active[]. This causes premature GC of jobs with dependents. */ -static int xzarray_thaw(meta_obj_t* obj, retain_fn_t should_retain, void* arg) { - xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj); - xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj); - - /* Pass 1: Merge all shadow entries into active. No retention checks. */ - for (size_t i = 0; i < priv->shadow_caps; i++) { - if (!atomic_load_explicit(&priv->shadow[i].inuse, memory_order_relaxed)) continue; /* empty slot, skip */ - - if (i < a->caps && atomic_load_explicit(&a->data[i].inuse, memory_order_relaxed)) { - /* Active was written during COW — free shadow's stale object */ - if (a->killer) a->killer(priv->shadow[i].data); - } else { - /* Copy shadow pointer to active (no retention check yet) */ - xzarray_ensure_items(a, i + 1); - /* Use explicit field copies — cannot struct-copy xzarray_item_t - * because it contains _Atomic _Bool inuse (C11 prohibits plain - * assignment of atomic fields). Thaw runs under kvmgr WRITE lock, - * so relaxed ordering is sufficient. */ - a->data[i].data = priv->shadow[i].data; - atomic_store_explicit(&a->data[i].inuse, - atomic_load_explicit(&priv->shadow[i].inuse, memory_order_relaxed), - memory_order_relaxed); - a->count++; - } - } - - /* Free shadow — all live pointers are now in active[] */ - free(priv->shadow); - priv->shadow = NULL; - priv->shadow_caps = 0; - - /* Pass 2: Run retention/GC checks on the fully-merged active[]. - * All entities are now visible, so job_has_dependents() finds dependents - * regardless of their original position in the shadow array. */ - if (should_retain != NULL) { - for (size_t i = 0; i < (size_t)a->caps; i++) { - if (!atomic_load_explicit(&a->data[i].inuse, memory_order_relaxed)) continue; - if (!should_retain(arg, a->data[i].data)) { - if (a->killer) a->killer(a->data[i].data); - atomic_store_explicit(&a->data[i].inuse, false, memory_order_relaxed); - a->data[i].data = NULL; - a->count--; - /* Caller must also remove kvtab entry for this index */ - } - } - } - - /* Reset scan position for normal vstore operation. - * After merge, some indices < shadow_caps have inuse=0 (GC'd entities) - * and new entries at indices >= shadow_caps have inuse=1 (added during COW). - * Starting from 0 ensures vstore reuses freed slots instead of growing - * the array monotonically across checkpoints. */ - a->pos = 0; - return 0; -} -``` - -### 3.8 Checkpoint Flow - -``` -persist_checkpoint() fires on timer thread - │ - ▼ - For each kvmgr (jobs, workers, queues, resources, configs): - rwlock_wlock() - xzarray_freeze() ← O(1) pointer swap - record checkpoint_lsn - rwlock_wunlock() - │ - ▼ - Serialize all shadows → Cap'n Proto → disk - (no locks held, controller serving requests normally, - reads go through COW two-level lookup transparently) - - **CRITICAL: kvtab access during serialization**: The serialization phase - reads shadow data without locks (shadow is immutable after freeze), but - MUST NOT access the kvmgr hash table (kvtab) during this phase. The kvtab - is shared mutable state — concurrent vstore calls can insert new keys while - serialization is iterating the hash table, creating a data race on the - hash chain pointers. Instead, serialization enumerates shadow entries by - walking the shadow zarray indices [0..shadow_caps) directly, without - consulting kvtab. Each shadow slot at index i contains an inuse flag and - a data pointer; serialization reads shadow[i].inuse and shadow[i].data - for each slot — these are immutable (no lock needed). The kvtab key - lookup is only needed during the freeze phase (under write lock) to build - a snapshot of key→index mappings, which is stored alongside the shadow - array for serialization use: - - **Serialization index snapshot (built during freeze, used during serialize)**: - During `xzarray_freeze()`, after the pointer swap but before releasing the - write lock, the checkpoint code captures key→index mappings from kvtab for - each live entity. For each kvmgr, a local struct array is allocated: - - ```c - typedef struct { - int index; /* shadow slot index */ - char key[XCE_MAXNAME_LEN]; /* key string — deep-copied during capture (not a borrowed - * pointer into kvtab). kvtab can rehash and move string storage - * after the write lock is released; owned copies remain valid - * throughout the lock-free serialization phase. */ - size_t key_len; /* key length (includes null terminator) */ - } slot_key_t; - ``` - - `xzarray_freeze()` returns after the pointer swap. While still holding the - kvmgr write lock (before releasing it), the caller captures key→index - mappings by walking the kvtab and building a `slot_key_t[]` array: - - ```c - /* Caller-side step: after vfreeze(obj) returns, still under kvmgr write lock */ - vfreeze(obj); /* xzarray_freeze — O(1) pointer swap */ - slot_key_t* keys = NULL; - size_t n_keys = 0; - kvmgr_capture_keys(kvmgr, &keys, &n_keys); /* walks kvtab under write lock; - * deep-copies each key string into - * slot_key_t.key via strncpy */ - /* Now release write lock. Serialization reads keys[] (no locks held). - * keys[] holds owned copies — safe against concurrent kvtab rehash. */ - ``` - - This approach avoids threading output parameters through the RTTI interface. - The `vfreeze_fn_t` signature remains `int (*vfreeze_fn_t)(meta_obj_t* obj)`. - Key capture is a separate caller-side function that walks the kvtab hash table - while the write lock is still held. The serialization phase reads the keys[] - array (no locks held) to map shadow slots to their keys in the snapshot. - After serialization completes and thaw runs, each `slot_key_t` in the array - holds an owned copy — no separate per-key free is needed. `free(keys)` releases - the array. - │ - ▼ - After serialization confirmed on disk: - For each kvmgr: - rwlock_wlock() - xzarray_thaw(should_retain): - - Merge shadow → active (pointer copies for untouched entries) - - Apply retention policy → free expired/hidden entities (GC) - - Free shadow - kvtab_gc(): ← remove hash entries for GC'd indices - rwlock_wunlock() - - **CRITICAL: kvtab_gc must run inside the same write-lock section as thaw.** - `kvtab_gc_keys()` MUST be called before releasing the kvmgr write lock. Freed - zarray indices (where `inuse` was set to 0 by thaw's GC pass) become - available for reuse by the next `vstore()` call. If `kvtab_gc_keys()` runs - after the write lock is released, a concurrent `vstore()` (holding only a - READ lock) can allocate the freed index for a new entity before `kvtab_gc` - removes the stale key→index mapping, causing the old key to resolve to the - new entity's data — a hash→zarray inconsistency. The checkpoint flow above - (freeze → serialize → thaw → kvtab_gc → unlock) is correct because all four - operations run under the same write lock. - │ - ▼ - Delete WAL segments covered by this checkpoint -``` - -**Cross-kvmgr freeze ordering and referential consistency**: The checkpoint flow freezes kvmgrs sequentially (jobs → workers → queues → resources → configs), releasing each kvmgr's write lock after its freeze. This means the shadow snapshots for different kvmgrs capture state at slightly different LSN points. A job serialized in the jobmgr snapshot may reference a worker (via `exec_host`) that was added after the workermgr snapshot was taken. This is acceptable because WAL replay after snapshot loading corrects all inconsistencies — all mutations between the earliest and latest freeze points are captured in the WAL entries that follow the snapshot. On recovery, `persist_recover()` loads the snapshot and then replays WAL entries in strict LSN order, which restores full consistency. The checkpoint_lsn recorded in `persist_t` (used for WAL trimming) is set to the **minimum** LSN across all kvmgrs at the time of each freeze. Using the minimum ensures WAL entries between the earliest and latest kvmgr freeze points are preserved — entries covering LSN [min_freeze..max_freeze] are needed to bring kvmgrs frozen at earlier LSNs up to date during recovery replay. Using the maximum would trim precisely those bridging WAL entries and corrupt the cross-kvmgr recovery. - -**Capture order (CRITICAL)**: `checkpoint_lsn` must be set **after all kvmgr freezes complete**, not incrementally during the freeze loop. Record each kvmgr's freeze-time LSN into a local array (`uint64_t freeze_lsns[N_KVMGRS]`), then `checkpoint_lsn = min(freeze_lsns[0..N_KVMGRS-1])` in one assignment after the loop. If `checkpoint_lsn` is updated inside the loop (e.g., `checkpoint_lsn = min(checkpoint_lsn, current_lsn)` at each step), the intermediate values are visible to WAL-trim decisions before all freezes complete — a racing checkpoint-triggered trim could delete WAL entries that a later freeze still needs. -``` - -**Non-kvmgr zarrays**: `cluster_t` also contains standalone `zarray_t` fields (endpoints, configs, sharedinst) that are not managed by kvmgr. These are handled differently during checkpoint: -- `c->endpoints`: Not serialized in the snapshot. Endpoints are transient network connections that are rebuilt on recovery (workers/schedulers reconnect after restart). The deferred endpoint cleanup mechanism (Part 4.9) handles cleanup during normal operation, not checkpoint. -- `c->configs`: Serialized under a separate write lock (`lock(c->configmgr, RTTI_LOCK_READ)` for reads, `RTTI_LOCK_WRITE` for freeze/thaw). Configs are read-only during normal operation and rare writes (admin config changes). Config freezes are serialized as a separate Cap'n Proto array in the snapshot. -- `c->sharedinst`: Not serialized. Shared instances are reconstructed from config on recovery. - -**Shadow lifetime and shutdown safety**: `xzarray_thaw()` calls `free(a->shadow)` and sets `a->shadow = NULL`. The serialization thread reads shadow without any lock (shadow is "immutable after freeze"). This is safe **only if thaw never runs while the serialization thread is still reading**. The normal checkpoint flow guarantees this: `persist_checkpoint_timer_fn()` (running on the delaymgr timer thread) does the freeze and key-capture under the write lock, then **spawns a dedicated serialization pthread** for the slow I/O (Cap'n Proto serialization + fsync). Thaw is called by that same serialization pthread after the snapshot is confirmed on disk. The delaymgr timer thread returns immediately after spawning, keeping the delaymgr responsive. The serialization pthread ID is stored in `persist_t.snap_thread` so that `cluster_fini()` can join it. `pthread_cond_timedwait(&persist.checkpoint_done, ...)` in `cluster_fini()` is the normal-path wait; `pthread_join(persist.snap_thread, NULL)` is the deadline-exceeded path that ensures no use-after-free when the checkpoint is cancelled but the serialization thread is still running. **The distinction matters**: "same checkpoint timer thread" is intentionally avoided — the timer thread is shared across all delaymgr tasks and must not block on I/O; only the dedicated serialization pthread can block. However, the **shutdown path** must also observe this constraint: if `cluster_fini()` or a shutdown signal fires while a checkpoint is in progress, it must wait for the serialization thread to complete before calling `xzarray_thaw()`. Implement with a `persist_t.checkpoint_in_progress` atomic flag **and** a `pthread_cond_t checkpoint_done` condition variable. **The condition variable must use a dedicated `pthread_mutex_t checkpoint_mutex` — NOT `persist.lock` (which is a `mylock_t`).** `pthread_cond_timedwait()` requires a `pthread_mutex_t*`. `mylock_t` is typedef `pthread_mutex_t` (confirmed by `mylock.h`), so the type concern does not apply. The real reason for using a dedicated `checkpoint_mutex` is deadlock avoidance: `persist.lock` is held by vcpu threads during `persist_wal_sync_append()` (which may block on `repl_wait_ack()`), so `cluster_fini()` attempting to acquire `persist.lock` for the timed wait would deadlock against a vcpu thread holding it. The dedicated `checkpoint_mutex` avoids this: it is never held during replication waits. On shutdown: `cluster_fini()` acquires `checkpoint_mutex`, checks `checkpoint_in_progress`. If set, it calls `pthread_cond_timedwait(&persist.checkpoint_done, &checkpoint_mutex, &deadline)` where `deadline` is `now + 5 seconds` (configurable). The checkpoint thread signals `checkpoint_done` after completing serialization and thaw (or after canceling). If the deadline expires, `cluster_fini()` cancels the in-progress checkpoint (sets a cancel flag) and skips thaw — incomplete checkpoint state is cleaned by the next restart's `persist_recover()`. This avoids busy-waiting on the atomic flag and provides bounded shutdown latency. **IMPORTANT**: after the 5-second deadline expires and the cancel flag is set, `cluster_fini()` must `pthread_join()` the checkpoint/serialization thread before proceeding to free any cluster state. The serialization thread reads shadow arrays and `slot_key_t` arrays that hold pointers to kvtab keys — if `cluster_fini()` proceeds to free the kvmgr and its contents before the serialization thread exits, a use-after-free occurs. The join ensures the serialization thread observes the cancel flag, exits its loop, and returns before any cluster state is destroyed. - -**Crash safety during checkpoint**: If the controller crashes mid-checkpoint (during shadow serialization or mid-thaw), the previous snapshot on disk remains valid. On recovery, `persist_recover()` loads the last valid snapshot and replays WAL entries from that snapshot's LSN forward — the incomplete checkpoint is simply ignored. WAL segments are only deleted after the new snapshot is fully written and `fsync`'d to disk. This ensures atomicity: a checkpoint either completes fully (new snapshot + WAL trim) or is treated as if it never happened. - -**Mid-thaw crash and COW in-memory state**: If the controller crashes mid-thaw (some shadow entries merged back to `active[]`, some not yet merged, `priv->shadow != NULL`), the COW in-memory state is lost on restart — it was never persisted. `persist_recover()` starts from the last completed snapshot and replays WAL entries from that point. Mutations that were applied to `active[]` during the COW phase but **not yet written to WAL** (e.g., a `vput` that occurred after the freeze but before the WAL write via the hook) are also lost. This is consistent with the WAL durability model: only mutations that have been written to WAL and fsynced are durable. The COW mechanism is a transient optimization for non-blocking checkpoint; it provides no additional durability guarantee beyond the WAL. Implementers must NOT attempt to recover COW state from WAL — on restart, `xzarray_freeze()` runs anew from the fully recovered state. - -**Lock hold times:** -- Freeze: ~nanoseconds (pointer swap) -- Thaw + GC: O(shadow_caps) pointer copies + retention checks. For 10K entries, ~50-100 microseconds. Note: if `should_retain` calls `job_has_dependents()` for job entities, the retention check becomes O(n_terminal_jobs × n_total_jobs) in the worst case because `job_has_dependents()` scans all jobs per call. At XCE's current scale (typically <1K finished jobs in a retention window), this is fast. For deployments with >10K jobs, consider building a reverse dependency index to make `job_has_dependents()` O(k) where k is the number of dependents per job, reducing GC to O(shadow_caps + n_terminal_jobs × k). - -### 3.9 Lock Hierarchy for COW - -XCE uses a two-lock hierarchy in kvmgr: - -``` -kvmgr rwlock (outer) — protects kvtab + serializes freeze/thaw - └─ xzarray mutex (inner) — protects individual vstore/vget/vput/vclear operations -``` - -- **kvmgr `my_rwlock_t`**: The outer lock. Taken as read-lock for normal lookups (`xkvmgr_find`). Taken as write-lock for additions (`xkvmgr_add`) and for COW freeze/thaw coordination. -- **xzarray `mylock_t`**: The inner lock. Two modes: - - **Normal mode**: NOT taken inside `xzarray_put_vec()` — callers hold the kvmgr WRITE lock for mutations, which provides serialization; the inner mutex adds no value and is skipped. - - **COW mode**: Taken inside `xzarray_store()` for the entire find-slot-and-write sequence (R-22 fix). During COW, multiple vcpu threads hold the kvmgr READ lock concurrently, so the outer lock does not serialize concurrent `vstore()` calls. The inner mutex is the only serialization point preventing two threads from claiming the same free slot. `vput()` in COW mode is still NOT protected by the inner mutex — `vput()` writes to a known index (never allocates a new slot), so there is no find-slot race. - - **Thaw mode**: NOT taken during thaw — the kvmgr WRITE lock is held for the entire thaw+GC pass, which serializes all access. - -**COW lock protocol:** - -``` -freeze: - rwlock_wlock(kvmgr) ← serialize: no concurrent lookups during swap - xzarray_freeze(data) ← O(1) pointer swap, no xzarray mutex needed - rwlock_wunlock(kvmgr) - -serialize shadows: - (no locks held — shadows are immutable) - -thaw + GC: - rwlock_wlock(kvmgr) ← serialize: block lookups during merge - xzarray_thaw(data, retain) ← merges shadow→active, frees GC'd entities - (kvmgr write lock held; inner mutex not needed) - kvtab_gc_keys(kvmgr) ← prune stale hash entries - rwlock_wunlock(kvmgr) -``` - -**Per-object locks are NOT nested inside kvmgr lock.** The kvmgr lock serializes container-level operations (key→index lookup, COW coordination). Per-object locks (`job_t.lock`, `worker_t.lock`) are taken after releasing the kvmgr lock — the lookup returns a referenced pointer, and the caller locks the object separately. - -**GC use-after-free mitigation — reference-counted object lifecycle (CRITICAL)**: During thaw+GC, the kvmgr WRITE lock is held, which prevents new kvtab lookups. However, a vcpu thread that previously completed a kvtab lookup (under READ lock) and obtained a pointer to a `job_t` may still be accessing that job object after releasing the READ lock. If thaw+GC frees that job during the window between the vcpu thread's READ unlock and its next access, the vcpu thread dereferences freed memory — a use-after-free. - -This window exists because XCE's lock protocol is: (1) acquire READ lock, (2) kvtab lookup → get pointer, (3) release READ lock, (4) use pointer to access the object. Between steps 3 and 4, the checkpoint thread can acquire the WRITE lock, run thaw+GC, and free the object. - -**Mitigation — epoch-based reclamation with generation counter**: The kvmgr gains a `uint64_t epoch` counter, incremented atomically at the start of `xzarray_thaw()` (inside the WRITE lock, before any GC frees). Vcpu threads record `kvmgr.epoch` at lookup time (step 1). After each per-object operation, the vcpu thread checks whether `kvmgr.epoch` has changed. If it has, the pointer may be stale and the vcpu must re-lookup the object. This is a lightweight generation counter — no per-object reference counting is needed, just an atomic counter check after each kvmgr lookup. - -Alternative (simpler but higher overhead): vcpu threads hold the kvmgr READ lock for the entire duration of their per-object operation (lookup → use → release). This prevents GC from acquiring the WRITE lock until all READ holders are done. The downside is longer READ lock hold times, which delays checkpoint and increases the checkpoint "freeze" pause. At XCE's scale (10K jobs, ~100μs per job operation), holding the READ lock across a per-job operation is acceptable — the checkpoint thread waits at most the duration of one per-job operation for the WRITE lock. - -**Recommended approach**: Use the simpler per-operation READ lock hold for P0, with the epoch optimization as a P3 performance improvement if checkpoints cause detectable latency spikes. The P0 implementation holds the kvmgr READ lock across the entire per-object operation (lookup through use), preventing GC from freeing objects that any vcpu thread is actively accessing. This is safe because per-object operations (job status update, worker state change) are fast (<100μs), and the WRITE lock holder (checkpoint) only needs to wait for the brief overlap. - -**Why there is no lock race during serialization + thaw**: The concern is that concurrent `vput` calls during the "no locks held" serialization phase might interleave with `xzarray_thaw()`. This is safe because: -1. During serialization (no locks): `vput` writes to `active[]` under the kvmgr READ lock. These writes are to `active[i].data` — they never touch `shadow[]`. -2. During thaw (WRITE lock held): `xzarray_thaw()` merges `shadow[]` → `active[]`. The write lock blocks all concurrent `vput` (read-lock) calls. No interleave is possible. -3. Thaw's merge logic: for each `shadow[i]`, if `active[i].inuse == 1` (set by a concurrent vput during serialization), thaw takes the "active wins" branch and frees the stale shadow pointer. This is the intended behavior — active is newer. - -This matches the existing pattern in `handle_jobsubmit()` (`job.c:455-525`): `lock(c->jobmgr, RTTI_LOCK_READ)` for lookup, release, then per-object lock for modification. - -### 3.10 Checkpoint Timer - -The checkpoint thread uses the existing `delaymgr` timer system. **`delaymgr` will be enhanced to change `sleep(precision)` to `nanosleep(precision_ms)` in `delay.c`**, giving sub-second granularity. **The `delay_task()` delay parameter remains in seconds throughout** — only the underlying sleep precision changes. The `delay.c` `ts_nextrun = now + delay` arithmetic is unchanged (seconds-based). All existing callers continue to pass seconds; the precision reduction only affects how tightly the timer fires relative to its scheduled time. - -```c -/* In delay.c — change sleep(precision) to nanosleep with precision_ms granularity. - * The ts_nextrun / run_period fields remain in seconds (time_t). No caller changes - * are required. Default precision_ms for the cluster delaymgr: 200ms, set via: - * delaymgr_init(&c->delaymgr, 200 /* precision_ms */, run_delay_tasks) - * - * Existing callers at cluster.c:756 (heartbeat) continue passing seconds. */ - -/* In persist_start() — register periodic checkpoint timer (1-second poll) */ -int ckpt_timer = delay_task(&(c->delaymgr), - 1, /* 1-second poll interval (seconds) */ - -1, /* recurring */ - (delay_task_entry_fn_t)persist_checkpoint_timer_fn, - (void*)c); -/* The periodic checkpoint (snap_interval_s, e.g. 60s) is counted inside - * persist_checkpoint_timer_fn() via a tick counter, not as a separate delay_task. */ -``` - -**Checkpoint overlap handling**: If a timer-driven checkpoint fires while a previous checkpoint is still in progress (checked via `persist_t.checkpoint_in_progress` atomic flag), the timer callback logs an `INFO` message ("checkpoint already in progress, skipping") and returns. The size-based inline trigger (Part 3.11) provides a safety valve for rapid WAL growth. - -**Checkpoint timer for inline-triggered checkpoints**: The timer callback checks `persist_t.checkpoint_requested` on each 1-second tick, responding within ~1 second of the flag being set. - -Benefits: -- No raw `pthread_create`/`pthread_join` management for the timer thread -- `delaymgr` already handles timer cancellation in `cluster_fini()` -- Consistent with how all periodic tasks are managed -- No semantic change to delay_task() callers — all pass seconds as before - -**P0-3 implementation note**: The delaymgr `nanosleep()` change (`delay.c`) is a prerequisite for P0-3. The change is: rename `precision` field to `precision_ms` and replace `sleep(precision)` with `struct timespec ts = {0, precision_ms * 1000000L}; nanosleep(&ts, NULL);`. Update `delaymgr_init()` call in `cluster.c:604` from `precision=2` to `precision_ms=200`. Audit all `delaymgr_init()` callers to confirm they all pass precision_ms in milliseconds. - -### 3.11 Snapshot File Format - -Serialized from shadow arrays while no locks are held: - -``` -┌─────────────────────────┐ -│ SnapshotHeader (Capnp) │ -│ - lsn: last WAL LSN │ -│ - timestamp │ -│ - term │ -│ - version: UInt32 (=1) │ -│ - nJobs, nQueues, nWorkers │ -│ - checksum: CRC32(body) │ -│ - headerChecksum: CRC32(header fields) │ -├─────────────────────────┤ -│ Job array (Capnp) │ -├─────────────────────────┤ -│ Queue array (Capnp) │ -├─────────────────────────┤ -│ Worker array (Capnp) │ -├─────────────────────────┤ -│ Resource array (Capnp) │ -├─────────────────────────┤ -│ Config array (Capnp) │ -└─────────────────────────┘ -``` - -**Snapshot CRC scope**: The `SnapshotHeader.checksum` (CRC32) covers the Cap'n Proto–serialized body (Job array, Queue array, Worker array, Resource array, Config array) **excluding** the `SnapshotHeader` struct itself. The header is verified separately via `SnapshotHeader.headerChecksum`. After `headerChecksum` passes, `nJobs`, `nQueues`, `nWorkers` are trusted as authoritative counts — a secondary validation against actual deserialized array lengths is a **debug assertion only** (`assert(nJobs_from_header == actual_cap'nproto_list_count)`), not a required safety check. Treating the counts as untrusted after a passed headerChecksum would create a contradictory two-truth-source design. This split design avoids the chicken-and-egg problem where the CRC needed for body verification is itself part of the data being verified. - -- Time-based: every `snap_interval_s` (default 60s) -- Size-based: when WAL exceeds `snap_wal_size_mb` (default 64 MB) -- Shutdown: final checkpoint in `cluster_run()` before exiting the main loop - -**Inline size check in WAL append path**: The timer-driven size check alone creates a window where WAL can grow unbounded between timer ticks (e.g., 10K submits/sec × ~200 bytes = 2MB/sec, filling 64MB in ~32s, which may fall between two 60s timer ticks). Both `persist_wal_async()` and `persist_wal_sync_append()` must check cumulative WAL size after writing and trigger an inline checkpoint if `current_segment_bytes > snap_wal_size_mb`. - -**Inline checkpoint mechanism — atomic flag, not direct freeze**: The calling thread (which holds the kvmgr READ lock from the hook call chain) **cannot** directly call `persist_checkpoint()`, because `persist_checkpoint()` acquires the kvmgr WRITE lock — and a thread holding a READ lock that attempts to acquire a WRITE lock on the same POSIX rwlock results in undefined behavior (deadlock per POSIX). Instead, the inline trigger sets an atomic flag: `atomic_store(&persist->checkpoint_requested, 1)`. The checkpoint timer thread (running via `delaymgr`) checks this flag on each wake (every `snap_interval_s`, or sooner if the flag is set — use a shorter poll interval like 1-2 seconds when the flag is set). When `checkpoint_requested` is observed, the timer thread calls `persist_checkpoint()` (it holds no kvmgr lock, so acquiring the WRITE lock is safe). After the checkpoint completes, the timer thread clears the flag. This approach: (1) avoids the rwlock self-deadlock, (2) ensures checkpoint always runs on a thread that does not hold the kvmgr READ lock, and (3) bounds the WAL growth to `snap_wal_size_mb + max_wal_per_poll_interval` (at 2-second poll, ~4MB additional WAL at 2MB/sec). If an inline checkpoint is already in progress (checked via `persist_t.checkpoint_in_progress` atomic flag), the WAL append skips setting `checkpoint_requested` — the in-progress checkpoint will trim the WAL when it completes. - -**WAL growth during an in-progress checkpoint**: When `checkpoint_in_progress` is set, the inline trigger is suppressed. WAL growth during this window is bounded by the checkpoint duration times the submission rate. For pathological cases (very large shadow state, slow serialization disk), the WAL can grow significantly. **Mitigation**: if cumulative WAL size exceeds `2 × snap_wal_size_mb` while `checkpoint_in_progress` is set, log a `WARN("WAL size %.1f MB exceeds 2x threshold during in-progress checkpoint")`. If WAL size exceeds `wal_max_size_mb` (default 2 GB), trigger an emergency checkpoint even if one is in progress — the existing checkpoint must be completing soon (serialization is lock-free, so a stuck checkpoint indicates an I/O problem). Operators should size `snap_wal_size_mb` based on expected peak throughput and worst-case checkpoint duration. A rule of thumb: `snap_wal_size_mb ≥ peak_submit_rate_KB/s × worst_case_checkpoint_seconds / 1024`. +> **SUPERSEDED.** This section (originally: a copy-on-write dual-zarray checkpoint mechanism with freeze/thaw semantics and a two-level vget/vput/vstore lookup) described a design that was later replaced. See: `src/docs/persistence-and-ha-design.md` findings and the closed spec `ha-entity-snapshots` in `.kiro/specs/` for what shipped instead — the actual checkpoint path (`persist_checkpoint()` / `write_snapshot()` in `src/controller/persist.c`) collects entities directly via `collect_job_frames()` / `collect_queue_frames()` / `collect_host_frames()` and writes Cap'n Proto–encoded frames; `zarray.h` has no freeze/thaw/COW/shadow API. --- @@ -1299,428 +622,7 @@ The new additions needed: ## Part 5: HA Failover — LSF-Style Priority Election -### 5.1 Design Philosophy - -Instead of Raft consensus (~5000 LOC), use LSF's simple priority-based model: - -- A fixed list of controller candidates is defined in cluster config, ordered by priority -- The first (highest-priority) candidate that is alive becomes primary -- "Am I the highest-priority alive candidate?" is the entire election algorithm -- **Split-brain risk**: Under a network partition, a standby may promote itself while the primary is still alive but unreachable. This is the same trade-off LSF makes — priority-based election cannot prevent split-brain without quorum. XCE accepts this risk with the following assumptions and mitigations: (1) controllers share a dedicated management network in the same LAN with low partition probability — primary and standby controllers are expected to be co-located in the same data center, so network partitions between them are abnormal events; (2) term-based fencing prevents stale standbys from accepting new writes; (3) workers always reconnect to the current primary after partition heals. If split-brain is unacceptable, a quorum-based protocol (Raft) would be required instead. - -**Split-brain impact analysis**: Under split-brain, the original primary continues serving its established API server connections. Since end users never connect directly to the controller — only through the API server — user requests continue flowing to the original primary. The promoted standby also accepts work if an API server connects to it, but in practice the API server maintains a long-lived TCP connection to the original primary and does not switch during a partition (this is confirmed by the current `RegisterToBackend()` implementation in `clib.go`, which uses `nmasters = 1` with a single long-lived connection and no reconnection logic). **However, if the partition drops the API server's TCP connection to the original primary, the API server has no reconnection path until P2.5-7 (libxce discovery for API server) is implemented.** After P2.5-7, the API server would use `xce_cluster_config_t` to reconnect to any candidate, which could redirect it to the promoted standby — in that case, user requests would flow to the promoted standby and writes would diverge from the original primary's WAL. - -**Worker movement during split-brain**: If the partition also severs the TCP connection between some workers and the original primary, those workers will reconnect (via libxce discovery) to the promoted standby. These workers then report their job status to the standby, which applies the updates and writes them to its local WAL. When the partition heals, the standby (lower term) must step down, and these workers must reconnect to the original primary. The critical reconciliation step is: **after partition heals, the restored primary must receive state updates from workers that were served by the standby during the partition.** This is self-healing — workers always re-notify their current job state on reconnection (existing behavior: `handle_worker_join` re-registers the worker, and the scheduler re-queries job state). No explicit state merge protocol is needed beyond the existing worker re-join flow, because workers are the source of truth for their own runtime state. - -The actual risk is **replication divergence and potential data loss**: the standby's WAL diverges from the primary's WAL during the partition. **Writes accepted by the promoting standby during a network partition are at risk of loss** — when the partition heals, term-based fencing ensures the lower-term controller steps down, and the diverged WAL entries from the standby are discarded during full re-sync from the primary (which has the authoritative higher-term WAL). This means any job submits ACKed by the standby during the partition are not durable. At XCE's scale (a few thousand entities), full re-sync is bounded by the partition duration and is handled by full re-sync rather than partial merge. If preserving standby-served writes during partitions is required, a quorum-based protocol (Raft) would be needed instead. - -**Deployment assumption: LAN-only**. XCE HA controllers all run on the same LAN (dedicated management network). The following table enumerates the specific failure modes that can cause LAN partitions, their expected recovery times, and the blast radius for split-brain data loss: - -| Failure Mode | Expected Duration | Blast Radius | Mitigation | -|--------------|-------------------|--------------|------------| -| Top-of-rack switch failure | 5–30 min (auto-failover or manual swap) | All controllers on that switch are isolated | Deploy controllers across separate switches/racks | -| Switch port failure (single) | 1–10 min (STP reconvergence) | One controller isolated; remaining 2 have quorum connectivity | Redundant uplinks per controller | -| Cable cut (copper/fiber) | 30 min – 4 hours (physical replacement) | Controllers on opposite sides of the cut | Diverse cable paths between controllers | -| NIC failure (single controller) | 5–15 min (operator restarts on backup NIC) | One controller down; no partition if remaining 2 can communicate | Dual-NIC bonding (active-passive) | -| Misconfigured VLAN or ACL | 10 min – 2 hours (operator detects and corrects) | Intermittent or partial isolation | Change management controls on switch config | -| Kernel NIC driver bug (hang) | 5–30 min (operator detect + reboot) | One controller appears dead but process is running | Health monitoring with NIC-level liveness checks | -| STP broadcast storm | 1–10 min (storm suppression) | All controllers may see intermittent connectivity | Storm control on management switches | - -**Split-brain window duration**: The window opens when the standby's heartbeat timer fires (default `heartbeat_timeout_ms = 10000ms`) and closes when the partition heals and the higher-term message reaches the old primary. **Data at risk**: all job submits ACKed by the standby during this window are discarded on partition healing. At typical submit rates (1–10 submits/second), the worst case for a 10-minute partition is ~600–6000 job submissions at risk of loss. - -**Quantitative risk model**: At XCE's expected submit rate of 1–10 jobs/second and a 99.9% LAN availability (8.76 hours downtime/year), the expected annual data loss is: `partition_probability × mean_partition_duration × submit_rate`. For a well-managed LAN: P(partition per year) ≈ 0.001 (conservative), mean duration ≈ 15 minutes, submit rate ≈ 5/sec → expected lost submits/year ≈ 4.5. Operators must evaluate whether this risk is acceptable for their workload. - -**Three-controller asymmetric partition scenario**: With 3 controllers (ctrl-1 primary, ctrl-2 standby P=1, ctrl-3 standby P=2), consider an asymmetric partition where ctrl-1 can reach ctrl-2 but not ctrl-3, and ctrl-2 can reach ctrl-1 but not ctrl-3. ctrl-3 detects heartbeat timeout from ctrl-1 and probes candidates with priority < 2. If ctrl-3 also cannot reach ctrl-2, it receives no ACK within `election_timeout_ms` and promotes itself to primary — creating two primaries (ctrl-1 and ctrl-3). This is the same split-brain risk already acknowledged in Part 5.1, with the same mitigation: term-based fencing resolves the conflict when the partition heals. The lower-term primary (determined by which controller incremented term first) steps down. In this specific scenario, ctrl-3 would have a lower term than ctrl-1 (assuming ctrl-1 was already primary before the partition), so ctrl-3 steps down when connectivity resumes. If the partition also allows an API server to connect to ctrl-3, any jobs submitted to ctrl-3 during the partition are at risk of loss (same as the two-controller case). This scenario illustrates why the `split_brain_risk_accepted: true` config key is mandatory — even single-LAN deployments can produce asymmetric partitions during switch failures or cable cuts (see the failure mode table above). - -This is fundamentally different from WAN or cross-DC deployments where partitions can persist indefinitely due to routing protocol convergence, BGP flap dampening, or ISP outages. For cross-site or multi-data-center HA requirements, the recommended approach is to deploy **separate XCE clusters** per site with cross-cluster job forwarding (out of scope), not a single replicated HA cluster. - -**Deployment requirements**: LSF-style priority election is appropriate under the following conditions: (1) controllers share a **dedicated management network** on the same LAN — cross-DC or WAN deployments have unacceptably high partition probability; (2) **≤3 controller candidates** — with more candidates, cascading promotions during instability become harder to reason about; (3) operators understand that job submits ACKed by a split-brain standby are **not durable** and are discarded on partition healing. If any of these conditions cannot be met, a quorum-based protocol (Raft) is required instead. - -**Required operator acknowledgment**: To prevent accidental deployment without understanding the split-brain risk, HA mode requires an explicit opt-in config key: - -```yaml -ha: - split_brain_risk_accepted: true # REQUIRED — HA refuses to start without this - candidates: [...] -``` - -`cluster_init()` must refuse to start HA mode if `split_brain_risk_accepted != true`, logging: `"HA split-brain risk has not been acknowledged — set ha.split_brain_risk_accepted: true in cluster.bin (via config tool from cluster.yaml) after reading the HA deployment requirements"`. - -**Operator visibility for degraded mode**: When `sync_timeout_ms` expires and the primary falls back to local-only ACK, this is NOT silent: -1. Log at `ERROR` level (not `WARN`) — degraded replication is a durability-affecting event -2. Increment `ctrl_repl_degraded_count` metric (alongside `ctrl_split_brain_detected`) -3. **Early split-brain signal (state-change log, not per-submit)**: When the primary first enters the combined state of (degraded replication AND all heartbeat sends failing), log **once** at `ERROR`: `"Primary entering isolated mode — cannot reach any standby for sync ACK or heartbeat. Split-brain risk: if a standby has promoted itself, submitted jobs may be lost. ctrl_isolated_primary_count=%u"`. Increment `ctrl_isolated_primary_count` on state entry. When the primary reconnects to any standby, log **once** at `INFO`: `"Primary exiting isolated mode after %us — standby reconnected"`. Do **not** emit this log on every job submit — sustained degradation at 10 submits/sec produces 600 identical ERROR entries per minute, flooding logs and suppressing the signal. Track isolation state with a `_Atomic int ha_isolated;` flag in `ha_state_t`; set on entry, clear on reconnect. Also add `primary_isolated_since` (`struct timespec`, CLOCK_MONOTONIC) to `ha_state_t` to report isolation duration on reconnect. -3. The degraded state persists until the standby reconnects — subsequent submits in degraded mode each log at `ERROR` and increment the counter - -**Split-brain detection condition**: `ctrl_split_brain_detected` is incremented when a controller receives a message (heartbeat, replication, election probe, or client request) from a peer that believes a DIFFERENT controller is primary. Specifically: -- A standby receives `OP_heartbeat` from a controller whose term is LOWER than the standby's current term (the sender is a stale primary that hasn't learned of the promotion) -- A primary receives a message (election probe, `OP_replJoin`) from a controller whose term is HIGHER than its own (a higher-priority candidate has promoted itself) -- A primary receives `OP_heartbeat` from another controller that claims to be primary with a term EQUAL to or HIGHER than the receiver's term - -This detection is passive — it requires network communication, not active probing. During an actual network partition where no communication is possible, split-brain cannot be detected (this is the fundamental limitation of LSF-style election without quorum). The metric alerts the operator once communication is restored and the controller can observe the divergence. - -**Trade-off**: Non-preemptive — if a higher-priority candidate recovers, it does NOT preempt the current primary. This avoids disruptive failback. Admin can trigger manual failover. - -### 5.2 Candidate Configuration - -The `MasterCandidate` struct already exists in `controller.capnp` with `name`, `host`, `port`. Priority is **list position** (index 0 = highest priority). - -```yaml -# cluster.yaml — human-editable config, processed by config tool -controller: - candidates: - - name: ctrl-1 - host: 10.0.1.1 - port: 7000 - - name: ctrl-2 - host: 10.0.1.2 - port: 7000 - - name: ctrl-3 - host: 10.0.1.3 - port: 7000 -``` - -**Configuration pipeline**: The controller does **not** parse YAML directly. A separate configuration tool (CLI, out of scope for this design) validates `cluster.yaml` and produces a Cap'n Proto-encoded binary configuration file that the controller reads at startup. The validation tool checks: -- At least one candidate is defined -- No duplicate host:port pairs -- `ha.split_brain_risk_accepted: true` is set when HA mode is enabled -- `ha.split_brain_risk_accepted` is `true` when more than one candidate is configured - -The controller reads only the binary Cap'n Proto config file — it has no YAML parsing dependency, no text config parsing, and treats its config as a trusted, validated blob. This eliminates config injection risks and simplifies the controller's dependency tree. - -### 5.3 Role Management - -```c -typedef enum { CTRL_PRIMARY = 0, CTRL_STANDBY = 1 } ctrl_role_t; - -typedef struct { - _Atomic uint64_t term; /* C11 atomic election term (fencing token) — see Part 5.7 */ - _Atomic int32_t role; /* C11 atomic — use atomic_load/store_explicit, see Part 5.7 */ - int my_priority; /* My index in candidates[] */ - replicator_t repl; /* Replicate to standbys (primary only) */ - _Atomic uint64_t applied_lsn; /* Last applied WAL entry (standby only). - * Must be _Atomic because it is written by the iomgr/vtask - * thread during OP_replEntry processing and read by the - * watchdog thread when constructing OP_replJoin for catch-up. - * Use atomic_store_explicit/load_explicit with release/acquire. */ - endpoint_t* primary_ep; /* Connection to current primary via iomgr SSL pipeline (standby only). - * LIFECYCLE: set by iomgr accept/connect — iomgr owns the endpoint - * and the ha_state_t struct only holds a borrowed reference. - * On connection loss, iomgr sets primary_ep = NULL _before_ freeing - * the endpoint. All access must check for NULL; no dereference - * after NULL. The standby's catch-up thread reads primary_ep under - * the ha_state_t mutex (ha->lock), which serializes it against - * iomgr's NULL-then-free sequence. The watchdog thread also - * checks primary_ep == NULL under ha->lock before heartbeat - * comparison. This borrow-with-mutex pattern ensures no use-after-free: - * iomgr holds ha->lock while clearing the pointer, so any - * concurrent dereference sees either a valid endpoint or NULL. */ - char primary_host[XCE_MAXNAME_LEN]; - uint32_t primary_port; - struct timespec last_heartbeat; /* Last heartbeat from primary (CLOCK_MONOTONIC, standby) */ - _Atomic int promotion_failed; /* Non-zero if last promotion attempt failed — retry on next watchdog tick. - * Must be _Atomic: written by ha_promote_to_primary() on the watchdog - * thread, read by ha_check_heartbeat() on the same thread but also - * visible to vcpu threads checking promotion status. Relaxed ordering. */ - _Atomic int promotion_retry_count; /* Number of consecutive failed promotion attempts. - * Must be _Atomic because it is incremented by the - * watchdog thread and read by vcpu threads during - * promotion status checks. Use atomic_load/store - * with relaxed ordering. */ - _Atomic int in_catch_up; /* C11 atomic — non-zero while standby is processing - * historical catch-up entries. Must be _Atomic because - * it is written by the iomgr/vtask thread (on - * OP_replJoin receipt and OP_replCatchUpDone) and - * read by vcpu threads during ACK suppression - * (Part 6.4). Use atomic_store/load_explicit with - * release/acquire ordering. - * - * LIFECYCLE (standby-side, this field): - * Set to 1 by the standby when it sends OP_replJoin - * with applied_lsn < primary's committed_lsn. - * Cleared to 0 by the standby when it sends - * OP_replCatchUpDone (after applying snapshot + all - * bridging WAL entries up to committed_lsn). - * The standby initiates completion — it decides when - * its applied_lsn has reached committed_lsn. - * - * PRIMARY-SIDE MIRROR: standby_conn_t.in_catch_up - * (Appendix A.3) is cleared by the primary when it - * receives OP_replCatchUpDone from this standby. - * See Part 6.6 and P2-5 for the full protocol. - * - * Gates ACK suppression: during catch-up, the standby - * does NOT send OP_replAck for historical entries - * (see Part 6.4). */ - ha_config_t config; /* HA configuration (timeouts, intervals) */ - pthread_mutex_t lock; /* Protects primary_ep and last_heartbeat against concurrent - * iomgr NULL-then-free and watchdog reads. mylock_t is - * typedef pthread_mutex_t (confirmed by mylock.h), but an - * explicit pthread_mutex_t field documents the pthread_cond - * usage intent and avoids any future mylock_t divergence. - * Initialized in ha_init(). */ -} ha_state_t; - -typedef struct { - uint32_t heartbeat_interval_ms; /* Primary heartbeat interval (default: 3000). Minimum 2000ms — the HA watchdog pthread (P1-3) uses nanosleep() so there is no delaymgr truncation concern. The 2000ms floor is a practical recommendation: values below ~1000ms risk false timeout triggers on loaded management LANs (SSL handshake + scheduling jitter). Enforce heartbeat_interval_ms >= 2000 in ha_init() as a conservative safe floor; operators needing faster heartbeats may lower this with care. */ - uint32_t heartbeat_timeout_ms; /* Standby heartbeat timeout (default: 10000) */ - uint32_t election_timeout_ms; /* Shared deadline for all parallel election probes (default: 2000) */ - uint32_t probe_connect_timeout_ms; /* Per-probe TLS connect timeout for am_i_primary() probe threads - * (default: election_timeout_ms / MAX_HA_CANDIDATES, minimum 500ms). - * Each probe thread uses this as the poll() timeout in its - * non-blocking connect loop, so it wakes up at least once per - * probe_connect_timeout_ms to check the shared election_done flag. - * This bounds the worst-case overshoot of election_timeout_ms - * to one probe_connect_timeout_ms interval per thread. - * ha_init() enforces: probe_connect_timeout_ms >= 500ms AND - * probe_connect_timeout_ms <= election_timeout_ms. */ - uint32_t max_promotion_retries; /* Max consecutive failed promotion attempts before giving up (default: 5) */ -} ha_config_t; -``` - -Added to `cluster_t` as `ha_state_t ha;`. - -### 5.4 Startup Sequence - -``` -Controller starts - │ - ▼ - Read candidates[] from config - Find my_priority (my position in list) - │ - ▼ - ┌──────────────────────────────────┐ - │ am_i_primary()? │ - │ │ - │ Launch one pthread per candidate │ - │ with priority < my_priority. │ - │ Each thread makes a synchronous │ - │ blocking TLS dial (no iomgr yet) │ - │ and sends OP_heartbeat probe. │ - │ │ - │ Single shared deadline: │ - │ election_timeout_ms (2000ms) │ - │ enforced via condvar/timedwait │ - │ │ - │ If ANY thread gets a live ACK │ - │ before deadline: → NO (standby) │ - │ Cancel remaining threads. │ - │ │ - │ If deadline expires, all probes │ - │ failed/timed out: → YES (primary)│ - └──────┬──────────────┬────────────┘ - │ │ - PRIMARY STANDBY - │ │ - ▼ ▼ - cluster_init() cluster_init() - persist_init() persist_init() - persist_recover() persist_recover() ← iomgr NOT yet running - persist_start() persist_start() workers get ECONNREFUSED, - connect to primary retry via xce_reconnect() - cluster_start() (via iomgr, now running) ← iomgr starts here - ha_heartbeat_watch() workers can now connect - standby_loop() -``` - -> **Promotion recovery**: A standby runs `persist_recover()` on startup (populating kvmgrs from its local snapshot + WAL) and then receives ongoing replication via `apply_wal_entry()` calls from `OP_replEntry`. By the time promotion occurs, in-memory state is already complete from both sources. It must **not** call `persist_recover()` again — that would replay the local WAL and double-apply every entry. Instead, the promotion path should: (1) validate CRC chain from the standby's own `persist_t.checkpoint_lsn` forward (already maintained by startup recovery + replication), (2) flush the local WAL fsync to ensure all replicated entries are on disk, (3) update the local WAL `term` to the new election term, (4) start replication to remaining standbys from the current LSN. If the promoted standby has gaps in its WAL (async entries not yet replicated before the old primary died), it requests the missing entries from the old primary's WAL on disk via catch-up (see Part 7 catch-up protocol). - -> **Startup ordering note**: `am_i_primary()` runs before `cluster_init()` (which starts iomgr/SSL/vtaskmgr). The election probe therefore launches one `pthread` per lower-priority candidate; each thread makes a synchronous blocking TLS dial using the same cert/key pair as the HA pipeline but without the event loop. All threads share a single `election_timeout_ms` deadline enforced via a `pthread_cond_timedwait`. -> -> **IMPORTANT — thread cancellation safety**: `pthread_cancel()` is **NOT** used to stop election threads. `pthread_cancel()` running through OpenSSL shutdown handlers in a thread mid-handshake can deadlock (cancellation points inside TLS negotiation are not safe). Instead, use a shared `atomic_int election_done` flag (from ``) with `atomic_store()` / `atomic_load()` operations: -> - The main thread sets `atomic_store(&election_done, 1)` when the first ACK or the deadline fires -> - Each probe thread checks `atomic_load(&election_done)` after each probe attempt and exits cleanly if set (closing its TLS connection and freeing resources) -> - The main thread calls `pthread_join()` on all threads (which return promptly since they check the flag) -> - If a thread is blocked during TLS dial, use a non-blocking `connect()` + `poll()` with a short per-attempt timeout so each thread can check `election_done` between probe retries -> - Total worst-case latch is bounded by one `poll()` timeout interval per thread -> -> After the election resolves, `cluster_init()` starts iomgr and all HA traffic uses the iomgr/SSL/endpoint pipeline. This avoids a circular dependency between election and iomgr startup, and parallel probing bounds worst-case startup latency to one `election_timeout_ms` regardless of cluster size. -> -> -> **`ha_init()` mutex initialization ordering**: `ha_init()` must be called **before** `am_i_primary()` (the election probe), since `ha_state_t.lock` (the `pthread_mutex_t` protecting `primary_ep` and `last_heartbeat`) is initialized in `ha_init()`. Step 5.5 starts heartbeats before `cluster_start()` (via a dedicated pthread) — this pthread accesses `ha->lock`, which must already be initialized. Promotion (`ha_promote_to_primary()`) does NOT re-initialize the mutex — the mutex is initialized once in `ha_init()` and persists for the controller's lifetime across promote→demote→re-promote cycles. Re-initializing a pthread_mutex_t that may have waiters is undefined behavior. -> -> **`ha_init()` startup enforcement**: At startup, `ha_init()` validates: `heartbeat_interval_ms >= 2000` (the HA watchdog uses `nanosleep()` and is not subject to `delay_task()` integer-second truncation; the 2000ms floor is a conservative practical minimum to avoid false heartbeat-timeout triggers from SSL handshake jitter on loaded management LANs); `heartbeat_timeout_ms >= heartbeat_interval_ms`; `election_timeout_ms >= 500`; `ncandidates >= 1`. **Duplicate candidate validation**: `ha_init()` scans `candidates[]` and rejects any config where two entries share the same `host` (case-insensitive `strcasecmp`) or the same `host:port` pair. Duplicate candidate names cause ambiguous election probes — an entry appearing twice would receive two heartbeat monitors, trigger double replication connections, and confuse the priority ordering. The validation must log a `FATAL` error naming both duplicate entries and return `-EINVAL`. See `ha_config_t` in Part 5.3 for field documentation. - -### 5.5 Monitor Thread (Standby) - -The standby uses the controller's iomgr/vtask heartbeat infrastructure to monitor higher-priority candidates. The primary sends periodic heartbeat messages (`OP_heartbeat`) to connected standbys at `ha_config.heartbeat_interval_ms` (default 3000ms), and standbys watch for heartbeat timeouts from higher-priority controllers at `ha_config.heartbeat_timeout_ms` (default 10000ms — approximately 3 missed heartbeats): - -```c -/* The monitor is not a separate thread with raw TCP probes. - Instead, it uses the existing iomgr/vtask/endpoint pipeline: - - 1. The primary sends OP_heartbeat to all connected standbys - at HA_HEARTBEAT_INTERVAL_MS (default 3000ms). - 2. Each standby tracks last_heartbeat_time for the primary endpoint. - 3. If last_heartbeat_time + HA_HEARTBEAT_TIMEOUT_MS < now, - the primary is considered dead -> promote self. - - This provides application-level liveness (not just TCP connectivity) - and reuses the SSL/endpoint/vtask pipeline that all controller - connections already use. */ - -/* ha_check_heartbeat() — called from the dedicated HA watchdog pthread (NOT delaymgr). - * P1-3 requires a dedicated pthread with nanosleep() or timerfd_create(CLOCK_MONOTONIC) - * for sub-second heartbeat detection. delaymgr has 2-second tick granularity - * (cluster.c initialises it with precision=2), making it unsuitable here. - * Uses CLOCK_MONOTONIC — immune to NTP jumps and DST changes that would cause - * spurious promotions (clock jumped forward) or missed timeouts (clock jumped back). */ -static void ha_check_heartbeat(cluster_t* c) -{ - ha_state_t* ha = &c->ha; - - if ((ctrl_role_t)atomic_load_explicit(&ha->role, memory_order_acquire) != CTRL_STANDBY) - return; - - /* Read primary_ep and last_heartbeat under ha->lock — iomgr clears primary_ep - * (NULL-before-free) while holding this same lock, so we see either a valid - * pointer or NULL, never a dangling one. */ - pthread_mutex_lock(&ha->lock); - int no_primary = (ha->primary_ep == NULL); - struct timespec last_hb = ha->last_heartbeat; /* snapshot under lock */ - pthread_mutex_unlock(&ha->lock); - - if (no_primary) - return; - - struct timespec now; - clock_gettime(CLOCK_MONOTONIC, &now); - int64_t elapsed_ms = (now.tv_sec - last_hb.tv_sec) * 1000LL - + (now.tv_nsec - last_hb.tv_nsec) / 1000000LL; - if (elapsed_ms > (int64_t)ha->config.heartbeat_timeout_ms) { - /* Primary heartbeat timeout — promote self. - * If a previous promotion attempt failed (promotion_failed flag set), - * retry promotion unless the retry limit is exceeded. */ - if (ha->promotion_failed && - atomic_load_explicit(&ha->promotion_retry_count, memory_order_relaxed) >= (int)ha->config.max_promotion_retries) { - ERROR("cannot promote after %d attempts — manual intervention required", - atomic_load_explicit(&ha->promotion_retry_count, memory_order_relaxed)); - return; - } - ha_promote_to_primary(c); - } -} -``` - -**Heartbeat detection latency**: The heartbeat watchdog runs on a dedicated `pthread` with `nanosleep()` (see P1-3 and `ha_check_heartbeat()` above), identical to the WAL fsync timer approach in Part 2.4. This achieves sub-second detection granularity. The actual primary-failure detection latency is `heartbeat_timeout_ms + up to heartbeat_interval_ms` milliseconds — the timeout fires, plus up to one heartbeat interval of jitter before the next check. For the default `heartbeat_timeout_ms = 10000` and `heartbeat_interval_ms = 3000`, worst-case detection is 13 seconds. For HPC workloads requiring sub-10-second failover, reduce `heartbeat_timeout_ms` and `heartbeat_interval_ms` accordingly. - -### 5.6 Promotion Sequence - -When a standby promotes to primary: - -1. **Stop WAL replay** from old primary -2. **Increment term** (`ha.term++`) — fencing token for stale standbys -3. **Flush the local WAL** (`persist_flush()` — fsync the last WAL segment to ensure all replicated entries are on disk) and **verify WAL consistency** (validate CRC chain from the standby's own last snapshot LSN forward — standbys run `persist_recover()` on startup and maintain their own local WAL and snapshots, so the validation starting point is the standby's `persist_t.checkpoint_lsn`). Do **NOT** call `persist_recover()` again — the standby's kvmgrs are already populated from replication (via `apply_wal_entry()` during normal `OP_replEntry` processing) AND from the startup recovery pass. Calling `persist_recover()` a second time would double-apply entries. Only a WAL flush + CRC chain validation is needed. **WAL gap handling**: Async-replicated entries that the old primary committed but had not yet replicated before crashing are **lost** — the dead primary cannot serve catch-up. This is a known limitation of the async replication model: non-sync WAL entries (job starts, finishes, rusage updates) may be lost on failover. The promoted primary's view is consistent but potentially stale; async state gaps self-heal through worker reconnection (see step 9). **WAL consistency guarantee**: the standby's local WAL is a strict prefix of the old primary's WAL — standbys only write entries received via `OP_replEntry` from the primary. No divergence is possible if the standby was continuously connected; the standby's WAL cannot contain entries the old primary did not originate. The only split-brain divergence scenario (standby accepted writes from clients during a partition) cannot occur because standbys do not accept writes — they only redirect to the primary. **CRC chain restart on promotion (COH-5)**: After the WAL flush, `persist_flush()` MUST reset `persist_t.last_replicated_crc` to `WAL_CRC_NONE`. This signals that the promoted primary is starting a new replication chain — any remaining standbys have not received entries from this new primary yet, so there is no valid "previous entry CRC" to chain against. The first entry this primary broadcasts will carry `prev_crc = WAL_CRC_NONE`, which the standby receiver accepts as a valid chain start (see Part 6.4 catch-up and segment-boundary handling). Without this reset, the new primary would broadcast `prev_crc = last_replicated_crc` (the CRC of the last entry the old primary sent), which no standby would recognize — causing a chained-CRC mismatch disconnect on the first broadcast. -4. **Switch role** to `CTRL_PRIMARY` — only after recovery succeeds. **If recovery fails, abort this promotion attempt and set `ha.promotion_failed = 1`**. The standby remains in `CTRL_STANDBY` role, but since the old primary is dead (that triggered this promotion), the next heartbeat watchdog check (`ha_check_heartbeat()`) will re-fire `ha_promote_to_primary()`. The `promotion_failed` flag gates retry: the standby retries promotion on each watchdog tick unless: (a) a new heartbeat is received from a higher-priority candidate (another controller won the race), or (b) a configurable `max_promotion_retries` (default: 5) is exceeded, at which point the standby logs `FATAL "cannot promote after %d attempts — manual intervention required"` and stops retrying. This prevents a permanent no-primary state while bounding the retry window. -5. **Register persistence hooks** — standby did not register persistence hooks during startup (hooks are registered only after `persist_recover()` completes, and only on the primary path; standbys register them here on promotion). **Idempotent registration guard**: `ha_promote_to_primary()` must check whether hooks are already registered before adding them (e.g., via a `ha->hooks_registered` flag in `ha_state_t`). Without this guard, a promote→demote→re-promote cycle would accumulate duplicate hooks that fire N times per event, causing double WAL writes and replication sends. -5.5 **Start heartbeat immediately** — begin sending heartbeat messages to other standbys BEFORE `cluster_start()` (step 8). The promoted standby must establish its authority as the new primary by sending heartbeats with its incremented term. This prevents other standbys from timing out and promoting themselves during the potentially lengthy replicator initialization steps. - -**Pre-iomgr mechanism**: At this point in the promotion sequence, `cluster_start()` (and thus iomgr) has not yet run. The heartbeat messages are sent via the **same raw TCP/SSL pthread mechanism as the election probes** (Part 5.4): one short-lived `pthread` per lower-priority standby, each making a synchronous blocking TLS dial, sending a single `OP_heartbeat` frame with the new term and role, and closing the connection. This is intentionally distinct from the iomgr/endpoint pipeline. After `cluster_start()` (step 8), all subsequent HA traffic (heartbeats, replication) switches to the iomgr/SSL/endpoint pipeline permanently. - -The heartbeat thread is launched with the new term and role set to PRIMARY, but `cluster_start()` (step 8) is deferred until all preparation steps complete. **CRITICAL**: The heartbeat messages sent during this pre-operational phase contain only term and role — no LSN or state information. Other standbys receiving these heartbeats update their `ha.term` and `primary_host` to the new primary, but do not expect any replication data until step 8 is reached. **Bounded timeout for iomgr follow-up**: After a standby receives a pre-iomgr heartbeat and updates `primary_host`, it starts a countdown of `2 × heartbeat_timeout_ms`. If no iomgr-level `OP_heartbeat` arrives within this window from the claimed new primary, the standby resets `primary_host` to empty and re-runs `am_i_primary()`. This bounds the "stuck on a dead promoted primary" scenario: if the promotion crashes mid-sequence (after step 5.5 pthreads sent heartbeats but before `cluster_start()` at step 8), other standbys detect the absence of iomgr heartbeats and re-elect independently. Without this timeout, a standby that updated `primary_host` but never receives an iomgr heartbeat from the promoted primary would wait indefinitely at `heartbeat_timeout_ms` intervals, never re-running the election. -6. **Initialize replicator** — establish connections to remaining standbys -7. **Scheduler gating is implicit** — the scheduler has not reconnected at this point in the promotion sequence (it reconnects in step 9 via `xce_reconnect()`). There is no active "gate the scheduler" protocol step: the scheduler simply isn't connected, so no scheduling can occur. When the scheduler reconnects (step 9), it self-heals by sending a full state-sync query to enumerate all PEND and RUN jobs, rebuilds its allocation table from scratch, and resumes scheduling from a clean slate — this is the actual gating mechanism. **No additional synchronization is required here.** Note: the old text said "pause or flush the scheduler's run queue before accepting writes" — this was incorrect; the scheduler's queue is a scheduler-internal data structure that the controller does not directly control. -8. **Start normal primary operation** — `cluster_start()`, accept connections -9. **Workers and scheduler reconnect automatically** — when the old primary's TCP connections drop, libxce's `xce_reconnect()` backoff+jitter loop engages on each worker and scheduler. They retry through the candidate list until they reach the promoted primary, then send a standard `OP_workerJoin` / `OP_schedulerJoin`. No special failover logic is needed in the promoted primary — workers rejoin via the existing `handle_worker_join()` path and re-report their current job state. The scheduler re-queries job state when scheduling. The promoted primary is functionally equivalent to a freshly booted controller that has replayed its WAL — it has a consistent but potentially stale view of job lifecycle states, which is corrected as workers reconnect and update their status. No "re-push config" logic is needed on promotion: the join response carries runtime config as usual. - -This means async-replicated job lifecycle transitions (start, finish, fail, stop, resume, rusage) that were lost when the old primary crashed are not present in the promoted standby's WAL. The promoted standby's view of these jobs may be stale (e.g., showing a job as PEND when it is actually RUNNING). Workers correct this state when they reconnect and re-report their current job status. Until all workers reconnect, the scheduler should treat job state as uncertain and avoid scheduling decisions that depend on stale information. - -**Job state during the reconnection window**: Jobs that were in `JOB_STAT_RUN` on the old primary but have no matching update in the standby's WAL enter a **transient unknown state**. The promoted primary marks such jobs with a sentinel flag (e.g., `JOB_STAT_TRANSIENT`) to signal uncertainty to operators and the scheduler. The scheduler must not schedule new work onto workers that own transient jobs (their status is unknown — the job may have completed, crashed, or still be running). When the **owning worker** (the specific worker that held the job at promotion time, identified by `job_t.exec_host`) reconnects via `handle_worker_join()`, it re-reports its current job status and the job transitions from `JOB_STAT_TRANSIENT` to its actual state (`JOB_STAT_RUN`, `JOB_STAT_EXIT`, `JOB_STAT_DONE`, etc.). **Only the owning worker's reconnect clears `JOB_STAT_TRANSIENT` for that specific job** — reconnection of any other worker does not affect transient jobs on different workers. `reconciliation_progress` is a monitoring convenience metric (aggregate reconnect count); it is not the mechanism that resolves individual transient jobs. This guarantees eventual consistency: no job state is permanently lost, only temporarily uncertain until the owning worker reconnects. If a worker never reconnects (hardware failure, network isolation), jobs on that worker remain transient until an operator intervenes — this is the same as today's behavior when a worker goes `WORKER_UNAVAILABLE`. - -**Stale scheduler state**: The scheduler maintains internal state derived from controller state (resource allocations, pending job counts per queue, scheduling constraints). This internal state is also stale after failover. The scheduler heals by re-initializing from scratch when it reconnects to the promoted primary via `xce_reconnect()`: on reconnect, the scheduler sends a full state-sync query to enumerate all PEND and RUN jobs, rebuilds its allocation table, and resumes scheduling from a clean slate. Admin-initiated changes (queue configuration changes, admin holds on jobs) are `WAL_QUEUE_CTRL` / `WAL_WORKER_CTRL` — **ASYNC-replicated events** and subject to the same loss window as job lifecycle events. If the old primary crashed before these entries were replicated to the standby, they are absent from the promoted standby's WAL. **The scheduler must re-query all state** (job, queue, and worker configuration) on reconnect — it cannot selectively trust queue/worker configuration as more reliable than job lifecycle state. Both are subject to the async replication lag at the moment of failure. - -**Reconciliation window definition**: The reconciliation window is the interval between promotion (step 2) and the point where all workers that were connected to the old primary have re-joined the promoted primary. **Scheduler behavior during the reconciliation window**: (1) New job submits are accepted normally — these have authoritative state. (2) Scheduling of PEND jobs onto reconnected workers proceeds normally. (3) Scheduling of PEND jobs onto **unreconnected** workers is blocked — the scheduler does not know whether those workers still have capacity (transient jobs may still be running). (4) The scheduler tracks a `reconciliation_progress` metric: `(workers_reconnected / workers_at_promotion_time)` where `workers_at_promotion_time` = count of workers in the promoted standby's kvmgr with status NOT including `XCE_WORKER_UNAVAIL` or `XCE_WORKER_HIDDEN` at the moment promotion completes (step 2). This count is derived from the standby's own replicated state — no data from the dead primary is needed. (5) Once `reconciliation_progress >= 1.0` (all workers have reconnected), normal scheduling resumes on all workers. **Expected window duration**: 30–120 seconds for a healthy cluster (workers reconnect within one `worker_heartbeat_interval`, typically 15–30 seconds, plus one scheduling cycle for the scheduler to catch up). If workers are slow to reconnect (e.g., due to DNS or SSL re-handshake latency), the window extends. If a worker never reconnects, its jobs remain in `JOB_STAT_TRANSIENT` until the operator intervenes — these workers are excluded from `reconciliation_progress` after a configurable `worker_reconnect_timeout_s` (default: 300 seconds). **Stuck-worker alarm**: when any worker hits the `worker_reconnect_timeout_s` boundary without reconnecting, log at `WARN`: `"Worker %s did not reconnect within %us after failover — %d transient jobs on this worker require manual resolution"` and expose `ctrl_reconciliation_stuck_workers` gauge (count of workers excluded from `reconciliation_progress` due to timeout). This gives operators a concrete list of workers requiring investigation rather than requiring them to correlate `ctrl_transient_jobs_stuck` with worker connection state. - -### 5.7 Term (Fencing Token) - -The `term` field prevents split-brain: - -- Incremented on every promotion -- Included in every WAL entry and replication message -- A standby receiving a message with a lower term than its own ignores it -- Workers/scheduler reconnect and verify term on join - -**Partition heal — old primary returns**: When a partitioned primary recovers network connectivity, it resumes sending heartbeats and replication messages with its old (lower) term. The promoted primary (higher term) ignores these messages (term < own term rule above). The partitioned primary receives a heartbeat reply or replication message from the promoted primary carrying the higher term. On receiving any message with `term > ha.term`, the old primary must detect that it has been superseded and demote itself: set `ha.role = CTRL_STANDBY`, stop accepting writes, and reconnect to the promoted primary as a standby. The `handle_msg()` path for all incoming messages must check `hdr.term > ha.term` and call `ha_demote_to_standby()` when this condition is detected, then re-process the message in standby mode. This prevents the old primary from continuing to accept writes after the partition heals, eliminating the split-brain write window. - -**Asymmetric partition — old primary never receives higher-term message**: If the partition only partially heals (old primary can reach workers and API servers but NOT the promoted standby), the old primary continues accepting job submits indefinitely — it never receives a higher-term message and cannot self-demote. This is the **unbounded split-brain scenario** acknowledged in Part 5.1. Mitigation: the promoted primary sends heartbeats to the old primary's address (via the pre-iomgr mechanism in step 5.5 and the iomgr pipeline after step 8). If the asymmetric partition severs primary→standby but allows standby→primary traffic, the old primary will eventually receive a heartbeat-reply with the new term. If the partition is bidirectionally severed between the two controllers, no automatic resolution exists — operator intervention is required (SIGUSR1 manual failover on the old primary, or force-restart it). The `ctrl_split_brain_detected` metric captures this condition once any cross-controller message is received after partition heal. - -**Hook disabling on demotion**: `ha_demote_to_standby()` must disable the persistence hooks that were registered during promotion (step 5 above). The chosen approach is **disable-in-place via `xce_disable_hook()`** (to be added to `src/lib/hook.c`) rather than true removal. - -**Memory ordering**: `hook_item_t.fn` must be declared as `_Atomic hook_fn_t` (C11 atomic function pointer). `xce_disable_hook()` stores NULL with `atomic_store_explicit(memory_order_release)`. `xce_run_hooks()` loads `fn` into a local variable with `atomic_load_explicit(memory_order_acquire)`, then checks and calls the local — two reads of the same non-atomic variable can diverge on weakly-ordered architectures, but a single atomic load into a local is a single read with a visibility guarantee. - -**Slot reuse to bound list growth**: On each promote→demote cycle, a NULL slot is left in the hook list and a new slot is added on re-promote. To prevent unbounded growth, `xce_add_hook()` must scan for an existing NULL `fn` slot of the same `type` before appending a new entry. If a matching NULL slot is found, it reuses it (sets `fn` and `data`). This bounds the hook list to max(ever-registered) slots regardless of cycle count. - -Disabling must happen before the role switch to `CTRL_STANDBY` so that no in-flight requests trigger hooks after the role has changed. The `hooks_registered` idempotency guard (see step 5) is reset to 0 on disable so that the next promote re-registers correctly. The role-check defense-in-depth (checking `ha.role == CTRL_PRIMARY` inside each hook function before proceeding) provides a second layer: even if a hook entry fires during the disable window, it does nothing because the role has already switched to STANDBY. - -**In-flight request window during demotion (CRITICAL)**: Between the higher-term message arriving at `handle_msg()` and the demotion taking effect, vcpu threads may already be executing `handle_jobsubmit()` past the `AFTER_EVENT` hook. These in-flight requests have the following risks: -1. The persistence hook already called `persist_wal_sync_append()`, which is writing to the local WAL and may have already sent `repl_broadcast()`. Standbys following the higher-term primary will ignore the lower-term replication message. -2. After `ha_demote_to_standby()` sets `ha.role = CTRL_STANDBY`, the vcpu thread continues executing and eventually calls `xce_reply()`, ACKing the client for a job that is NOT replicated to the new primary's standbys. - -**Mitigation — role check in hot paths**: To bound this window, both the persistence hook and `repl_broadcast()` must check `ha.role` before proceeding: -- `persist_job_hook_fun()`: before calling `persist_wal_sync_append()`, check `(ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_PRIMARY`. If `ha.role != CTRL_PRIMARY`, return `-E_XCE_NOT_PRIMARY` without writing WAL or replicating. The caller (`handle_jobsubmit`) treats this as a failed submit and replies to the client with an error. -- `repl_broadcast()`: before sending to any standby, check the primary's `(ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_PRIMARY`. If the role has changed, skip the broadcast and return an error. -These checks use `atomic_load_explicit(&c->ha.role, memory_order_acquire)` — `ha.role` must be an atomic field (or at minimum guarded by a compiler barrier) so that the vcpu thread sees the demotion promptly. **The corresponding write in `ha_demote_to_standby()` must use `atomic_store_explicit(&c->ha.role, CTRL_STANDBY, memory_order_release)` — a plain write (`ha.role = CTRL_STANDBY`) does not pair with the acquire load and may not be visible to vcpu threads in time, especially on weakly-ordered architectures or with compiler optimizations that cache the value in a register. The release-acquire pair is essential for the in-flight window mitigation to work correctly.** Note: these checks are defense-in-depth — the primary mechanism is the `handle_msg()` term check. The in-flight window is bounded by the time between `ha_demote_to_standby()` setting the role and the vcpu thread reaching the next role check, which is at most one sync replication timeout cycle (`sync_timeout_ms`, default 50ms). Any client ACK sent during this window is acknowledged as a split-brain risk (see Part 5.1). - -### 5.8 Non-Preemptive Failover - -``` -Time ─────────────────────────────────────────────▶ - -ctrl-1 (P=0): ████████░░░░░░████████████████████████ - PRIMARY DEAD PRIMARY (recovered, - │ does NOT reclaim) - │ -ctrl-2 (P=1): ░░░░░░░░██████████████████████████████ - STANDBY PRIMARY (promoted at T1) -``` - -When ctrl-1 recovers: -1. It probes for candidates with priority < 0 → none exist -2. It becomes primary? **No** — it probes ctrl-2 and finds it alive -3. ctrl-1 stays standby and connects to ctrl-2 as primary -4. Admin can trigger manual failover to restore optimal controller placement: - -**Manual failover protocol**: To move primary back to a higher-priority candidate (e.g., ctrl-1 after recovery), the operator sends `SIGUSR1` to the current primary (ctrl-2). The SIGUSR1 signal handler **must only set an atomic flag**: `atomic_store(&ha->failover_requested, 1)`. It does NOT call `ha_demote_to_standby()` directly — that function performs operations (mutex locks, fsync, SSL writes, memory allocation) that are not async-signal-safe per POSIX. The failover is executed by the dedicated HA watchdog thread (the same pthread running `ha_check_heartbeat()` — P1-3), which polls `ha->failover_requested` on each tick. When the flag is observed, the watchdog thread calls `ha_demote_to_standby()` from its safe thread context, which: (a) unregisters persistence hooks (see Part 5.7 hook unregistration), (b) sets `ha.role = CTRL_STANDBY` using `atomic_store_explicit` with `memory_order_release`, (c) stops accepting new writes, (d) flushes and sync-replicates any pending WAL entries, (e) sends a `OP_steppingDown` message to all connected standbys with its current term and the election deadline. On receiving `OP_steppingDown`, the highest-priority connected standby (ctrl-1) immediately begins its election (calls `ha_promote_to_primary()`). The stepping-down controller then reconnects to the newly promoted primary as a standby. **During the transfer window (~1-2 election_timeout_ms), no controller is primary — job submits receive `EAGAIN` and must retry.** This is the same brief unavailability as an automatic failover. The `SIGUSR1` approach avoids needing a dedicated admin RPC or CLI command in the initial implementation; a future `xce hostcontrol --failover` CLI can wrap this by sending the signal remotely via SSH. Add `atomic_int failover_requested;` to `ha_state_t` for this purpose. - -**SIGUSR1 access control note**: Signal delivery to a process requires the sender to have the same UID (or be root). The controller runs as a dedicated service account; on HPC management nodes (login nodes, admin nodes), where multiple user accounts may exist, any process running as the controller's service account can trigger a failover. **Mitigation (in scope for P1-3a)**: add `--disable-sigusr1-failover` as a startup flag (single `atomic_int sigusr1_failover_enabled` field checked in the signal handler before setting `failover_requested`). Operators on shared management nodes should deploy with this flag set; `xce hostcontrol --failover` (SSH-authenticated) becomes the only failover path once available. The flag is one additional line in the signal handler and should ship with P1-3a rather than being deferred to future work. - -**Split-brain demotion audit log**: When `ha_demote_to_standby()` is triggered by receiving a higher-term message (partition heal path — NOT the SIGUSR1 path), it must log the range of WAL entries this controller generated during the split-brain window before discarding them: `ERROR("Split-brain demotion: discarding local WAL entries lsn=[%lu..%lu] (term=%lu) — these %lu job submits were ACKed to clients but are NOT durable. Operators: treat job IDs in this LSN range as lost. ctrl_split_brain_lost_submits+=%lu", divergence_start_lsn, current_lsn, ha->term, entry_count, entry_count)`. `divergence_start_lsn` is the LSN at which this controller's term diverged from the promoted primary's term (recorded when this controller first started accepting writes after the partition began). Increment `ctrl_split_brain_lost_submits` counter by `entry_count`. This log gives operators a bounded audit trail without requiring a quorum protocol or external log aggregation. - -**Demotion failure handling**: If `persist_flush()` fails during `ha_demote_to_standby()` (step d above), the controller has already set `ha.role = CTRL_STANDBY` and incremented term. The safest recovery is to **not rollback** the role change — instead, log a `FATAL("persist_flush failed during demotion — WAL may be incomplete. Manual investigation required.")`. The demoted controller will not accept new writes (role is STANDBY), and the WAL will be flushed again on the next checkpoint. A role rollback (setting `ha.role = CTRL_PRIMARY` on flush failure) would leave the cluster in a split-brain-eligible state where the original primary continues accepting writes while a new primary may already be promoting. Failing forward as STANDBY is safer: the cluster temporarily has no primary until another standby wins election, which is the same state as a normal failover. - -### 5.9 HA Connection Security (Trusted LAN) - -XCE controllers run on a **dedicated management LAN**. All HA connections (primary→standby replication, standby→primary heartbeat, election probes) use the same iomgr/SSL pipeline as worker/scheduler connections, providing transport encryption. No mutual TLS (mTLS) client certificate verification is required between controllers. - -**Trust model**: The management network is the security boundary. Only nodes on the management LAN can reach controller HA ports. This is the same model used by traditional HPC middleware (LSF, SLURM, PBS) — network-level access control (firewall rules, dedicated VLAN) replaces PKI-based peer authentication. - -- **Election probe**: The initial `am_i_primary()` probe (before iomgr is started — see Part 5.4 startup sequencing note) launches one `pthread` per lower-priority candidate, each making a **synchronous blocking TCP/SSL dial**. All threads share a single `election_timeout_ms` deadline; the first live response wins. Peer identity is confirmed by host:port matching against the `--peers` candidate list. After the election resolves, all subsequent HA traffic uses the iomgr/SSL pipeline for heartbeat and replication. -- **No PKI requirement**: HA does not require a cluster CA or controller-specific certificates. The existing SSL infrastructure (used for worker/scheduler connections) is reused for controller-to-controller channels via iomgr. **Election probe SSL context**: The pre-iomgr election probes (Part 5.4) cannot reuse the iomgr SSL context because iomgr has not yet started. Each probe thread creates a standalone `SSL_CTX` using the same cert/key pair as the iomgr context. Extract a shared helper `ssl_ctx_create(cert_path, key_path)` (or equivalent) used by both the election probe threads and iomgr initialization to avoid code duplication. The election probe SSL contexts must be freed after all probe threads join (`pthread_join`), before `cluster_start()` initializes iomgr's own SSL context. No mTLS or PKI changes are required — the same cert/key pair is used throughout. -- **Optional enhancement — SSL_VERIFY_PEER for controller-to-controller channels**: The current design validates peer identity by host:port matching against `--peers`. Enabling one-way TLS server certificate CN/SAN verification (`SSL_CTX_set_verify(SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT)`) on the HA client side would convert this network-layer assertion to a cryptographic one — at no additional PKI cost since the same cert/key infrastructure already exists. This is not required for the trusted-LAN model but would be a low-effort P3 hardening step (`P3-8`): configure `SSL_CTX_set_verify` on the election probe SSL context and the iomgr HA connection path; verify the server cert's CN or SAN matches the candidate's configured hostname. The `--ca-cert` flag added for libxce (P2.5-5) would serve this purpose. Deploy only if the management network boundary cannot be enforced via firewall/VLAN. - -### 5.10 New Opcodes and Cap'n Proto Additions - -```capnp -/* In HostRole enum: */ -standby @5; /* NEW — standby controller role */ - -/* In OPCode enum (existing entries go up to @21 — clrpreason): - * @22 is intentionally skipped in the OPCode enum to leave room for a future - * non-HA opcode. The new HA opcodes start at @23. */ -replEntry @23; /* Replicated WAL entry (primary → standby) */ -replAck @24; /* ACK for sync-replicated entry (standby → primary) */ -redirect @25; /* Redirect to current primary (standby → client) */ -heartbeat @26; /* Primary heartbeat to standbys (primary → standby) */ -steppingDown @27; /* Primary is stepping down (primary → standby, triggered by SIGUSR1) */ -replSnapshot @28; /* Snapshot transfer for catch-up (primary → standby, chunked) */ -replCatchUpDone @29; /* Signal that catch-up snapshot is loaded (standby → primary) */ - -/* In Request struct (struct ordinals are separate from OPCode enum values): - * @22 here is the Request struct field ordinal — it does NOT conflict with - * the OPCode @22 gap above. These are different namespaces in Cap'n Proto. */ -term @22 :UInt64 $C.mapname("term"); /* NEW — fencing token */ -``` +> **SUPERSEDED.** This section (originally: a non-quorum, priority-list election model — "am I the highest-priority alive candidate?" — chosen instead of Raft consensus, gated by a mandatory `ha.split_brain_risk_accepted: true` config key) described a design that was later replaced. See: `src/docs/persistence-and-ha-design.md` findings and the closed specs `raft-library`, `raft-ha-core-safety`, `ha-prevote`, `election-iomgr-refactor`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`, and `ha-single-leader-role` in `.kiro/specs/` for what shipped instead — the actual implementation is a full quorum-based Raft protocol (`src/lib/raft/raft.c`, `ha_has_write_quorum()`, `ha_start_election_async()` in `src/controller/ha.c`). The `split_brain_risk_accepted` config key was never shipped. --- @@ -2207,107 +1109,7 @@ This catch-up mechanism is analogous to etcd's snapshot + WAL catch-up, simplifi ## Part 7: Mixed Sync/Async Replication Protocol -### 7.1 Design Rationale - -The user's mental model is: **"if the API server ACKed my job submit, the job survives a crash."** This requires sync replication for job submits only. - -All other events (job start, finish, fail, rusage, queue/worker control) are self-healing: -- If a job start is lost, the worker will re-notify on reconnect -- If a finish is lost, the job appears running until the worker re-reports -- Queue/host control is infrequent and can be re-sent by the admin - -**Sync replication** = WAL write + wait for at least one standby ACK before ACKing the client. -**Async replication** = WAL write + fire-and-forget send to standbys. - -### 7.2 Sync Mode: Job Submit Flow - -Sync replication sends WAL entries to **all** standbys in parallel, then waits for at least one ACK: - -``` -API Server Controller (Primary) Standby-1 Standby-2 - │ │ │ │ - │ job submit request │ │ │ - │──────────────────────────▶│ │ │ - │ │ BEFORE_EVENT hook │ │ - │ │ job_submit() │ │ - │ │ AFTER_EVENT hook ─┐ │ │ - │ │ │ │ │ - │ │ persist_wal_sync()│ │ │ - │ │ WAL write + fsync│ │ │ - │ │ send(entry) ──────────────────────▶│ - │ │ send(entry) ──────────────────▶│ │ - │ │ wait_any_ack() │ │ - │ │◀─── ACK(ack_lsn) ───────│ │ - │ │ │ │ │ - │ │ ← hook returns │ │ │ - │ │ │ │ - │ reply(jobid) │ │ │ - │◀──────────────────────────│ │ │ - │ │ │ │ -``` - -**Parallel fan-out**: The primary sends the WAL entry to all connected standbys in parallel (fan-out, not sequential). It then waits for at least one ACK within the timeout. The first ACK received satisfies the durability guarantee — remaining standby ACKs are tracked for lag monitoring but do not block the client reply. This maximizes replication durability without adding latency proportional to the number of standbys. - -**Partial fan-out failure handling**: If `repl_broadcast()` encounters a send failure on one or more standby endpoints (e.g., `SSL_write` returns an error, or the endpoint's write FSM reports a broken connection), the broadcast continues sending to remaining standbys — a single failed send does not abort the broadcast. Failed standbys are marked for disconnection: `repl_remove_standby()` is called after the broadcast completes (not during, to avoid mutating the `standbys[]` array while iterating it). If no standby ACKs arrive within `sync_timeout_ms`, `persist_wal_sync_append()` returns `-ETIMEDOUT` (degraded mode, Part 7.4). If all standbys fail to send, the result is the same as having zero connected standbys: the entry is locally durable (WAL + fsync) but not replicated. - -**Key change**: `xce_reply()` is moved **after** `AFTER_EVENT` hooks in `handle_jobsubmit()`. This allows the persistence hook to complete sync replication before the client is ACKed. - -### 7.3 Async Mode: All Other Events - -``` -Worker Controller (Primary) Standby - │ │ │ - │ job start notify │ │ - │──────────────────────────▶│ │ - │ │ mark_job_start() │ - │ │ AFTER_EVENT hook ──┐ │ - │ │ │ │ - │ │ persist_wal_async()│ │ - │ │ WAL write │ │ - │ │ send(entry) ─────────▶│ - │ │ return immediately │ - │ │ │ │ - │ │ ← hook returns │ │ - │ │ │ - │ reply(OK) │ │ - │◀──────────────────────────│ │ -``` - -### 7.4 Degraded Mode - -When no standby ACKs within the timeout (`sync_timeout_ms`, default 50ms): - -1. **Do NOT block the client indefinitely** — timeout is hard -2. **ACK the client based on local WAL durability alone** — the primary's local WAL is already fsynced -3. **Log a warning** — monitoring tooling should alert on degraded replication -4. **Self-heal later** — when the standby reconnects, it will catch up via WAL replay or full snapshot transfer (if WAL was trimmed during the degraded window — see Part 6.6 catch-up protocol) - -**Degraded mode is transient**: In a production cluster, replication degradation only occurs during cluster upgrade, cold boot (standbys not yet connected), or network partitions. A monitoring tool or operator should watch for sustained degraded-mode warnings and investigate immediately. The cluster health system (see Part 5.5 heartbeat monitoring) tracks which standbys are connected and their replication lag. Sustained degraded operation indicates a problem requiring intervention, not a normal operating state. - -> **Out of scope**: `repl_degraded_timeout_s` and restricted mode are removed from scope (see Out of Scope section). Do **not** implement `repl_degraded_timeout_s`, `E_XCE_REPL_DEGRADED`, or restricted-mode state transitions. The text below describes the concept for context only. Monitoring and operator escalation handle sustained degradation. - -**Degraded mode timeout** *(concept only — not in scope)*: `repl_degraded_timeout_s` would add a configurable window after which the primary transitions to restricted mode (refusing new `WAL_JOB_SUBMIT` with `E_XCE_REPL_DEGRADED`). This prevents an unbounded accumulation of locally-durable-only submissions that would be lost if the primary itself fails. The escape hatch (`repl_degraded_timeout_s = 0`) would disable restricted mode for planned maintenance windows. - -**Client-visible behavior during split-brain — intentional CAP trade-off**: API server clients do not receive any explicit error or warning during a split-brain event — the controller continues to ACK job submits normally. However, job submits ACKed by the lower-term controller during a split-brain window are **silently lost** when the partition heals and the lower-term controller steps down (its diverged WAL is discarded). There is no application-level notification to the client. - -**This silence is a deliberate consequence of choosing availability over consistency under partition (AP in CAP terminology).** During a network partition, a CP system (like Raft) would refuse to accept writes until quorum is re-established — guaranteeing consistency at the cost of availability. XCE chooses the opposite: the system continues to accept job submits during a partition (availability), accepting that those submits may be lost on healing (consistency sacrifice). Real-time split-brain detection is fundamentally impossible without quorum — any mechanism that detects split-brain in real-time at the client level is equivalent to implementing distributed consensus, which is exactly what Raft provides at ~5000 LOC cost. The `split_brain_risk_accepted: true` config key is the operator's acknowledgment of this CAP position. - -Post-partition, the `ctrl_split_brain_detected` counter and the ERROR-level degraded replication log provide operator visibility. Operators who require at-most-once or exactly-once job submission semantics must either use quorum-based HA (Raft) or implement client-side deduplication using an idempotency key. - -**Split-brain is unlikely in practice**: In a typical deployment, API servers maintain a persistent connection to a single controller. During a network partition, the API server stays connected to its original controller. Since API servers are not replicated and do not run their own election, they cannot become "split-brained" themselves — they simply become isolated from the primary if the primary fails. The split-brain risk applies to scenarios where multiple API servers connect to different controllers during a partition, which requires a configuration where API servers are load-balanced across controllers. The LSF-style election + redirect mechanism (Part 10.3) ensures that API servers connected to a standby are redirected to the current primary, minimizing the window where an API server could submit to a lower-term controller. - -### 7.5 Sync Timeout Tuning - -| Setting | Latency Impact | Durability Guarantee | -|---------|----------------|---------------------| -| 10 ms | Low — client sees ~10ms extra | Strong if standby disk latency < 10ms; degrades to local-WAL-only if standby cannot fsync within the window (the protocol requires fsync-before-ACK, so any ACK received is already fsynced on the standby) | -| 50 ms | Moderate | Good — likely one standby has persisted | -| 100 ms | Higher | Strong — high probability all standbys persisted | -| No timeout | Unbounded | Strongest — but blocks on standby failure | - -**Recommended default**: 50ms. At ~1000 submits/sec, average inter-submit gap is 1ms. A 50ms timeout adds 5% to average submit latency while providing strong durability. If all standbys are healthy (RTT < 5ms), ACKs arrive well within the timeout and there's no waiting. - -**Standby WAL sync analysis**: The standby calls `persist_wal_sync_append()` for every `WAL_JOB_SUBMIT` entry received via replication, which performs a local fsync before sending ACK. In the primary's 50ms sync timeout window, the standby must: (1) receive the entry via SSL/vtask, (2) write to local WAL, (3) fsync, (4) send ACK. If the standby's WAL is on a slow disk (HDD, shared storage), the fsync alone could take 10-30ms, consuming most of the 50ms budget. Operators should configure `sync_timeout_ms` based on the slowest standby's WAL disk latency — on HDD-backed standbys, consider raising the timeout to 100ms. The interaction between standby checkpoint (which holds the WAL lock) and sync replication fsync is additive; if both are on the same disk, the worst-case latency is `fsync_time + checkpoint_hold_time`. +> **SUPERSEDED.** This section (originally: single-standby-ACK durability — "wait for at least one standby ACK" — with a degraded mode that ACKs the client on local WAL alone if zero standbys ACK within `sync_timeout_ms`) described a design that was later replaced. See: `src/docs/persistence-and-ha-design.md` findings and the closed specs `ha-commit-engine`, `ha-commit-durability`, `ha-commit-barrier-rollout`, and `ha-consistency-invariants` in `.kiro/specs/` for what shipped instead — the commit engine (`src/controller/commit.c`, `repl_wait_majority_ack()` in `src/controller/repl.c`) requires majority-quorum ACK and removed the zero-ACK-succeeds short-circuit. --- @@ -4147,7 +2949,7 @@ All system-level tunable constants are consolidated into `include/config.h` so t #define DEFAULT_HB_INTERVAL_MS 3000 /* Primary heartbeat interval (ms) */ #define DEFAULT_HB_TIMEOUT_MS 10000 /* Standby heartbeat timeout (ms) */ #define DEFAULT_ELECTION_TIMEOUT_MS 2000 /* Election probe deadline (ms) */ -#define DEFAULT_SYNC_TIMEOUT_MS 50 /* Sync replication ACK timeout (ms) */ +#define DEFAULT_SYNC_TIMEOUT_MS 500 /* Sync replication ACK timeout (ms) */ #define DEFAULT_MAX_PROMOTION_RETRIES 5 /* Max failed promotion attempts */ #define DEFAULT_HA_CERT_WARN_DAYS 30 /* Warn if cert expires within N days */ #define MAX_TERM_JUMP 10000 /* Maximum term increment accepted in a single replication diff --git a/src/docs/resreq-ast-cse-design.md b/src/docs/resreq-ast-cse-design.md index 0d8964b7..5ab317aa 100644 --- a/src/docs/resreq-ast-cse-design.md +++ b/src/docs/resreq-ast-cse-design.md @@ -1,6 +1,6 @@ # XCE Resource Requirement AST & CSE Design -> **Status**: Draft +> **Status**: Implemented — CSE intern table, GC, flatten/const-fold passes, and grammar are live in `src/lib/xce_select.c` / `src/lib/xce_select.peg`; `sched_meet_host_selection` and GC mark/sweep are wired into the scheduler (`src/scheduler/src/scheduler.c`, `src/scheduler/src/core_session.c`). > **Last Updated**: 2026-04-27 > **Revision**: 6 (multi-round review: resmgr API, atomic memory order, sel_make_key op2, makeString TOCTOU) > **Target**: `src/lib/` (C shared library), `src/include/resreq.h` @@ -108,8 +108,9 @@ to them but should not be considered final for those use cases. - GC idle invariant predicates (`sched_session_active`, `sched_eval_in_progress`, `staging_merge_complete`) — required to enforce the §5.5 runtime guard that prevents use-after-free when GC runs. These are tested as stubs in Phase 1 - and wired to real `xce_scheduler_t` fields in Phase 6 (see Phase 6 for - the stub-to-production migration contract). + and wired to real scheduler fields in Phase 6 (see Phase 6 for + the stub-to-production migration contract; the fields ended up in + `struct xce_sched_core_` in `core.h`, not `struct xce_scheduler_` — see §5.5). - live_jobs materialization — iterating `sched->jobs` to produce the flat `const xce_sched_job_t**` array required by `gc_sweep_select_nodes`. This is a Phase 6 deliverable, not a standalone library feature. @@ -1073,7 +1074,14 @@ if (sched_session_active(sched) || sched_eval_in_progress(sched) staging_merge_complete(sched)); return; /* do not sweep — prevent use-after-free */ } -gc_sweep_select_nodes(live_jobs, njobs); +/* Mark phase is a separate, caller-driven step — see the three-function + * protocol below. gc_sweep_select_nodes only sweeps; it does not mark. */ +gc_clear_marks(); +for (size_t i = 0; i < njobs; i++) { + /* dispatch on live_jobs[i]->resreq->tag and call gc_mark_select_tree() + * on each reachable select root — see the pseudocode below */ +} +gc_sweep_select_nodes(NULL, 0); ``` Where `sched_session_active`, `sched_eval_in_progress`, and @@ -1086,13 +1094,18 @@ development and unit testing), these predicates are implemented as thin wrappers around a `fake_scheduler_t` struct declared in `test_select.c` — a test-only scaffold with the same three `_Atomic int` fields. In Phase 6 (scheduler wiring), the wrapper bodies are updated to reference the real -`xce_scheduler_t` fields added in that phase. The function signatures do -not change between phases; only the struct pointer inside each wrapper body -changes. No Phase 1 stub code leaks into the production implementation. +scheduler core fields added in that phase (see below for where they ended +up). The function signatures do not change between phases; only the +struct pointer inside each wrapper body changes. No Phase 1 stub code +leaks into the production implementation. -**Phase 6 predicate fields in `xce_scheduler_t`:** Three `_Atomic int` -fields must be added to `struct xce_scheduler_` (in -`src/scheduler/include/scheduler.h`): +**Phase 6 predicate fields:** Three `_Atomic int` fields must be added to +hold this state. As implemented, they live in `struct xce_sched_core_` +(in `src/scheduler/include/core.h`) rather than in `struct xce_scheduler_` +in `scheduler.h` — the later scheduler-core core/driver-split refactor +(`.kiro/specs/scheduler-core/`) relocated the scheduler's session-lifecycle +state into a dedicated `xce_sched_core_t` embedded in the scheduler, and +these predicate fields moved with it: ```c _Atomic int session_active; /* 1 during scheduling session, 0 between */ @@ -1100,60 +1113,84 @@ _Atomic int eval_in_progress; /* 1 while eval_select loop is running, 0 other _Atomic int staging_merge_complete; /* 1 after staging merge completes, 0 before */ ``` -These fields are initialized to 0 in the scheduler's init function -(Phase 6 wiring) and set/cleared at the appropriate session-loop -transitions. The predicate wrappers read them via -`atomic_load_explicit(..., memory_order_acquire)` to ensure visibility -from the allocator thread's writes. The `fake_scheduler_t` in Phase 1 -tests uses the same field names and types, so the predicate wrapper -bodies are mechanically identical — only the container struct pointer -differs. +These fields are initialized to 0 in the core's init function and +set/cleared at the appropriate session-loop transitions (see +`src/scheduler/src/core_session.c`, referenced via `core->session_active`, +`core->eval_in_progress`, `core->staging_merge_complete`). The predicate +wrappers read them via `atomic_load_explicit(..., memory_order_acquire)` +to ensure visibility from the allocator thread's writes. The +`fake_scheduler_t` in Phase 1 tests uses the same field names and types, +so the predicate wrapper bodies are mechanically identical — only the +container struct pointer differs. + +**API shape:** mark and sweep are two separate exported functions, not one +combined pass. The scheduler (caller) owns the mark phase in full — +including the `resreq->tag` dispatch over live jobs — and only then invokes +the sweep: ``` +gc_clear_marks(void) + Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1: skip NULL + slots (vacated by zarray_del from a prior GC sweep); clear gc_mark on + every live node. + +gc_mark_select_tree(xce_select_node_t* node) + Recursive DFS from `node`, setting gc_mark on every reachable node. + Early-returns on a NULL node or a node whose gc_mark is already set, so + shared sub-nodes are marked exactly once even though multiple jobs reach + them. The **caller** (scheduler) invokes this once per live job's + select root(s), performing the resreq->tag dispatch itself: + - XCE_RESREQ_PRIMARY: gc_mark_select_tree(resreq->u.primary->select->root). + - XCE_RESREQ_COMPOUND: iterate resreq->u.compound->primaries and call + gc_mark_select_tree on each primary's select->root. + - XCE_RESREQ_ALTER: iterate resreq->u.alter->compounds, then each + compound's primaries, then call gc_mark_select_tree on each primary's + select->root. + (xce_sched_job_t has xce_resreq_t* resreq; xce_job_t only has char* resreq + and cannot access the parsed AST.) + gc_sweep_select_nodes(const xce_sched_job_t** live_jobs, size_t njobs) - 1. Mark pass: - a. Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1: - skip NULL slots (vacated by zarray_del from a prior GC sweep); - clear gc_mark on every live node. - b. For each live job in live_jobs[0..njobs-1]: dispatch on - job->resreq->tag and walk all reachable select roots, setting - gc_mark on every reachable node. (Shared sub-nodes are marked - once — gc_mark is boolean.) - - XCE_RESREQ_PRIMARY: mark from resreq->u.primary->select->root. - - XCE_RESREQ_COMPOUND: iterate resreq->u.compound->primaries and - mark each primary's select->root. - - XCE_RESREQ_ALTER: iterate resreq->u.alter->compounds, then each - compound's primaries, then each primary's select->root. - (xce_sched_job_t has xce_resreq_t* resreq; xce_job_t only has char* resreq - and cannot access the parsed AST.) - 2. Sweep pass: - a. Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1: - skip NULL slots; collect (slot_idx, node*) pairs for unmarked nodes. - **Note:** `slot_idx` here is the loop index `i` from the zarray - iteration — it is NOT stored in `xce_select_node_t` (the struct has no - `slot_idx` field). The GC sweep obtains the correct `zarray_del` argument - directly from the forward-iteration index, not from any per-node field. - b. For each unmarked node (iterate collected pairs): - - Reconstruct the intern key: `sel_node_key_t key = sel_node_to_key(node)` - (see §12 Appendix). The commutativity sort must be applied during - key reconstruction, identical to the sort applied in `makeSelectNode` - step 1, otherwise the reconstructed key hashes to a different bucket - and `key_del` silently no-ops, leaking the kvtab entry. - - Remove its key from sel_node_table.table via key_del. - (sel_node_table is initialized with datakiller=NULL so key_del - removes only the hash table entry — the node value is NOT freed by - key_del; see discussion below on datakiller.) - - If node->type == SEL_STR: call `dereference(node->u.strval)` to - release the xstring refcount acquired by `makeString()`. - All SEL_STR nodes are created via `makeString()`, so strval is - always an xstring-interned pointer — `free()` is incorrect here. - - free(node). - - zarray_del(&sel_node_table.nodes, slot_idx) — NULLs the slot and - decrements count. Does NOT swap-remove; remaining slot indices are - stable. NULL-slot skipping in step 1a handles future iterations. + Sweep-only — does **not** mark. The mark phase above + (gc_clear_marks + gc_mark_select_tree per live node) must already have + been run by the caller before this function is invoked; `live_jobs` and + `njobs` are retained only for API-compatibility and are unused inside + the function body (`(void)live_jobs; (void)njobs;`). Every production + call site therefore invokes `gc_sweep_select_nodes(NULL, 0)` after + completing its own mark phase. + a. Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1: + skip NULL slots; collect (slot_idx, node*) pairs for unmarked nodes. + **Note:** `slot_idx` here is the loop index `i` from the zarray + iteration — it is NOT stored in `xce_select_node_t` (the struct has no + `slot_idx` field). The GC sweep obtains the correct `zarray_del` argument + directly from the forward-iteration index, not from any per-node field. + b. For each unmarked node (iterate collected pairs): + - Reconstruct the intern key: `sel_node_key_t key = sel_node_to_key(node)` + (see §12 Appendix). The commutativity sort must be applied during + key reconstruction, identical to the sort applied in `makeSelectNode` + step 1, otherwise the reconstructed key hashes to a different bucket + and `key_del` silently no-ops, leaking the kvtab entry. + - Remove its key from sel_node_table.table via key_del. + (sel_node_table is initialized with datakiller=NULL so key_del + removes only the hash table entry — the node value is NOT freed by + key_del; see discussion below on datakiller.) + - If node->type == SEL_STR: call `dereference(node->u.strval)` to + release the xstring refcount acquired by `makeString()`. + All SEL_STR nodes are created via `makeString()`, so strval is + always an xstring-interned pointer — `free()` is incorrect here. + - free(node). + - zarray_del(&sel_node_table.nodes, slot_idx) — NULLs the slot and + decrements count. Does NOT swap-remove; remaining slot indices are + stable. NULL-slot skipping in step 1a handles future iterations. ``` -The `gc_mark_tree` helper does a recursive post-order mark: +**Protocol summary:** `gc_clear_marks()` → `gc_mark_select_tree(root)` per +live node → `gc_sweep_select_nodes(NULL, 0)` → `recompact_node_ids()`. Both +real call sites (`src/scheduler/src/scheduler.c` and +`src/scheduler/src/core_session.c`) follow this exact sequence, with an +`atomic_thread_fence(memory_order_seq_cst)` between the mark phase and the +sweep call to make mark writes visible before the sweep reads them. + +The `gc_mark_select_tree` helper does a recursive post-order mark: ```c /* Mark all nodes reachable from root (recursive DFS). @@ -1166,8 +1203,14 @@ The `gc_mark_tree` helper does a recursive post-order mark: * Recursion depth: bounded by the post-flatten AST depth, which is at most * SELECT_MAX_DEPTH - 1 levels (the parse-time nesting limit). Any increase * to SELECT_MAX_DEPTH must be reviewed against stack-frame size to avoid - * stack overflow. Cycles are impossible — interned nodes form a DAG. */ -static void gc_mark_tree(xce_select_node_t* node) + * stack overflow. Cycles are impossible — interned nodes form a DAG. + * + * As implemented, this is a non-static function exported from + * `xce_select.c` and declared in the public `resreq.h` header — not a + * `static` helper local to `resreq.c` — because the mark phase, including + * the resreq->tag dispatch below, is the caller's (scheduler's) + * responsibility. See §5.5. */ +void gc_mark_select_tree(xce_select_node_t* node) { if (node == NULL || node->gc_mark) return; /* NULL root or already marked */ node->gc_mark = 1; @@ -1178,12 +1221,12 @@ static void gc_mark_tree(xce_select_node_t* node) break; /* unary */ case SEL_NOT: - gc_mark_tree(node->u.operand); + gc_mark_select_tree(node->u.operand); break; /* binary */ default: /* SEL_AND, SEL_OR, SEL_GT, SEL_LT, SEL_GE, SEL_LE, SEL_EQ, SEL_NE */ - gc_mark_tree(node->u.binop.left); - gc_mark_tree(node->u.binop.right); + gc_mark_select_tree(node->u.binop.left); + gc_mark_select_tree(node->u.binop.right); break; } } @@ -1200,7 +1243,7 @@ xlist_t* list = &resreq->u.compound->primaries; for (int i = 0; i < xlist_length(list); i++) { xce_primary_resreq_t* p = (xce_primary_resreq_t*)xlist_get(list, i); if (p && p->select && p->select->root) - gc_mark_tree(p->select->root); + gc_mark_select_tree(p->select->root); } ``` @@ -1904,9 +1947,9 @@ eval) or data races (concurrent makeSelectNode + eval without the intern table lock). Phase 6 must enforce this with a runtime thread-ID check in `eval_select` and `gc_sweep_select_nodes`. -**GC single-thread guarantee:** `gc_mark_tree` writes the non-atomic -`gc_mark` field on interned nodes. This is safe because `gc_sweep_select_nodes` -is called exclusively from the allocator thread (a single-threaded context +**GC single-thread guarantee:** `gc_mark_select_tree` writes the non-atomic +`gc_mark` field on interned nodes. This is safe because the mark phase and +`gc_sweep_select_nodes` are called exclusively from the allocator thread (a single-threaded context — see §8.5.3 pre-session lifecycle). All `gc_mark` writes from a single GC sweep are sequential; no second thread accesses `gc_mark` concurrently. If the single-thread guarantee is ever relaxed, `gc_mark` must be made @@ -2200,15 +2243,19 @@ memory savings shown in §8.5.5. │ atomic_thread_fence(memory_order_seq_cst) must be inserted after │ │ all `primary->select->root = NULL` writes and before gc_sweep. │ │ Without the fence, the CPU or compiler may reorder the NULL store │ -│ after the GC mark pass begins, causing gc_mark_tree to observe a │ -│ stale non-NULL root and retain a node that should be swept. │ +│ after the GC mark pass begins, causing gc_mark_select_tree to │ +│ observe a stale non-NULL root and retain a node that should be │ +│ swept. │ │ In C11: atomic_thread_fence(memory_order_seq_cst); │ │ Equivalent: make `select->root` an _Atomic pointer and use │ │ atomic_store_explicit(..., memory_order_release) for the unlink. │ -│ 4. Run gc_sweep_select_nodes(live_jobs, njobs) — mark pass visits │ -│ all live jobs including newly arrived ones, so nodes referenced │ -│ only by incoming jobs are correctly retained. │ -│ 5. Run recompact_node_ids() — reassign dense 0..N-1 IDs │ +│ 4. gc_clear_marks(); gc_mark_select_tree(root) per live job's select │ +│ root(s) — mark pass visits all live jobs including newly arrived │ +│ ones, so nodes referenced only by incoming jobs are correctly │ +│ retained. (Caller-driven mark phase — see §5.5.) │ +│ 5. Run gc_sweep_select_nodes(NULL, 0) — sweep only; mark already │ +│ done in step 4 above (live_jobs/njobs args unused — see §5.5). │ +│ 6. Run recompact_node_ids() — reassign dense 0..N-1 IDs │ │ After recompact: current_node_count = sel_node_next_id │ │ = count of live interned nodes with dense IDs 0..N-1. │ └───────────────────────────────────────────────────────────────────────┘ @@ -2542,7 +2589,8 @@ are made. scheduler wiring. In Phase 1, these predicates are backed by a **test-only `fake_scheduler_t` struct** (declared in `test_select.c`) with the same three `_Atomic int` fields that Phase 6 will add to the - real `xce_scheduler_t` (see §5.5 "Predicate lifecycle" and Phase 6 + real scheduler core struct (as implemented: `struct xce_sched_core_` + in `core.h` — see §5.5 "Predicate lifecycle" and Phase 6 "Add GC predicate fields"). The Phase 1 stub functions wrap the fake struct; Phase 6 replaces them with wrappers around the real struct. No Phase 1 stub code leaks into the production implementation — the @@ -2567,14 +2615,20 @@ The grammar is the test oracle for all parse-level assertions.* - Fill `struct xce_resreq_select_` with `xce_select_node_t* root`. - Declare `makeSelectNode`, `eval_select`, `eval_select_inner`, `select_eval_ctx_init`, `select_eval_ctx_reset`, `select_eval_ctx_fini`, - `sel_node_table_init`, `sel_node_table_fini`, `gc_sweep_select_nodes`, - `recompact_node_ids`. - (`gc_mark_tree` is a `static` helper in `resreq.c` — §5.5 — and must - NOT be declared in the public header.) + `sel_node_table_init`, `sel_node_table_fini`, `gc_clear_marks`, + `gc_mark_select_tree`, `gc_sweep_select_nodes`, `recompact_node_ids`. + (As implemented, `gc_clear_marks` and `gc_mark_select_tree` ARE + declared in the public header and exported from `xce_select.c` — + not `static` helpers local to `resreq.c` — because the mark phase, + including the resreq->tag dispatch, is the caller's (scheduler's) + responsibility; see §5.5.) - `gc_sweep_select_nodes` signature: `void gc_sweep_select_nodes(const xce_sched_job_t** live_jobs, size_t njobs);` (`xce_sched_job_t` has `xce_resreq_t* resreq`; `xce_job_t` has only - `char* resreq` and cannot reach the parsed AST.) + `char* resreq` and cannot reach the parsed AST. As implemented, the + `live_jobs`/`njobs` parameters are unused inside the function — every + call site passes `NULL, 0` after running the mark phase separately + via `gc_clear_marks`/`gc_mark_select_tree`; see §5.5.) Caller: scheduler pre-session, after staging merge. Called after all job cleanup (select root unlinks) and before `recompact_node_ids`. Precondition: no `eval_select` call is in progress (GC idle invariant, §5.5). @@ -2935,8 +2989,11 @@ allocations — but these must be found and updated. `host_get_string` accessors. **4f. Mark-and-sweep GC and recompaction (§5.5/§8.5.4)** - - `gc_sweep_select_nodes()` with resreq->tag dispatch for COMPOUND/ALTER. - - `gc_mark_tree()` helper. + - `gc_clear_marks()` and `gc_mark_select_tree()` — mark phase; the + caller performs the resreq->tag dispatch for COMPOUND/ALTER and + invokes `gc_mark_select_tree()` per reachable select root. + - `gc_sweep_select_nodes()` — sweep-only; the caller must run the mark + phase above first (see §5.5). - `recompact_node_ids()` — reassign dense 0..N-1 IDs after GC (§8.5.4). #### Phase 5: Build wiring @@ -3173,27 +3230,34 @@ wires the select evaluator into the scheduler's existing session loop: **GC phase (§8.5.3 "Staging + GC phase"):** 1. Merge new jobs from staging buffer, parse their resreq strings (calling `makeSelectNode`), and unlink completed jobs. - 2. `gc_sweep_select_nodes(live_jobs, njobs)` — reclaim unreachable nodes. - Running GC after the staging merge ensures nodes referenced only by - newly arrived jobs are marked live and retained. + 2. `gc_clear_marks()` then `gc_mark_select_tree(root)` per live job's + select root(s) (dispatching on `resreq->tag`), then + `gc_sweep_select_nodes(NULL, 0)` — reclaim unreachable nodes. As + implemented, marking is the caller's responsibility and is a + separate step from the sweep call — see §5.5. Running GC after the + staging merge ensures nodes referenced only by newly arrived jobs + are marked live and retained. 3. `recompact_node_ids()` — reassign dense 0..N-1 IDs (§8.5.4). **Pre-session capacity check (§8.5.3 "Pre-session capacity check"):** 4. Check whether eval context arrays need to grow or shrink to accommodate the new `max_node_id` (§8.5.3 step 1). -- **Add GC predicate fields to `xce_scheduler_t`** in - `src/scheduler/include/scheduler.h`: add `_Atomic int session_active`, - `_Atomic int eval_in_progress`, and `_Atomic int staging_merge_complete` - to the struct. These back the predicate functions below; the struct - declaration must be updated before the functions can reference them. - Use `_Atomic int` (not `volatile int`) to match the `_Atomic uint32_t - sel_node_next_id` pattern (§5.3) and to provide correct atomicity on ARM - where `volatile` alone does not guarantee atomic reads/writes. +- **Add GC predicate fields to the scheduler's core struct**: add + `_Atomic int session_active`, `_Atomic int eval_in_progress`, and + `_Atomic int staging_merge_complete`. These back the predicate + functions below; the struct declaration must be updated before the + functions can reference them. Use `_Atomic int` (not `volatile int`) + to match the `_Atomic uint32_t sel_node_next_id` pattern (§5.3) and to + provide correct atomicity on ARM where `volatile` alone does not + guarantee atomic reads/writes. **As implemented,** these fields live + in `struct xce_sched_core_` (`src/scheduler/include/core.h`), not + `struct xce_scheduler_` (`scheduler.h`) — see §5.5. - **GC predicate implementation:** The `sched_session_active`, `sched_eval_in_progress`, and `staging_merge_complete` predicates - (§5.5) wrap the new fields above added to `xce_scheduler_t` in Phase 6. + (§5.5) wrap the new fields above added to the scheduler's core struct + in Phase 6. Lifecycle of each flag (all use `_Atomic int` — use `atomic_store_explicit(..., memory_order_release)` for writes and @@ -3238,11 +3302,16 @@ wires the select evaluator into the scheduler's existing session loop: requires runtime guards for safety-critical invariants — `assert()` is compiled out with `NDEBUG` and must not be relied upon here. -- **live_jobs materialization:** `gc_sweep_select_nodes` takes a flat - `const xce_sched_job_t**` array, but the scheduler stores jobs in - `sched->jobs` as a `var`/kvmgr keyed by jobid. Phase 6 must - materialize this array using a plain heap-allocated C array to avoid - the fragile cast assumption of `(const xce_sched_job_t**)zarray_t.data`: +- **live_jobs materialization:** The mark phase (`gc_clear_marks()` then + `gc_mark_select_tree()` per live job's select root(s), dispatching on + `resreq->tag` — see §5.5) needs a flat list of live jobs, but the + scheduler stores jobs in `sched->jobs` as a `var`/kvmgr keyed by jobid. + Phase 6 must materialize this array using a plain heap-allocated C + array to avoid the fragile cast assumption of + `(const xce_sched_job_t**)zarray_t.data`. `gc_sweep_select_nodes` + itself is called afterward as `gc_sweep_select_nodes(NULL, 0)` — it + does not take the array (see §5.5); the array only feeds the mark + phase: ```c size_t live_cap = 64, n = 0; @@ -3263,7 +3332,12 @@ wires the select evaluator into the scheduler's existing session loop: } live_arr[n++] = job; } - gc_sweep_select_nodes(live_arr, n); + gc_clear_marks(); + for (size_t i = 0; i < n; i++) { + /* dispatch on live_arr[i]->resreq->tag and call gc_mark_select_tree() + * on each reachable select root — see §5.5 */ + } + gc_sweep_select_nodes(NULL, 0); free(live_arr); ``` diff --git a/src/docs/xce.md b/src/docs/xce.md index d70d1027..42316679 100644 --- a/src/docs/xce.md +++ b/src/docs/xce.md @@ -9,11 +9,13 @@ available: - **scheduler** — start xce scheduler daemon - **apiserver** — start xce REST API server - **shell** — interactive bash shell for troubleshooting -- **xcecli** — CLI tool for cluster interaction (job submit, query, admin) If you run xce without any parameter or with a parameter not listed above, it will show this help screen. +Note: `xcecli` is not a sub-command of `xce` — it is a separate standalone +Go binary invoked directly. See the [xcecli](#xcecli) section below. + ## controller Start the xce controller daemon with the given parameters after the @@ -21,7 +23,7 @@ Start the xce controller daemon with the given parameters after the Example: ``` -xce controller --config /etc/xce/controller.yaml +xce controller --workdir /var/lib/xce/controller --myname ctrl01 --peers ctrl01:9990,ctrl02:9990 ``` ## scheduler @@ -45,9 +47,9 @@ authentication and RBAC authorization. The xce CLI (`src/frontend/xcecli/`) provides command-line access to the cluster. Common operations: -- `xcecli job submit ` — submit a job -- `xcecli job list` — list jobs -- `xcecli queue status` — show queue state +- `xcecli jobs submit ` — submit a job +- `xcecli jobs list` — list jobs +- `xcecli queues list` — show queue state Built as a standalone Go binary; communicates with the API server over its REST interface. diff --git a/src/lib/CLAUDE.md b/src/lib/CLAUDE.md index c4905083..fe032d24 100644 --- a/src/lib/CLAUDE.md +++ b/src/lib/CLAUDE.md @@ -1,29 +1,28 @@ # XCE Shared Library (libxce) ## Overview -Shared C library providing common utilities, data structures, and IPC primitives used by the controller, worker, and scheduler daemons. +Shared C library providing common utilities, data structures, and IPC primitives used by the controller, worker, and scheduler daemons, plus the Go apiserver (via CGO, for IPC with the controller). ## Build ```bash -# From workspace root -nix develop -meson build && ninja -C build +/xce-build lib # debug + ASAN + clean build + runs tests ``` +For full build system details, see `.kiro/steering/build-deployment.md`. ## Architecture -- Produces `libxce.a` (static) and/or `libxce.so` (shared) +- Produces `libxce.a` (static only — the `shared_library('xce', ...)` target in `meson.build` is currently commented out; no `.so` is built) - Linked by all C system components: controller, worker, scheduler -- Public headers exposed via `src/include/xce/` +- Public headers consumed by all components live flat in `src/include/` (e.g. `client.h`, `netio.h`, `request.h`, `vtaskmgr.h`, `persist.h`); only a handful of headers (`client_failover.h`, `cluster.h`, `cluster_def.h`, `errnox.h`, `types.h`) live under `src/include/xce/` - Test suite in `src/lib/testsuite/` ## Conventions - C standard: gnu17 - Format with `clang-format` (config at repo root if present) -- All public API headers go in `src/include/xce/` — not in `src/lib/` +- Public API headers go in `src/include/` (most headers) or `src/include/xce/` (a small legacy subset) — not in `src/lib/` - Implementation files stay in `src/lib/` ## Key Files -- `src/include/xce/` — Public header files consumed by all components +- `src/include/` — Public header files consumed by all components (`src/include/xce/` holds a small legacy subset) - `src/lib/testsuite/` — Library unit tests ## See Also diff --git a/src/scheduler/CLAUDE.md b/src/scheduler/CLAUDE.md index fd2a26e1..a0cd3038 100644 --- a/src/scheduler/CLAUDE.md +++ b/src/scheduler/CLAUDE.md @@ -10,10 +10,10 @@ C-based job scheduling engine with a plugin architecture. Determines job-to-node ``` ## Architecture -- **Core** (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c`) — passive, event-driven `xce_sched_core_t` state machine. Zero controller I/O (no `netio`/`xceapi`/`client`/`delay`/`idgen` — enforced by `test_core_isolation.c`); `core_clock.c` is the one sanctioned exception (`clock_gettime`, virtualized two-base+speed clock formula, `xce_core_now()`). Picker→allocator handoff is a strict `jbuf` cbuf capacity-1 rendezvous — this is the project's determinism invariant, not a tuning knob; never raise it without re-deriving Req 5 AC 7 in `.kiro/specs/scheduler-core/requirements.md`. +- **Core** (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c`) — passive, event-driven `xce_sched_core_t` state machine. Zero controller I/O (no `netio`/`xceapi`/`client`/`delay`/`idgen`); `core_clock.c` is the one sanctioned exception (`clock_gettime`, virtualized two-base+speed clock formula, `xce_core_now()`). `test_core_isolation.c` mechanically enforces this only for `core.c`/`core.h` (source-text scan + `nm(1)` undefined-symbol check on the isolated `sched_core_isolation_lib` archive, meson.build) — `core_events.c`, `core_session.c`, and `core_clock.c` are not covered by that test and must be kept isolation-clean by review. Picker→allocator handoff is a strict `jbuf` cbuf capacity-1 rendezvous — this is the project's determinism invariant, not a tuning knob; never raise it without re-deriving Req 5 AC 7 in `.kiro/specs/scheduler-core/requirements.md`. - **Picker plugin interface** (`plugin.h`'s `xce_job_picker_iface_t`, `plugins/builtin_fcfs.c`) — job-selection strategies are pluggable; `scheduler.c`'s `sched_job_picker()` branches on whether a plugin still uses the legacy `job_elect_fn`-only shape (`plugins/fcfs.c`, kept as-is for a compat-wrapper exercise, not the live default) vs the native hook interface (`builtin_fcfs`, the registered default). New scheduling policies → add as a picker plugin under `src/plugins/`, do not modify the core loop. - **Drivers** — `driver.c` (normal mode: capnp↔core event translation, decision publishing, deferred admin-request handling) and `sim.c`/`sim_loop.c`/`sim_record.c`/`sim_verify.c` (simulation mode: `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`). Both drive the identical core session/decision-callback protocol — one core, two consumers. -- **Replay/recording** — `recording.c`/`recording_events.c`/`recording_session.c` (live capture) and `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` (state save/restore) read/write the `.replay` capnp binary stream (`src/schema/controller.capnp`); controlled via `OPCode_schedAdmin` (`src/controller/cluster.c`) and `xceadmin scheduler record start|stop|status` / `scheduler dump `. +- **Replay/recording** — `recording.c`/`recording_events.c`/`recording_session.c` (live capture) and `sim.c`/`sim_loop.c` (simulation-side decode) do the actual capnp encode/decode of the `.replay` binary stream (`src/schema/controller.capnp`); `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` only populate/consume the plain in-memory `sched_snapshot_data_t` struct that `recording.c`'s `write_snapshot_entry()` then serializes — no capnp or file I/O lives in the snapshot files themselves. Controlled via `OPCode_schedAdmin` (`src/controller/cluster.c`) and `xceadmin scheduler record start|stop|status` / `scheduler dump `. - **Interactive REPL** (`repl.c`/`repl_break.c`/`repl_info.c`/`repl_params.c`, `--interactive`) — single-step/breakpoint/state-inspection/snapshot-capture control over the simulation driver; drives the exact same `sched_sim_driver_run()` as batch `--sim`, gated only by `sim->interactive`, so REPL and batch runs are decision-equivalent by construction (`test_repl_equivalence.c`). Links against `libxce` from `src/lib/`. Full design/requirements: `.kiro/specs/scheduler-core/`. @@ -23,7 +23,7 @@ Links against `libxce` from `src/lib/`. Full design/requirements: `.kiro/specs/s - Format with `clang-format` - New scheduling policies → add as a plugin under `src/plugins/`, do not modify the core engine - Shared types defined in `src/include/xce/` (libxce) -- Files stay under the repo's 800-line cap — several core/sim/repl/snapshot/recording modules are deliberately split across multiple `*_events.c`/`*_session.c`/`*_wire.c`-style files for this reason; keep new logic in the file matching its concern rather than growing one file past the cap. +- Files stay under the repo's 800-line cap — several core/sim/repl/snapshot/recording modules are deliberately split across multiple `*_events.c`/`*_session.c`/`*_wire.c`-style files for this reason; keep new logic in the file matching its concern rather than growing one file past the cap. Not yet true project-wide: `scheduler.c` (1382 lines) and `plugins/builtin_fcfs.c` (1026 lines) are pre-existing exceptions over the cap with no split or exemption note — do not use them as precedent for new files, and prefer splitting them opportunistically rather than growing them further. ## See Also - Shared library: `src/lib/CLAUDE.md` diff --git a/src/worker/CLAUDE.md b/src/worker/CLAUDE.md index 7d4d4790..d80be6de 100644 --- a/src/worker/CLAUDE.md +++ b/src/worker/CLAUDE.md @@ -5,15 +5,13 @@ C daemon running on each compute node. Manages job execution, hardware monitorin ## Build ```bash -# From workspace root -nix develop -meson build && ninja -C build +/xce-build worker # lib + worker daemon (see build-deployment.md) ``` ## Architecture -- `src/xcecd.c` — Main daemon entry point -- `src/hardware_monitor.c` — CPU/memory/GPU resource monitoring -- `src/process_manager.c` — Job process lifecycle (spawn, track, reap) +- `src/main.c` — Main daemon entry point (`int main`); links against `xcecd_srcs` from `src/xcecd.c` and friends +- `src/hardware_monitor.c` — CPU/memory resource monitoring (no GPU) +- `src/process_worker_manager.c` + `src/uv_proc.c` — Job process lifecycle (fork `xcecd_worker`, spawn/exec the job command); `src/process_manager.c` is unused dead code for job spawn (its `process_manager_start_process` is never called) but is still wired up for generic create/update/destroy/reap in `xcecd.c` Links against `libxce` from `src/lib/`.