diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 65540cf2..c0ba7170 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -37,12 +37,15 @@ disown`) before relying on subagent dispatch for unattended use. `cluster.bin` from `src/configs/*.yaml` fresh on every deploy - `openssl` must be on `PATH` (provided by `nix develop`) — `deploy-ha.sh` generates a fresh self-signed TLS cert/key for every deploy -- Optional: run `cd src/frontend && make build-server` first if you want an - apiserver step (Step 4d) — a separate Go build system from the meson build - controller/worker/scheduler share, producing `/tmp/apiserver`. - `deploy-ha.sh` never runs this itself; without it, Step 3 simply skips - generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as - the existing worker/scheduler convention. +- Optional: run `/xce-build apiserver` first if you want an apiserver step + (Step 4d) — the same meson build controller/worker/scheduler share, + producing `/frontend/apiserver/xce_apiserver`. + `deploy-ha.sh` never runs this itself. As a legacy fallback, a binary + manually placed at `/tmp/apiserver` is also picked up if the build-dir + path is absent (not guaranteed to be the same build generation as + `$BINARY`). Without either, Step 3 simply skips generating + `scripts/start-apiserver.sh` (soft-skip, not an error), same as the + existing worker/scheduler convention. - Run `/xce-build worker` so both worker binaries exist in the same build dir as `$BINARY`: `xcecd_virtual` (default) and `xcecd` (the real, root-gated production daemon). Step 3 auto-picks which one @@ -162,10 +165,12 @@ This: convention. See Step 4c. - Also generates `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}` and `$TOP/scripts/start-apiserver.sh` **if** a pre-built apiserver binary - exists at `/tmp/apiserver` (`make build-server`'s default output path — - **not** alongside `$BINARY`, since the apiserver is a separate Go build, - not part of this meson build dir) — same soft-skip-with-warning - convention as worker/scheduler. See Step 4d. + exists — checked in order: first `/frontend/apiserver/xce_apiserver` + (same build dir as `$BINARY`, produced by `/xce-build apiserver`), then the + legacy fixed path `/tmp/apiserver` if the build-dir path is absent (used + as-is, with a logged caveat that it's not guaranteed to be the same build + generation as `$BINARY`) — same soft-skip-with-warning convention as + worker/scheduler if neither exists. See Step 4d. - Also **always** generates `$TOP/scripts/start-all.sh`, a combined shortcut for Steps 4/4b/4c — see below. @@ -291,9 +296,12 @@ a warning if no wheel exists in the same build dir as `$BINARY`, same as ## Step 4d — Start the apiserver (optional) `deploy-ha.sh` also generates `$TOP/scripts/start-apiserver.sh` **if** a -pre-built apiserver binary exists at `/tmp/apiserver` (soft-skipped with a -warning, not a hard error, if missing — see Prerequisites). Start it once at -least one controller is up, the same way as the worker/scheduler: +pre-built apiserver binary exists — preferring +`/frontend/apiserver/xce_apiserver` (built via `/xce-build +apiserver`), falling back to a binary manually placed at the legacy +`/tmp/apiserver` path if the build-dir one is absent — soft-skipped with a +warning, not a hard error, if neither exists (see Prerequisites). Start it +once at least one controller is up, the same way as the worker/scheduler: ```bash "$BASH_BIN" "$TOP/scripts/start-apiserver.sh" & @@ -450,7 +458,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins | A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy | | `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER=""` before the `exec` line | | `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) | -| `scripts/start-apiserver.sh` not generated | No binary at `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) — this is a separate Go build, not part of `$BINARY`'s meson build dir | `cd src/frontend && make build-server`, then redeploy (Step 3) | +| `scripts/start-apiserver.sh` not generated | No binary at `/frontend/apiserver/xce_apiserver` or `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build apiserver`, then redeploy (Step 3) — or place a binary at `/tmp/apiserver` (legacy path, not guaranteed to be the same build generation) | | Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template | | Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first | | `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists | diff --git a/.claude/skills/xce-test-resources/SKILL.md b/.claude/skills/xce-test-resources/SKILL.md index 3f448cbe..94e9f8f3 100644 --- a/.claude/skills/xce-test-resources/SKILL.md +++ b/.claude/skills/xce-test-resources/SKILL.md @@ -105,7 +105,8 @@ script.) processes afterward. Tear the deployment down manually afterward (its PIDs are visible via `ps`/`pgrep` scoped to `DIR`). - Other overridable env vars (all optional, default to this repo's own - meson build dir / `/tmp/apiserver`): `$XCE_RESVERIFY_BUILD_DIR`, + meson build dir, e.g. `$BUILD_DIR/frontend/apiserver/xce_apiserver` for the + apiserver binary): `$XCE_RESVERIFY_BUILD_DIR`, `$XCE_RESVERIFY_BIN` (controller), `$XCE_RESVERIFY_SCHED_BIN`, `$XCE_RESVERIFY_APISERVER_BIN`, `$XCE_RESVERIFY_XT` (the `xce_test` binary), `$XCE_RESVERIFY_PEERS` (defaults to a single-node diff --git a/.kiro/specs/eliminate-frontend-makefile/tasks.md b/.kiro/specs/eliminate-frontend-makefile/tasks.md index 248acfb7..88768d90 100644 --- a/.kiro/specs/eliminate-frontend-makefile/tasks.md +++ b/.kiro/specs/eliminate-frontend-makefile/tasks.md @@ -37,7 +37,7 @@ - _Boundary: deploy-ha.sh, tests/regression/run.sh_ - _Depends: 1.1, 2_ -- [ ] 3.3 (P) Update live steering, skill, and status documentation +- [x] 3.3 (P) Update live steering, skill, and status documentation - Update every current-fact reference to a `src/frontend/Makefile` target or a Makefile-specific fixed path in: `tech.md` (Build Systems table and code block, plus the "Testing Requirements" lines), `build-deployment.md` (Go Frontend Build section), `testing.md` (Build targets bullets, Philosophy line, and the "What Not to Mock" CGO-bridge line), `structure.md`, `frontend/CLAUDE.md`, root `CLAUDE.md`'s Key Files entry, `src/docs/deployment.md`'s Build gotcha section, `xce-deploy-ha/SKILL.md` (all four references, including the Step 4d intro), `xce-test-resources/SKILL.md`'s `$XCE_RESVERIFY_APISERVER_BIN` default description, `src/frontend/docs/status.md`, and `src/docs/apiserver-backend-tasks.md` - Replace each with the equivalent Meson-based guidance (`/xce-build apiserver`, `meson test --suite apiserver`, direct `tests/regression/run.sh` invocation) already established by tasks 1-3.2 - Observable: none of the 11 files contains a live `make (build-server|build-cli|test|test-backend|test-integration|test-all)` or `/tmp/apiserver`/`/tmp/xce_static` reference presented as current fact @@ -83,5 +83,6 @@ ## Implementation Notes +- Task 3.3 (round 1 rejection, fixed in round 2): when removing a stale Makefile reference from a doc, check whether the underlying script it describes still has a genuine fallback behavior before deleting the mention entirely. `run-resource-verify.sh` (task 3.1) removed its `/tmp/apiserver` fallback outright, so `xce-test-resources/SKILL.md` correctly dropped all mention of it. `deploy-ha.sh` (task 3.2) deliberately KEPT its `/tmp/apiserver` fallback logic (only removed the `make build-server` association from its text) — so `xce-deploy-ha/SKILL.md` needed to keep describing that fallback accurately (just without the Make instruction), not delete it. Two visually-similar edits, two different correct outcomes, because the two underlying scripts' behaviors diverged. - Task 2: `/xce-build all` currently aborts (`set -euo pipefail`) at the pre-existing `meson test --print-errorlogs` step, BEFORE reaching the migrated Go-test line — two unrelated, already-logged C-side failures (bug-1723, LeakSanitizer on `xcelib_test`; bug-1758, `sched_core_events_test` deterministic failure), both predating this spec. Task 2's `all`-case line itself is verified (byte-identical to the fully-verified `apiserver`-case line), but this means task 4.1's "final go/no-go gate" and task 4.2's post-deletion re-validation will hit the SAME pre-existing abort when running `/xce-build all` — those tasks will need to either scope their validation to `/xce-build apiserver` + a separate `meson test` invocation that tolerates these 2 known-unrelated failures, or note them as pre-existing/out-of-scope rather than treating them as a regression from this spec. - Task 1.2: design.md's Testing Strategy suggests `librtti_dep` as the example dependency to remove when proving the new backend test() exercises the real CGO path. Empirically (implementer + independent reviewer both reproduced this live), removing `librtti_dep` does NOT reproduce a link failure for the `apiserver/backend/...` test package — none of `librtti`'s exported symbols are actually pulled into this specific link unit's undefined-symbol set. `c_capnproto_dep` (real `capn_*` undefined-reference errors from `request.c`/`controller.capnp.c`) is the dependency that genuinely reproduces the failure and was used instead. `apiserver_cgo_deps` itself was left byte-identical to task 1.1's state (verification-only task, no permanent code change). diff --git a/.kiro/steering/build-deployment.md b/.kiro/steering/build-deployment.md index beac0b90..24fde466 100644 --- a/.kiro/steering/build-deployment.md +++ b/.kiro/steering/build-deployment.md @@ -137,14 +137,13 @@ kill -9 $(pgrep xce_scheduler) 2>/dev/null || true ## Go Frontend Build -Go builds use `src/frontend/Makefile` for development, Meson for production: +Go builds use Meson for both development and production — apiserver and xcecli have no separate build system: ```bash -make build-server # → /tmp/apiserver -make build-cli # → /tmp/xce_static (CGO_ENABLED=0) -make test # unit tests (CGO_ENABLED=0) -make test-backend # CGO-backed backend tests (needs nix dev shell) -make test-integration # full regression suite (needs XCE_BIN + SRV_URL) +/xce-build apiserver # → /frontend/apiserver/xce_apiserver, xcecli; runs unit tests (CGO_ENABLED=0) +meson test -C --suite apiserver # CGO-backed backend tests (needs nix dev shell) +XCE=/frontend/xcecli/xcecli URL= \ + src/frontend/tests/regression/run.sh # full regression suite ``` Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`. diff --git a/.kiro/steering/structure.md b/.kiro/steering/structure.md index 535534b5..c8a19dd7 100644 --- a/.kiro/steering/structure.md +++ b/.kiro/steering/structure.md @@ -95,7 +95,7 @@ Plugin-based components add their own subdirectory: **Go tests** (co-located `_test.go`): - Table-driven unit tests, `t.TempDir()` for fixtures - Global state reset helpers (e.g., `resetRBACState()`) for test isolation -- `make test` = unit only; `make test-all` = unit + integration +- `/xce-build apiserver` = unit tests only; add `meson test -C --suite apiserver` (backend) and `tests/regression/run.sh` (integration) for full coverage ## Code Organization Principles diff --git a/.kiro/steering/tech.md b/.kiro/steering/tech.md index b66a35aa..793389b8 100644 --- a/.kiro/steering/tech.md +++ b/.kiro/steering/tech.md @@ -27,19 +27,18 @@ All controller state is in-memory. No persistent database in the API tier; job/q | System | Scope | |--------|-------| -| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper | -| **Make** | Go development loop (build/test) from `src/frontend/` | +| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper — sole build/test path for apiserver/xcecli | | **Nix flakes** | Hermetic dev environment, AppImage packaging | ```bash # Full build (C + Go, debug + ASAN) /xce-build -# Go development loop (from src/frontend/) -make build-server # → /tmp/apiserver -make build-cli # → /tmp/xce_static (CGO_ENABLED=0) -make test # unit tests -make test-integration # regression suite (needs XCE_BIN + SRV_URL) +# Go development loop (apiserver + xcecli) +/xce-build apiserver # builds both binaries + runs unit tests +meson test -C --suite apiserver # CGO-backed backend tests +XCE=/frontend/xcecli/xcecli URL=http://localhost:8080 \ + src/frontend/tests/regression/run.sh # regression suite # Nix package builds (from src/) nix build .#xcecore # C daemons + apiserver @@ -78,8 +77,8 @@ Early return on every error. Log with zap fields before returning. - **TDD mandatory**: Write failing test first, then implementation. - **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures. - **C tests**: Add test case in `testsuite/` before modifying implementation. -- `make test` must pass without a live server. -- `make test-integration` requires `XCE_BIN` + `SRV_URL` env vars. +- `/xce-build apiserver`'s unit-test step must pass without a live server. +- `tests/regression/run.sh` requires `XCE` + `URL` env vars. ### Code Formatting diff --git a/.kiro/steering/testing.md b/.kiro/steering/testing.md index 0a2fed27..05d71b4c 100644 --- a/.kiro/steering/testing.md +++ b/.kiro/steering/testing.md @@ -5,7 +5,7 @@ - **TDD mandatory**: Write a failing test first, then the minimal implementation. The cycle is Red → Green → Refactor. - Test behavior, not implementation details. - Tests must run without a live server or external services. -- A change is not complete until `make test` (and `make test-integration` where applicable) passes. +- A change is not complete until `/xce-build apiserver`'s unit-test step (and `tests/regression/run.sh` where applicable) passes. ## Go Tests (apiserver / xcecli) @@ -44,9 +44,9 @@ func TestPermissions(t *testing.T) { **Fixtures**: use `t.TempDir()` for file-system state; never rely on absolute paths or external services. **Build targets**: -- `make test` — unit tests only (`CGO_ENABLED=0`) -- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir) -- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars) +- `/xce-build apiserver` — unit tests only (`CGO_ENABLED=0`), plus the build +- `meson test -C --suite apiserver` — CGO-backed backend tests (requires nix dev shell + build dir) +- `tests/regression/run.sh` — full regression suite (requires `XCE` and `URL` env vars, run directly) ## C Tests (controller / scheduler / lib) @@ -92,4 +92,4 @@ extern void reset_ha_state(ha_state_t *s); - Mock **external dependencies only** (network calls, file I/O that crosses component boundaries) - Do **not** mock the system under test - In-memory stores (`store/`) are real — use them directly in unit tests -- CGO bridge (`backend/`) is covered by `make test-backend` — do not stub it in unit tests +- CGO bridge (`backend/`) is covered by `meson test -C --suite apiserver` — do not stub it in unit tests diff --git a/CLAUDE.md b/CLAUDE.md index 01456675..c785eb36 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s - **Save/restore field-completeness**: any persistence mechanism (WAL fact, scheduler snapshot, checkpoint) must capture every field of the entity it claims to persist — a state that "recovers correctly" while silently missing a field it depends on (e.g. `job.state` correct but `job.allocs` lost) is a bug, not an acceptable simplification, unless the omission is explicitly documented as a scope decision (see `persistence.md`'s "Snapshot Save/Load Field-Completeness" + `APPLY_CLEAN`/bug-1678 sections). When adding or extending save/restore logic, verify the field round-trips through BOTH the write side and every read/apply seam, and run `/xce-audit-invariants` (controller dimension 7 / scheduler dimension 4) rather than assuming a passing test suite already proves it — `APPLY_CLEAN`-style verdicts check row counts and cursor position, not per-field completeness ## Key Files -- `src/frontend/Makefile` — primary build/test orchestration +- `src/tools/xce-build.sh` — primary build/test orchestration (Meson-based; see `build-deployment.md`) - `src/frontend/apiserver/cmd/main.go` — server startup, init sequence - `src/frontend/apiserver/router/routers.go` — all routes and middleware chain - `src/frontend/apiserver/controllers/rbac_check.go` — RBAC loading and enforcement diff --git a/src/docs/apiserver-backend-tasks.md b/src/docs/apiserver-backend-tasks.md index de395a44..0fa692a7 100644 --- a/src/docs/apiserver-backend-tasks.md +++ b/src/docs/apiserver-backend-tasks.md @@ -67,7 +67,7 @@ - `controllers/queues.go` — `preCheckQueueControlRequest` now returns `(bool, error)`; `ControlQueue` receives allFlag locally - `controllers/host_control_test.go` — 5 new tests: allFlag true/false, concurrent no-race (50 goroutines), invalid action, comment-too-long - `controllers/queue_control_test.go` — 5 new tests: same coverage for queues - - All tests passing (`make test`) + - All tests passing (via `/xce-build apiserver`'s test step) - [x] **#8 P0-2c: Register routes + RBAC for host control/select** - `router/routers.go` — `POST /xce/v1/hosts/control` → RBAC `host:control` (cluster_admin) @@ -117,7 +117,7 @@ ## Final - [x] **#14 Build and test verification** - - `make test` in `src/frontend/` passes — all backend, controllers, store, utils, xcecli tests green + - `/xce-build apiserver`'s test step passes — all backend, controllers, store, utils, xcecli tests green - `ninja -C build` for C components passes — required meson reconfigure + CGO val_helpers.h fix - RBAC verification — `host:control` (cluster_admin) and `host:view` (all authenticated) confirmed in routes + rbac.yaml - Input safety — comment limit 512 chars; select expression limit 4094 chars (1024 reduction deferred to P1-2b) diff --git a/src/docs/deployment.md b/src/docs/deployment.md index bf6ac087..eb39f7c7 100644 --- a/src/docs/deployment.md +++ b/src/docs/deployment.md @@ -240,11 +240,11 @@ apiserver -d /opt/xce/apiserver Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fast on malformed config). -### Build gotcha — use the Meson/Nix build, not `make build-server` +### Build gotcha — apiserver needs real CGO flags, not a bare `go build` -`make build-server` (`src/frontend/Makefile`) runs a bare `go build` with **no `CGO_CFLAGS`/`CGO_LDFLAGS` set**. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time. +A bare `go build` in `src/frontend/apiserver` (no `CGO_CFLAGS`/`CGO_LDFLAGS` set) is not a supported way to produce a working apiserver binary. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time. -**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). Reserve `make build-server`/`make test` for the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there). +**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). `/xce-build apiserver` also runs the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there); a manual `go build`/`go test` invocation is not the canonical path. ### Relationship to the controller diff --git a/src/frontend/CLAUDE.md b/src/frontend/CLAUDE.md index 0fa0db13..7979ef81 100644 --- a/src/frontend/CLAUDE.md +++ b/src/frontend/CLAUDE.md @@ -11,10 +11,10 @@ Go-based REST API server (Gin) and CLI tool (Cobra) that expose cluster manageme ## Build & Test ```bash -make build-server # → /tmp/apiserver -make build-cli # → /tmp/xce_static (CGO_ENABLED=0, static binary) -make test # unit tests only (no live server required) -make test-all # unit + integration (needs XCE_BIN and SRV_URL env vars) +/xce-build apiserver # → /frontend/apiserver/xce_apiserver, xcecli; runs unit tests (no live server required) +meson test -C --suite apiserver # CGO-backed backend tests +XCE=/frontend/xcecli/xcecli URL= \ + tests/regression/run.sh # integration (needs XCE and URL env vars) ``` ## API Server Architecture diff --git a/src/frontend/docs/status.md b/src/frontend/docs/status.md index c7fa7976..6525588d 100644 --- a/src/frontend/docs/status.md +++ b/src/frontend/docs/status.md @@ -64,7 +64,7 @@ Live test: `xcecli` against running `apiserver` with builtin test data. ### Infrastructure - [x] Nix flake dev shell (`nix develop`) — Go 1.25, PAM, OpenSSL, meson, gnumake, swag, gopls, delve -- [x] Root `Makefile`: `make test`, `make test-integration`, `make test-all` +- [x] Meson build/test: `/xce-build apiserver` (unit tests), `meson test -C --suite apiserver` (CGO-backed backend tests), `tests/regression/run.sh` (integration) - [x] Regression suite: `tests/regression/run.sh` — 29 tests, all pass - [x] Unit tests across: `apiserver/store`, `apiserver/utils`, `apiserver/controllers`, `xcecli/cmd/auth`, `xcecli/cmd/jobs`, `xcecli/cmd/roles`, `xcecli/cmd/users`, `xcecli/common/request`, `xcecli/models`