feat(eliminate-frontend-makefile): task 3.3 - update live documentation (11 files)

Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.

Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
This commit is contained in:
Rongsong Shen 2026-09-03 14:22:54 +00:00
parent 483cac3dcb
commit f5fbc6bcac
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 56 additions and 48 deletions

View file

@ -37,12 +37,15 @@ disown`) before relying on subagent dispatch for unattended use.
`cluster.bin` from `src/configs/*.yaml` fresh on every deploy
- `openssl` must be on `PATH` (provided by `nix develop`) — `deploy-ha.sh`
generates a fresh self-signed TLS cert/key for every deploy
- Optional: run `cd src/frontend && make build-server` first if you want an
apiserver step (Step 4d) — a separate Go build system from the meson build
controller/worker/scheduler share, producing `/tmp/apiserver`.
`deploy-ha.sh` never runs this itself; without it, Step 3 simply skips
generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as
the existing worker/scheduler convention.
- Optional: run `/xce-build apiserver` first if you want an apiserver step
(Step 4d) — the same meson build controller/worker/scheduler share,
producing `<build dir>/frontend/apiserver/xce_apiserver`.
`deploy-ha.sh` never runs this itself. As a legacy fallback, a binary
manually placed at `/tmp/apiserver` is also picked up if the build-dir
path is absent (not guaranteed to be the same build generation as
`$BINARY`). Without either, Step 3 simply skips generating
`scripts/start-apiserver.sh` (soft-skip, not an error), same as the
existing worker/scheduler convention.
- Run `/xce-build worker` so both worker binaries exist in the same build
dir as `$BINARY`: `xcecd_virtual` (default) and `xcecd` (the real,
root-gated production daemon). Step 3 auto-picks which one
@ -162,10 +165,12 @@ This:
convention. See Step 4c.
- Also generates `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}`
and `$TOP/scripts/start-apiserver.sh` **if** a pre-built apiserver binary
exists at `/tmp/apiserver` (`make build-server`'s default output path —
**not** alongside `$BINARY`, since the apiserver is a separate Go build,
not part of this meson build dir) — same soft-skip-with-warning
convention as worker/scheduler. See Step 4d.
exists — checked in order: first `<build dir>/frontend/apiserver/xce_apiserver`
(same build dir as `$BINARY`, produced by `/xce-build apiserver`), then the
legacy fixed path `/tmp/apiserver` if the build-dir path is absent (used
as-is, with a logged caveat that it's not guaranteed to be the same build
generation as `$BINARY`) — same soft-skip-with-warning convention as
worker/scheduler if neither exists. See Step 4d.
- Also **always** generates `$TOP/scripts/start-all.sh`, a combined shortcut
for Steps 4/4b/4c — see below.
@ -291,9 +296,12 @@ a warning if no wheel exists in the same build dir as `$BINARY`, same as
## Step 4d — Start the apiserver (optional)
`deploy-ha.sh` also generates `$TOP/scripts/start-apiserver.sh` **if** a
pre-built apiserver binary exists at `/tmp/apiserver` (soft-skipped with a
warning, not a hard error, if missing — see Prerequisites). Start it once at
least one controller is up, the same way as the worker/scheduler:
pre-built apiserver binary exists — preferring
`<build dir>/frontend/apiserver/xce_apiserver` (built via `/xce-build
apiserver`), falling back to a binary manually placed at the legacy
`/tmp/apiserver` path if the build-dir one is absent — soft-skipped with a
warning, not a hard error, if neither exists (see Prerequisites). Start it
once at least one controller is up, the same way as the worker/scheduler:
```bash
"$BASH_BIN" "$TOP/scripts/start-apiserver.sh" &
@ -450,7 +458,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
| A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy |
| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER="<first --peers candidate>"` before the `exec` line |
| `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) |
| `scripts/start-apiserver.sh` not generated | No binary at `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) — this is a separate Go build, not part of `$BINARY`'s meson build dir | `cd src/frontend && make build-server`, then redeploy (Step 3) |
| `scripts/start-apiserver.sh` not generated | No binary at `<build dir>/frontend/apiserver/xce_apiserver` or `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build apiserver`, then redeploy (Step 3) — or place a binary at `/tmp/apiserver` (legacy path, not guaranteed to be the same build generation) |
| Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template |
| Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first |
| `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists |

View file

@ -105,7 +105,8 @@ script.)
processes afterward. Tear the deployment down manually afterward (its PIDs
are visible via `ps`/`pgrep` scoped to `DIR`).
- Other overridable env vars (all optional, default to this repo's own
meson build dir / `/tmp/apiserver`): `$XCE_RESVERIFY_BUILD_DIR`,
meson build dir, e.g. `$BUILD_DIR/frontend/apiserver/xce_apiserver` for the
apiserver binary): `$XCE_RESVERIFY_BUILD_DIR`,
`$XCE_RESVERIFY_BIN` (controller), `$XCE_RESVERIFY_SCHED_BIN`,
`$XCE_RESVERIFY_APISERVER_BIN`, `$XCE_RESVERIFY_XT` (the `xce_test`
binary), `$XCE_RESVERIFY_PEERS` (defaults to a single-node

View file

@ -37,7 +37,7 @@
- _Boundary: deploy-ha.sh, tests/regression/run.sh_
- _Depends: 1.1, 2_
- [ ] 3.3 (P) Update live steering, skill, and status documentation
- [x] 3.3 (P) Update live steering, skill, and status documentation
- Update every current-fact reference to a `src/frontend/Makefile` target or a Makefile-specific fixed path in: `tech.md` (Build Systems table and code block, plus the "Testing Requirements" lines), `build-deployment.md` (Go Frontend Build section), `testing.md` (Build targets bullets, Philosophy line, and the "What Not to Mock" CGO-bridge line), `structure.md`, `frontend/CLAUDE.md`, root `CLAUDE.md`'s Key Files entry, `src/docs/deployment.md`'s Build gotcha section, `xce-deploy-ha/SKILL.md` (all four references, including the Step 4d intro), `xce-test-resources/SKILL.md`'s `$XCE_RESVERIFY_APISERVER_BIN` default description, `src/frontend/docs/status.md`, and `src/docs/apiserver-backend-tasks.md`
- Replace each with the equivalent Meson-based guidance (`/xce-build apiserver`, `meson test --suite apiserver`, direct `tests/regression/run.sh` invocation) already established by tasks 1-3.2
- Observable: none of the 11 files contains a live `make (build-server|build-cli|test|test-backend|test-integration|test-all)` or `/tmp/apiserver`/`/tmp/xce_static` reference presented as current fact
@ -83,5 +83,6 @@
## Implementation Notes
- Task 3.3 (round 1 rejection, fixed in round 2): when removing a stale Makefile reference from a doc, check whether the underlying script it describes still has a genuine fallback behavior before deleting the mention entirely. `run-resource-verify.sh` (task 3.1) removed its `/tmp/apiserver` fallback outright, so `xce-test-resources/SKILL.md` correctly dropped all mention of it. `deploy-ha.sh` (task 3.2) deliberately KEPT its `/tmp/apiserver` fallback logic (only removed the `make build-server` association from its text) — so `xce-deploy-ha/SKILL.md` needed to keep describing that fallback accurately (just without the Make instruction), not delete it. Two visually-similar edits, two different correct outcomes, because the two underlying scripts' behaviors diverged.
- Task 2: `/xce-build all` currently aborts (`set -euo pipefail`) at the pre-existing `meson test --print-errorlogs` step, BEFORE reaching the migrated Go-test line — two unrelated, already-logged C-side failures (bug-1723, LeakSanitizer on `xcelib_test`; bug-1758, `sched_core_events_test` deterministic failure), both predating this spec. Task 2's `all`-case line itself is verified (byte-identical to the fully-verified `apiserver`-case line), but this means task 4.1's "final go/no-go gate" and task 4.2's post-deletion re-validation will hit the SAME pre-existing abort when running `/xce-build all` — those tasks will need to either scope their validation to `/xce-build apiserver` + a separate `meson test` invocation that tolerates these 2 known-unrelated failures, or note them as pre-existing/out-of-scope rather than treating them as a regression from this spec.
- Task 1.2: design.md's Testing Strategy suggests `librtti_dep` as the example dependency to remove when proving the new backend test() exercises the real CGO path. Empirically (implementer + independent reviewer both reproduced this live), removing `librtti_dep` does NOT reproduce a link failure for the `apiserver/backend/...` test package — none of `librtti`'s exported symbols are actually pulled into this specific link unit's undefined-symbol set. `c_capnproto_dep` (real `capn_*` undefined-reference errors from `request.c`/`controller.capnp.c`) is the dependency that genuinely reproduces the failure and was used instead. `apiserver_cgo_deps` itself was left byte-identical to task 1.1's state (verification-only task, no permanent code change).

View file

@ -137,14 +137,13 @@ kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
## Go Frontend Build
Go builds use `src/frontend/Makefile` for development, Meson for production:
Go builds use Meson for both development and production — apiserver and xcecli have no separate build system:
```bash
make build-server # → /tmp/apiserver
make build-cli # → /tmp/xce_static (CGO_ENABLED=0)
make test # unit tests (CGO_ENABLED=0)
make test-backend # CGO-backed backend tests (needs nix dev shell)
make test-integration # full regression suite (needs XCE_BIN + SRV_URL)
/xce-build apiserver # → <build_dir>/frontend/apiserver/xce_apiserver, xcecli; runs unit tests (CGO_ENABLED=0)
meson test -C <build_dir> --suite apiserver # CGO-backed backend tests (needs nix dev shell)
XCE=<build_dir>/frontend/xcecli/xcecli URL=<url> \
src/frontend/tests/regression/run.sh # full regression suite
```
Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.

View file

@ -95,7 +95,7 @@ Plugin-based components add their own subdirectory:
**Go tests** (co-located `_test.go`):
- Table-driven unit tests, `t.TempDir()` for fixtures
- Global state reset helpers (e.g., `resetRBACState()`) for test isolation
- `make test` = unit only; `make test-all` = unit + integration
- `/xce-build apiserver` = unit tests only; add `meson test -C <build_dir> --suite apiserver` (backend) and `tests/regression/run.sh` (integration) for full coverage
## Code Organization Principles

View file

@ -27,19 +27,18 @@ All controller state is in-memory. No persistent database in the API tier; job/q
| System | Scope |
|--------|-------|
| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper |
| **Make** | Go development loop (build/test) from `src/frontend/` |
| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper — sole build/test path for apiserver/xcecli |
| **Nix flakes** | Hermetic dev environment, AppImage packaging |
```bash
# Full build (C + Go, debug + ASAN)
/xce-build
# Go development loop (from src/frontend/)
make build-server # → /tmp/apiserver
make build-cli # → /tmp/xce_static (CGO_ENABLED=0)
make test # unit tests
make test-integration # regression suite (needs XCE_BIN + SRV_URL)
# Go development loop (apiserver + xcecli)
/xce-build apiserver # builds both binaries + runs unit tests
meson test -C <build_dir> --suite apiserver # CGO-backed backend tests
XCE=<build_dir>/frontend/xcecli/xcecli URL=http://localhost:8080 \
src/frontend/tests/regression/run.sh # regression suite
# Nix package builds (from src/)
nix build .#xcecore # C daemons + apiserver
@ -78,8 +77,8 @@ Early return on every error. Log with zap fields before returning.
- **TDD mandatory**: Write failing test first, then implementation.
- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures.
- **C tests**: Add test case in `testsuite/` before modifying implementation.
- `make test` must pass without a live server.
- `make test-integration` requires `XCE_BIN` + `SRV_URL` env vars.
- `/xce-build apiserver`'s unit-test step must pass without a live server.
- `tests/regression/run.sh` requires `XCE` + `URL` env vars.
### Code Formatting

View file

@ -5,7 +5,7 @@
- **TDD mandatory**: Write a failing test first, then the minimal implementation. The cycle is Red → Green → Refactor.
- Test behavior, not implementation details.
- Tests must run without a live server or external services.
- A change is not complete until `make test` (and `make test-integration` where applicable) passes.
- A change is not complete until `/xce-build apiserver`'s unit-test step (and `tests/regression/run.sh` where applicable) passes.
## Go Tests (apiserver / xcecli)
@ -44,9 +44,9 @@ func TestPermissions(t *testing.T) {
**Fixtures**: use `t.TempDir()` for file-system state; never rely on absolute paths or external services.
**Build targets**:
- `make test` — unit tests only (`CGO_ENABLED=0`)
- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir)
- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars)
- `/xce-build apiserver` — unit tests only (`CGO_ENABLED=0`), plus the build
- `meson test -C <build_dir> --suite apiserver` — CGO-backed backend tests (requires nix dev shell + build dir)
- `tests/regression/run.sh` — full regression suite (requires `XCE` and `URL` env vars, run directly)
## C Tests (controller / scheduler / lib)
@ -92,4 +92,4 @@ extern void reset_ha_state(ha_state_t *s);
- Mock **external dependencies only** (network calls, file I/O that crosses component boundaries)
- Do **not** mock the system under test
- In-memory stores (`store/`) are real — use them directly in unit tests
- CGO bridge (`backend/`) is covered by `make test-backend` — do not stub it in unit tests
- CGO bridge (`backend/`) is covered by `meson test -C <build_dir> --suite apiserver` — do not stub it in unit tests

View file

@ -39,7 +39,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s
- **Save/restore field-completeness**: any persistence mechanism (WAL fact, scheduler snapshot, checkpoint) must capture every field of the entity it claims to persist — a state that "recovers correctly" while silently missing a field it depends on (e.g. `job.state` correct but `job.allocs` lost) is a bug, not an acceptable simplification, unless the omission is explicitly documented as a scope decision (see `persistence.md`'s "Snapshot Save/Load Field-Completeness" + `APPLY_CLEAN`/bug-1678 sections). When adding or extending save/restore logic, verify the field round-trips through BOTH the write side and every read/apply seam, and run `/xce-audit-invariants` (controller dimension 7 / scheduler dimension 4) rather than assuming a passing test suite already proves it — `APPLY_CLEAN`-style verdicts check row counts and cursor position, not per-field completeness
## Key Files
- `src/frontend/Makefile` — primary build/test orchestration
- `src/tools/xce-build.sh` — primary build/test orchestration (Meson-based; see `build-deployment.md`)
- `src/frontend/apiserver/cmd/main.go` — server startup, init sequence
- `src/frontend/apiserver/router/routers.go` — all routes and middleware chain
- `src/frontend/apiserver/controllers/rbac_check.go` — RBAC loading and enforcement

View file

@ -67,7 +67,7 @@
- `controllers/queues.go` — `preCheckQueueControlRequest` now returns `(bool, error)`; `ControlQueue` receives allFlag locally
- `controllers/host_control_test.go` — 5 new tests: allFlag true/false, concurrent no-race (50 goroutines), invalid action, comment-too-long
- `controllers/queue_control_test.go` — 5 new tests: same coverage for queues
- All tests passing (`make test`)
- All tests passing (via `/xce-build apiserver`'s test step)
- [x] **#8 P0-2c: Register routes + RBAC for host control/select**
- `router/routers.go` — `POST /xce/v1/hosts/control` → RBAC `host:control` (cluster_admin)
@ -117,7 +117,7 @@
## Final
- [x] **#14 Build and test verification**
- `make test` in `src/frontend/` passes — all backend, controllers, store, utils, xcecli tests green
- `/xce-build apiserver`'s test step passes — all backend, controllers, store, utils, xcecli tests green
- `ninja -C build` for C components passes — required meson reconfigure + CGO val_helpers.h fix
- RBAC verification — `host:control` (cluster_admin) and `host:view` (all authenticated) confirmed in routes + rbac.yaml
- Input safety — comment limit 512 chars; select expression limit 4094 chars (1024 reduction deferred to P1-2b)

View file

@ -240,11 +240,11 @@ apiserver -d /opt/xce/apiserver
Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fast on malformed config).
### Build gotcha — use the Meson/Nix build, not `make build-server`
### Build gotcha — apiserver needs real CGO flags, not a bare `go build`
`make build-server` (`src/frontend/Makefile`) runs a bare `go build` with **no `CGO_CFLAGS`/`CGO_LDFLAGS` set**. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time.
A bare `go build` in `src/frontend/apiserver` (no `CGO_CFLAGS`/`CGO_LDFLAGS` set) is not a supported way to produce a working apiserver binary. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time.
**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). Reserve `make build-server`/`make test` for the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there).
**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). `/xce-build apiserver` also runs the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there); a manual `go build`/`go test` invocation is not the canonical path.
### Relationship to the controller

View file

@ -11,10 +11,10 @@ Go-based REST API server (Gin) and CLI tool (Cobra) that expose cluster manageme
## Build & Test
```bash
make build-server # → /tmp/apiserver
make build-cli # → /tmp/xce_static (CGO_ENABLED=0, static binary)
make test # unit tests only (no live server required)
make test-all # unit + integration (needs XCE_BIN and SRV_URL env vars)
/xce-build apiserver # → <build_dir>/frontend/apiserver/xce_apiserver, xcecli; runs unit tests (no live server required)
meson test -C <build_dir> --suite apiserver # CGO-backed backend tests
XCE=<build_dir>/frontend/xcecli/xcecli URL=<url> \
tests/regression/run.sh # integration (needs XCE and URL env vars)
```
## API Server Architecture

View file

@ -64,7 +64,7 @@ Live test: `xcecli` against running `apiserver` with builtin test data.
### Infrastructure
- [x] Nix flake dev shell (`nix develop`) — Go 1.25, PAM, OpenSSL, meson, gnumake, swag, gopls, delve
- [x] Root `Makefile`: `make test`, `make test-integration`, `make test-all`
- [x] Meson build/test: `/xce-build apiserver` (unit tests), `meson test -C <build_dir> --suite apiserver` (CGO-backed backend tests), `tests/regression/run.sh` (integration)
- [x] Regression suite: `tests/regression/run.sh` — 29 tests, all pass
- [x] Unit tests across: `apiserver/store`, `apiserver/utils`, `apiserver/controllers`, `xcecli/cmd/auth`, `xcecli/cmd/jobs`, `xcecli/cmd/roles`, `xcecli/cmd/users`, `xcecli/common/request`, `xcecli/models`