Rongsong Shen
19052674bd
The scaler is multi-threaded (monitor and demand threads); invoke() forked and, in the child, logged, snprintf'd and setenv()'d before execl(), which can deadlock on a lock another thread held at fork time and block the caller in waitpid() forever. run_script() now posix_spawn()s `sh <script> -f <in.json>`: stdout and stderr go to the output file through file actions, the provider variables (PRO_CONF_DIR, PRO_DATA_DIR, PRO_LOG_DIR, PROVIDER_NAME) are passed as the child's envp (provider_env()), the child starts with an empty signal mask and default dispositions, and waitpid() retries on EINTR. The scaler unit suite (scaler/testsuite) was never built; it is now the meson test "autoscaler test", linked with -Wl,--wrap=fork so it counts fork() calls during a provider call. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
110 lines
8.1 KiB
Markdown
110 lines
8.1 KiB
Markdown
# Technology Stack
|
|
|
|
## Architecture
|
|
|
|
```
|
|
xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer
|
|
controller / scheduler / worker / autoscaler
|
|
↑↑↑ all linked against libxce (src/lib/)
|
|
```
|
|
|
|
Request path inside apiserver: `Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store`
|
|
|
|
All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.
|
|
|
|
## Core Technologies
|
|
|
|
- **Go 1.23.4** — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no `go.work`.
|
|
- **C (gnu17)** — controller, worker, scheduler, autoscaler, libxce. No C++.
|
|
- **Gin v1.10.0** — HTTP router, middleware, binding
|
|
- **Cobra v1.9.1** — CLI framework for xcecli
|
|
- **Zap v1.27.0** — structured logging throughout (both Go modules)
|
|
- **Viper v1.19.0 / v1.20.1** — YAML config management
|
|
- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication
|
|
- **iomgr / vtaskmgr / delaymgr** — libxce async I/O framework: single-threaded epoll loop (iomgr), thread-pool packet dispatch (vtaskmgr), timer thread for periodic/blocking tasks (delaymgr)
|
|
|
|
## Build Systems
|
|
|
|
| System | Scope |
|
|
|--------|-------|
|
|
| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper — sole build/test path for apiserver/xcecli |
|
|
| **Nix flakes** | Hermetic dev environment, AppImage packaging |
|
|
|
|
```bash
|
|
# Full build (C + Go, debug + ASAN)
|
|
/xce-build
|
|
|
|
# Go development loop (apiserver + xcecli)
|
|
/xce-build apiserver # builds both binaries + runs unit tests
|
|
meson test -C <build_dir> --suite apiserver # CGO-backed backend tests
|
|
XCE=<build_dir>/frontend/xcecli/xcecli URL=http://localhost:8080 \
|
|
src/frontend/tests/regression/run.sh # regression suite
|
|
|
|
# Nix package builds (from src/)
|
|
nix build .#xcecore # C daemons + apiserver
|
|
nix build .#xcecore_debug # debug + ASAN
|
|
nix build .#xcecli # CLI only
|
|
```
|
|
|
|
## Development Standards
|
|
|
|
### Controller Pattern (Go)
|
|
|
|
```go
|
|
// Struct: {Resource}Controller (typically an empty struct{})
|
|
type JobController struct{}
|
|
|
|
// Handler: {Action}{Resource}
|
|
func (j JobController) CreateJob(ctx *gin.Context) {
|
|
// Validate input
|
|
var req models.JobSubmitRequest
|
|
if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
|
parsedIndices, err := preCheckJobRequest(&req)
|
|
if err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
|
|
|
// Business logic — package-level services function, not a controller field
|
|
err = services.SubmitJob(&newJob, arraySize)
|
|
if err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
|
|
|
ReturnSuccess(ctx, code, "submitted", result)
|
|
}
|
|
```
|
|
|
|
Early return on every error. Log with zap fields before returning.
|
|
|
|
### Testing Requirements
|
|
|
|
- **TDD mandatory**: Write failing test first, then implementation.
|
|
- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures.
|
|
- **C tests**: Add test case in `testsuite/` before modifying implementation.
|
|
- `/xce-build apiserver`'s unit-test step must pass without a live server.
|
|
- `tests/regression/run.sh` requires `XCE` + `URL` env vars.
|
|
|
|
### Code Formatting
|
|
|
|
- Go: `gofmt` / `goimports` (post-edit hooks configured)
|
|
- C: `clang-format` using project config at `xce/.clang-format`
|
|
|
|
### Process Creation (no fork in multi-threaded processes)
|
|
|
|
- **Rule**: a multi-threaded XCE process never calls `fork()`, `vfork()`, `forkpty()`, `daemon()`, `system()`, `popen()` or libuv's `uv_spawn()` (a `fork()`+`exec` on Linux). `fork()` copies only the calling thread, so a lock another thread held (malloc arena, logger mutex, stdio, ASAN) stays locked in the child, which then deadlocks before it reaches `exec`.
|
|
- **Threaded processes**: the controller, the scheduler, the `xcecd` daemon (`xcecd`, `xcecd_virtual`: thread pool, network, resource-sync and connector threads), and any process that starts libxce's delaymgr/netio/vtaskmgr threads (`xce_ctx_start()`).
|
|
- **How to start a child instead**: `posix_spawn()`/`posix_spawnp()` (glibc runs it with `CLONE_VM|CLONE_VFORK`, no copied user code), or a systemd unit over D-Bus as `xcecd` does for `xcecd_worker`. Setup that `posix_spawn()` cannot do (setuid, rlimits, cgroup join) belongs in a single-threaded process such as `xcecd_worker`, never in the threaded daemon.
|
|
- **Allowed**: `exec*()` without a fork (process replacement, e.g. the controller's HA self-restart); and a fork-family call in a process proven single-threaded, marked on the call line or the line above with `/* xce-fork-ok: <why> */`. `xcecd_worker` (the per-job process) is single-threaded: its call graph from `main()`, callbacks included, reaches no `pthread_create()` and no libuv thread-pool call (`uv_queue_work`, `uv_fs_*`, `uv_getaddrinfo`); re-prove it before adding a thread there.
|
|
- **Scope**: C code under `src/controller`, `src/scheduler`, `src/lib`, `src/worker`. Excluded (decision 2026-10-10): test code (`testsuite/`, `test*` files; forking to isolate a test case is fine), the harness (`src/harness`, `xce_test`), the autoscaler (not finalized; its provider calls use `posix_spawn()` since bug-1969, and its suite counts `fork()` calls through `-Wl,--wrap=fork`), and the Go frontend (`os/exec` is safe).
|
|
- **Enforcement**: `scripts/check-no-fork.py` runs as the meson test `no-fork rule` (suites lib, controller, scheduler, worker, so `/xce-build` runs it) and in the pre-commit hook on staged C files.
|
|
|
|
### Auth and RBAC
|
|
|
|
- Auth backend set in `config/auth.yaml` (`provider: config_file | pam | ldap | oauth2`). Only `config_file` and `pam` (Linux + CGO) are functional; `ldap` and `oauth2` are documented as future/stub providers whose `Authenticate()` always returns a "not yet implemented" error.
|
|
- Permissions are a plain action string (e.g. `"job:view"`, `"queue:submit"`) paired with a separate structured `domain` list (e.g. `{type: self}`, `{type: all}`) in `rbac.yaml` — not a combined `"action:scope"` string. `controllers.RequirePermission()` checks the action string only; domain/scope is evaluated separately.
|
|
- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware.
|
|
|
|
## Key Technical Decisions
|
|
|
|
1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both.
|
|
2. **In-memory API state** — no database in Go tier; `store.InitStores()` allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (`backend.RegisterToBackend()` + `backend.StartNotificationListener()` in `cmd/main.go`). Only RBAC (`config/rbac.yaml`) and auth (`config/auth.yaml`) are actually loaded from YAML at startup.
|
|
3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
|
|
4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
|
|
5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
|
|
6. **Single-writer HA with full Raft log replication** — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, `AppendEntries` request/reply, per-peer `next_index[]`/`match_index[]` progress vectors, `(term,index)` log matching/truncation, commit-index advance) implemented in the `src/lib/raft/` shared module (`raft.c`/`raft.h`) and validated by the `raft-log-replication`, `raft-commit-safety`, `raft-ha-core-safety`, and `ha-term-safety` specs.
|