# Technology Stack

## Architecture

```
xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer
                                                          controller / scheduler / worker / autoscaler
                                                          ↑↑↑ all linked against libxce (src/lib/)
```

Request path inside apiserver: `Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store`

All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.

## Core Technologies

- **Go 1.23.4** — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no `go.work`.
- **C (gnu17)** — controller, worker, scheduler, autoscaler, libxce. No C++.
- **Gin v1.10.0** — HTTP router, middleware, binding
- **Cobra v1.9.1** — CLI framework for xcecli
- **Zap v1.27.0** — structured logging throughout (both Go modules)
- **Viper v1.19.0 / v1.20.1** — YAML config management
- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication
- **iomgr / vtaskmgr / delaymgr** — libxce async I/O framework: single-threaded epoll loop (iomgr), thread-pool packet dispatch (vtaskmgr), timer thread for periodic/blocking tasks (delaymgr)

## Build Systems

| System | Scope |
|--------|-------|
| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper — sole build/test path for apiserver/xcecli |
| **Nix flakes** | Hermetic dev environment, AppImage packaging |

```bash
# Full build (C + Go, debug + ASAN)
/xce-build

# Go development loop (apiserver + xcecli)
/xce-build apiserver                          # builds both binaries + runs unit tests
meson test -C <build_dir> --suite apiserver   # CGO-backed backend tests
XCE=<build_dir>/frontend/xcecli/xcecli URL=http://localhost:8080 \
    src/frontend/tests/regression/run.sh      # regression suite

# Nix package builds (from src/)
nix build .#xcecore        # C daemons + apiserver
nix build .#xcecore_debug  # debug + ASAN
nix build .#xcecli         # CLI only
```

## Development Standards

### Controller Pattern (Go)

```go
// Struct: {Resource}Controller (typically an empty struct{})
type JobController struct{}

// Handler: {Action}{Resource}
func (j JobController) CreateJob(ctx *gin.Context) {
    // Validate input
    var req models.JobSubmitRequest
    if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return }
    parsedIndices, err := preCheckJobRequest(&req)
    if err != nil { ReturnError(ctx, code, err.Error(), nil); return }

    // Business logic — package-level services function, not a controller field
    err = services.SubmitJob(&newJob, arraySize)
    if err != nil { ReturnError(ctx, code, err.Error(), nil); return }

    ReturnSuccess(ctx, code, "submitted", result)
}
```

Early return on every error. Log with zap fields before returning.

### Testing Requirements

- **TDD mandatory**: Write failing test first, then implementation.
- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures.
- **C tests**: Add test case in `testsuite/` before modifying implementation.
- `/xce-build apiserver`'s unit-test step must pass without a live server.
- `tests/regression/run.sh` requires `XCE` + `URL` env vars.

### Code Formatting

- Go: `gofmt` / `goimports` (post-edit hooks configured)
- C: `clang-format` using project config at `xce/.clang-format`

### Process Creation (no fork in multi-threaded processes)

- **Rule**: a multi-threaded XCE process never calls `fork()`, `vfork()`, `forkpty()`, `daemon()`, `system()`, `popen()` or libuv's `uv_spawn()` (a `fork()`+`exec` on Linux). `fork()` copies only the calling thread, so a lock another thread held (malloc arena, logger mutex, stdio, ASAN) stays locked in the child, which then deadlocks before it reaches `exec`.
- **Threaded processes**: the controller, the scheduler, the `xcecd` daemon (`xcecd`, `xcecd_virtual`: thread pool, network, resource-sync and connector threads), and any process that starts libxce's delaymgr/netio/vtaskmgr threads (`xce_ctx_start()`).
- **How to start a child instead**: `posix_spawn()`/`posix_spawnp()` (glibc runs it with `CLONE_VM|CLONE_VFORK`, no copied user code), or a systemd unit over D-Bus as `xcecd` does for `xcecd_worker`. Setup that `posix_spawn()` cannot do (setuid, rlimits, cgroup join) belongs in a single-threaded process such as `xcecd_worker`, never in the threaded daemon.
- **Allowed**: `exec*()` without a fork (process replacement, e.g. the controller's HA self-restart); and a fork-family call in a process proven single-threaded, marked on the call line or the line above with `/* xce-fork-ok: <why> */`. `xcecd_worker` (the per-job process) is single-threaded: its call graph from `main()`, callbacks included, reaches no `pthread_create()` and no libuv thread-pool call (`uv_queue_work`, `uv_fs_*`, `uv_getaddrinfo`); re-prove it before adding a thread there.
- **Scope**: C code under `src/controller`, `src/scheduler`, `src/lib`, `src/worker`. Excluded (decision 2026-10-10): test code (`testsuite/`, `test*` files; forking to isolate a test case is fine), the harness (`src/harness`, `xce_test`), the autoscaler (not finalized; its provider calls use `posix_spawn()` since bug-1969, and its suite counts `fork()` calls through `-Wl,--wrap=fork`), and the Go frontend (`os/exec` is safe).
- **Enforcement**: `scripts/check-no-fork.py` runs as the meson test `no-fork rule` (suites lib, controller, scheduler, worker, so `/xce-build` runs it) and in the pre-commit hook on staged C files.

### Auth and RBAC

- Auth backend set in `config/auth.yaml` (`provider: config_file | pam | ldap | oauth2`). Only `config_file` and `pam` (Linux + CGO) are functional; `ldap` and `oauth2` are documented as future/stub providers whose `Authenticate()` always returns a "not yet implemented" error.
- Permissions are a plain action string (e.g. `"job:view"`, `"queue:submit"`) paired with a separate structured `domain` list (e.g. `{type: self}`, `{type: all}`) in `rbac.yaml` — not a combined `"action:scope"` string. `controllers.RequirePermission()` checks the action string only; domain/scope is evaluated separately.
- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware.

## Key Technical Decisions

1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both.
2. **In-memory API state** — no database in Go tier; `store.InitStores()` allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (`backend.RegisterToBackend()` + `backend.StartNotificationListener()` in `cmd/main.go`). Only RBAC (`config/rbac.yaml`) and auth (`config/auth.yaml`) are actually loaded from YAML at startup.
3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
6. **Single-writer HA with full Raft log replication** — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, `AppendEntries` request/reply, per-peer `next_index[]`/`match_index[]` progress vectors, `(term,index)` log matching/truncation, commit-index advance) implemented in the `src/lib/raft/` shared module (`raft.c`/`raft.h`) and validated by the `raft-log-replication`, `raft-commit-safety`, `raft-ha-core-safety`, and `ha-term-safety` specs.