myxce/.kiro/steering/tech.md

Technology Stack

Architecture

xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer
                                                          controller / scheduler / worker / autoscaler
                                                          ↑↑↑ all linked against libxce (src/lib/)

Request path inside apiserver: Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store

All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.

Core Technologies

• Go 1.23.4 — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no go.work.
• C (gnu17) — controller, worker, scheduler, autoscaler, libxce. No C++.
• Gin v1.10.0 — HTTP router, middleware, binding
• Cobra v1.9.1 — CLI framework for xcecli
• Zap v1.27.0 — structured logging throughout (both Go modules)
• Viper v1.19.0 / v1.20.1 — YAML config management
• Cap'n Proto — wire serialization for WAL and daemon-to-daemon replication
• iomgr / vtaskmgr / delaymgr — libxce async I/O framework: single-threaded epoll loop (iomgr), thread-pool packet dispatch (vtaskmgr), timer thread for periodic/blocking tasks (delaymgr)

Build Systems

System	Scope
Meson ≥ 1.5	C daemons + Go modules via tools/go_build wrapper — sole build/test path for apiserver/xcecli
Nix flakes	Hermetic dev environment, AppImage packaging

# Full build (C + Go, debug + ASAN)
/xce-build

# Go development loop (apiserver + xcecli)
/xce-build apiserver                          # builds both binaries + runs unit tests
meson test -C <build_dir> --suite apiserver   # CGO-backed backend tests
XCE=<build_dir>/frontend/xcecli/xcecli URL=http://localhost:8080 \
    src/frontend/tests/regression/run.sh      # regression suite

# Nix package builds (from src/)
nix build .#xcecore        # C daemons + apiserver
nix build .#xcecore_debug  # debug + ASAN
nix build .#xcecli         # CLI only

Development Standards

Controller Pattern (Go)

// Struct: {Resource}Controller (typically an empty struct{})
type JobController struct{}

// Handler: {Action}{Resource}
func (j JobController) CreateJob(ctx *gin.Context) {
    // Validate input
    var req models.JobSubmitRequest
    if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return }
    parsedIndices, err := preCheckJobRequest(&req)
    if err != nil { ReturnError(ctx, code, err.Error(), nil); return }

    // Business logic — package-level services function, not a controller field
    err = services.SubmitJob(&newJob, arraySize)
    if err != nil { ReturnError(ctx, code, err.Error(), nil); return }

    ReturnSuccess(ctx, code, "submitted", result)
}

Early return on every error. Log with zap fields before returning.

Testing Requirements

• TDD mandatory: Write failing test first, then implementation.
• Go unit tests: Table-driven, no external dependencies, t.TempDir() for fixtures.
• C tests: Add test case in testsuite/ before modifying implementation.
• /xce-build apiserver's unit-test step must pass without a live server.
• tests/regression/run.sh requires XCE + URL env vars.

Code Formatting

• Go: gofmt / goimports (post-edit hooks configured)
• C: clang-format using project config at xce/.clang-format

Process Creation (no fork in multi-threaded processes)

• Rule: a multi-threaded XCE process never calls fork(), vfork(), forkpty(), daemon(), system(), popen() or libuv's uv_spawn() (a fork()+exec on Linux). fork() copies only the calling thread, so a lock another thread held (malloc arena, logger mutex, stdio, ASAN) stays locked in the child, which then deadlocks before it reaches exec.
• Threaded processes: the controller, the scheduler, the xcecd daemon (xcecd, xcecd_virtual: thread pool, network, resource-sync and connector threads), and any process that starts libxce's delaymgr/netio/vtaskmgr threads (xce_ctx_start()).
• How to start a child instead: posix_spawn()/posix_spawnp() (glibc runs it with CLONE_VM|CLONE_VFORK, no copied user code), or a systemd unit over D-Bus as xcecd does for xcecd_worker. Setup that posix_spawn() cannot do (setuid, rlimits, cgroup join) belongs in a single-threaded process such as xcecd_worker, never in the threaded daemon.
• Allowed: exec*() without a fork (process replacement, e.g. the controller's HA self-restart); and a fork-family call in a process proven single-threaded, marked on the call line or the line above with /* xce-fork-ok: <why> */. xcecd_worker (the per-job process) is single-threaded: its call graph from main(), callbacks included, reaches no pthread_create() and no libuv thread-pool call (uv_queue_work, uv_fs_*, uv_getaddrinfo); re-prove it before adding a thread there.
• Scope: C code under src/controller, src/scheduler, src/lib, src/worker. Excluded (decision 2026-10-10): test code (testsuite/, test* files; forking to isolate a test case is fine), the harness (src/harness, xce_test), the autoscaler (not finalized; its provider calls use posix_spawn() since bug-1969, and its suite counts fork() calls through -Wl,--wrap=fork), and the Go frontend (os/exec is safe).
• Enforcement: scripts/check-no-fork.py runs as the meson test no-fork rule (suites lib, controller, scheduler, worker, so /xce-build runs it) and in the pre-commit hook on staged C files.

Auth and RBAC

• Auth backend set in config/auth.yaml (provider: config_file | pam | ldap | oauth2). Only config_file and pam (Linux + CGO) are functional; ldap and oauth2 are documented as future/stub providers whose Authenticate() always returns a "not yet implemented" error.
• Permissions are a plain action string (e.g. "job:view", "queue:submit") paired with a separate structured domain list (e.g. {type: self}, {type: all}) in rbac.yaml — not a combined "action:scope" string. controllers.RequirePermission() checks the action string only; domain/scope is evaluated separately.
• RBAC loaded from config/rbac.yaml, enforced by controllers.RequirePermission() middleware.

Key Technical Decisions

1. No Go workspace — apiserver and xcecli are independent modules; changes to shared types require updating both.
2. In-memory API state — no database in Go tier; store.InitStores() allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (backend.RegisterToBackend() + backend.StartNotificationListener() in cmd/main.go). Only RBAC (config/rbac.yaml) and auth (config/auth.yaml) are actually loaded from YAML at startup.
3. CGO bridge — backend/ package wraps C library calls; unit tests run CGO_ENABLED=0, backend tests need CGO_ENABLED=1 inside nix dev shell.
4. Swagger-first API docs — swaggo/swag annotations on handlers; docs served at runtime by Gin.
5. Structured logging — every log entry carries context via zap.String(), zap.Error(). Never use fmt.Println for operational output.
6. Single-writer HA with full Raft log replication — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, AppendEntries request/reply, per-peer next_index[]/match_index[] progress vectors, (term,index) log matching/truncation, commit-index advance) implemented in the src/lib/raft/ shared module (raft.c/raft.h) and validated by the raft-log-replication, raft-commit-safety, raft-ha-core-safety, and ha-term-safety specs.