myxce/src/controller/CLAUDE.md

# XCE Controller

## Overview
C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

## Build
```bash
# From workspace root
nix develop
meson build && ninja -C build
```

## Architecture
Core source files handle distinct domains:
- `job.c` — Job lifecycle management
- `queue.c` — Queue state and control
- `cluster.c` — Cluster topology
- `dataex.c` — Data exchange with other daemons

Links against `libxce` from `src/lib/`.

## Conventions
- C standard: gnu17
- Format with `clang-format`
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)

## HA Design Constraints

### Inbound vs. Outbound Connection Ownership (CRITICAL)

**Strict rule: inbound connections are for voting only. All persistent control channels are outbound from the primary.**

| Connection type | Direction | Purpose | Who closes |
|---|---|---|---|
| Vote RPC | Standby → Primary (outbound from standby) | `OPCode_requestVote` + reply only | Remote closes after reply |
| Standby control channel | Primary → Standby (outbound from primary) | ctrlhello, config push, WAL replication, heartbeat | `vote_ep_err_handler` / `standby_ep_err_handler_primary` on disconnect |

Rules:
- **Inbound vote connections** (accepted on the listener) are used **exclusively** for the vote exchange. After the reply is sent the primary does NOT reuse the connection for config/replication. The inbound ep is left open until the remote closes it.
- **Primary never uses an inbound connection to push state.** All state push (ctrlhello, config, WAL, heartbeat nop) goes over **outbound connections** stored in `ha->standby_endpoints`.
- During election: candidate opens outgoing vote connections (one per peer). After winning, `raft_cb_become_leader` promotes surviving vote connections to `standby_endpoints`.
- Late joiners: the primary's heartbeat task (`primary_heartbeat_fn`) scans the peers list and **opens a fresh outbound connection** to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello and registers the ep.
- `CTRL_STANDBY` must not call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary` after an election result — the primary will connect to it.

### Asynchronous, Non-Blocking I/O (CRITICAL)
All inter-controller communication uses iomgr (epoll-based) and vtaskmgr:
- `iomgr` dispatches inbound packets to vtask handlers — never block inside a vtask.
- `vtaskmgr` runs handlers on a thread pool; each handler must return quickly.
- Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — non-blocking.
- Blocking I/O (e.g., `fetch_config_from_primary`, `ha_find_and_connect_to_primary`) is only permitted from `delaymgr` worker threads, **never** from iomgr/vtask context.

### Heartbeat Model
XCE uses an asymmetric heartbeat model — different from standard Raft:
- The **standby** arms a watchdog timer (`ha_arm_heartbeat_timeout_vtask`) when it enters `CTRL_STANDBY`. If no message arrives from the primary within `heartbeat_timeout_ms`, it triggers re-election.
- The **primary does not send periodic heartbeats** to standbys. Instead, WAL replication entries, config pushes, and ctrlhello messages serve as liveness signals.
- `raft.c` (the election library) has no heartbeat loop — it is election-only. Do not add heartbeat callbacks to `raft_cbs_t`; that belongs in the HA layer.

### Connection Ownership
- Promoted vote connections (`ha->standby_endpoints`): owned by `vote_ep_err_handler`; freed when the standby disconnects.
- Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`.
- `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect.

## HA Cluster Deployment

Use the `/xce-deploy-ha` skill to deploy a local 3-node HA cluster for testing:

```
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
```

**Prerequisites:**
- Build binary: `/xce-build controller`
- TLS certs and `cluster.bin` at `ha/data/` (already in repo)

**Quick deploy (from repo root):**
```bash
# Deploy + start all 3 nodes with raft trace logging
bash src/tools/deploy-ha.sh \
  src/xce.build/controller/xce_controller \
  ha/data \
  /tmp/xce-deploy \
  "localhost:9991,localhost:9992,localhost:9993"

VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
```

**Verify health (after ~15s):**
```bash
# Primary elected and config pushed
grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
# Config files present on standbys
find /tmp/xce-deploy/ctrl-2/conf /tmp/xce-deploy/ctrl-3/conf -type f
```

**Clean restart** (keeps ctrl-1 config + all certs, wipes data/standby-conf/logs):
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
bash /tmp/xce-deploy/scripts/cleanup.sh
# Then restart nodes (no redeploy needed)
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
```

See `.claude/skills/xce-deploy-ha/SKILL.md` for full troubleshooting reference.

## Key Files
- `src/controller/` — Controller source files
- `include/` — Controller-specific headers
- `src/tools/deploy-ha.sh` — HA cluster deploy script

## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)