myxce/src/controller/CLAUDE.md

XCE Controller

Overview

C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

Build

# From workspace root
nix develop
meson build && ninja -C build

Architecture

Core source files handle distinct domains:

• job.c — Job lifecycle management
• queue.c — Queue state and control
• cluster.c — Cluster topology
• dataex.c — Data exchange with other daemons

Links against libxce from src/lib/.

Conventions

• C standard: gnu17
• Format with clang-format
• Component-specific headers in include/
• Shared types defined in src/include/xce/ (libxce)

HA Design Constraints

Inbound vs. Outbound Connection Ownership (CRITICAL)

Strict rule: inbound connections are for voting only. All persistent control channels are outbound from the primary.

Connection type	Direction	Purpose	Who closes
Vote RPC	Standby → Primary (outbound from standby)	OPCode_requestVote + reply only	Remote closes after reply
Standby control channel	Primary → Standby (outbound from primary)	ctrlhello, config push, WAL replication, heartbeat	vote_ep_err_handler / standby_ep_err_handler_primary on disconnect

Rules:

• Inbound vote connections (accepted on the listener) are used exclusively for the vote exchange. After the reply is sent the primary does NOT reuse the connection for config/replication. The inbound ep is left open until the remote closes it.
• Primary never uses an inbound connection to push state. All state push (ctrlhello, config, WAL, heartbeat nop) goes over outbound connections stored in ha->standby_endpoints.
• During election: candidate opens outgoing vote connections (one per peer). After winning, raft_cb_become_leader promotes surviving vote connections to standby_endpoints.
• Late joiners: the primary's heartbeat task (primary_heartbeat_fn) scans the peers list and opens a fresh outbound connection to any peer absent from standby_endpoints. On connect, on_standby_connected sends ctrlhello and registers the ep.
• CTRL_STANDBY must not call ha_find_and_connect_to_primary or schedule fetch_config_from_primary after an election result — the primary will connect to it.

Asynchronous, Non-Blocking I/O (CRITICAL)

All inter-controller communication uses iomgr (epoll-based) and vtaskmgr:

• iomgr dispatches inbound packets to vtask handlers — never block inside a vtask.
• vtaskmgr runs handlers on a thread pool; each handler must return quickly.
• Outgoing sends use xce_request(ep, &req) which enqueues to the endpoint's send buffer — non-blocking.
• Blocking I/O (e.g., fetch_config_from_primary, ha_find_and_connect_to_primary) is only permitted from delaymgr worker threads, never from iomgr/vtask context.

Heartbeat Model

XCE uses an asymmetric heartbeat model — different from standard Raft:

• The standby arms a watchdog timer (ha_arm_heartbeat_timeout_vtask) when it enters CTRL_STANDBY. If no message arrives from the primary within heartbeat_timeout_ms, it triggers re-election.
• The primary does not send periodic heartbeats to standbys. Instead, WAL replication entries, config pushes, and ctrlhello messages serve as liveness signals.
• raft.c (the election library) has no heartbeat loop — it is election-only. Do not add heartbeat callbacks to raft_cbs_t; that belongs in the HA layer.

Connection Ownership

• Promoted vote connections (ha->standby_endpoints): owned by vote_ep_err_handler; freed when the standby disconnects.
• Late-joiner inbound connections: owned by the cluster endpoint array (c->endpoints); the ep_private_t role identifies HostRole_controller for cleanup in handle_endpoint_error.
• ha->primary_ctrl_fd: set by handle_ctrlhello on the standby when the first ctrlhello arrives; used by ha_on_primary_ctrl_fd_rdhup to trigger re-election on disconnect.

HA Cluster Deployment

Use the /xce-deploy-ha skill to deploy a local 3-node HA cluster for testing:

/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

Prerequisites:

• Build binary: /xce-build controller
• TLS certs and cluster.bin at ha/data/ (already in repo)

Quick deploy (from repo root):

# Deploy + start all 3 nodes with raft trace logging
bash src/tools/deploy-ha.sh \
  src/xce.build/controller/xce_controller \
  ha/data \
  /tmp/xce-deploy \
  "localhost:9991,localhost:9992,localhost:9993"

VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &

Verify health (after ~15s):

# Primary elected and config pushed
grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
# Config files present on standbys
find /tmp/xce-deploy/ctrl-2/conf /tmp/xce-deploy/ctrl-3/conf -type f

Clean restart (keeps ctrl-1 config + all certs, wipes data/standby-conf/logs):

kill -9 $(pgrep xce_controller) 2>/dev/null || true
bash /tmp/xce-deploy/scripts/cleanup.sh
# Then restart nodes (no redeploy needed)
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &

See .claude/skills/xce-deploy-ha/SKILL.md for full troubleshooting reference.

Key Files

• src/controller/ — Controller source files
• include/ — Controller-specific headers
• src/tools/deploy-ha.sh — HA cluster deploy script

See Also

• Shared library: src/lib/CLAUDE.md
• Root guidelines: CLAUDE.md (repo root)