Rongsong Shen
f3babec5bc
Replaces blocking pthread_join() per thread with a CLOCK_MONOTONIC deadline + 5ms poll loop. CONF_PUSH_TIMEOUT_MS (500ms) now enforced. Matches design §6.3 aggregate timeout guarantee. - Add done field to push_ctx_t; push_thread sets it on exit - Add static wait_all_with_timeout() before push_config_to_standbys() - Wire push_config_to_standbys() to use wait_all_with_timeout() - Add #include <stdatomic.h> to config_repl.c - Guard push_ctx_t local typedef with #ifndef XCE_TESTING - Expose push_ctx_t and wait_all_with_timeout_for_test() in header under XCE_TESTING for unit test access - Add test_wait_all_with_timeout_slow_threads (timeout fires, acked=0) - Add test_wait_all_with_timeout_fast_threads (all complete, acked=3) - Update design doc §6.3 with actual implementation signature and 5ms poll interval note
124 KiB
XCE Controller: Configuration Replication Design
Status: Implemented (Phase 1 active; Phase 2 gated) Last Updated: 2026-05-13 Target:
src/controller/(C daemon) Depends on:persistence-and-ha-design.mdPart 5 (HA Failover) Branch:persistent_haImplementation commits:683c092–40c7a93(20 commits)
Table of Contents
- Problem Statement
- Goals and Non-Goals
- Config Storage Layout
- SHA-256 as Version and Integrity Token
- OP_replConfig Protocol
- OP_upgradeConfig Protocol
- Startup Sequence
- cluster_init() Phase Split
- am_i_primary() Implementation
- Error Handling
- Service Management and Deployment
- Files Changed
- WAL Record Bound
- Test Plan
1. Problem Statement
1.1 Current behaviour
Every controller candidate reads its cluster configuration (ClusterConfig Cap'n Proto blob)
from a local file at ${workdir}/${name}/conf/current_config on startup. This means:
- Every candidate node must be independently provisioned with a copy of the config file.
- Any config change (new queue, resource definition update) must be applied to every candidate separately. There is no enforcement mechanism — a missed copy is silent.
- A standby running a stale config promotes with wrong queue/resource definitions. The mismatch is only discovered when the promoted standby starts serving clients.
1.2 Proposed behaviour
The primary (highest-priority candidate) is the sole config authority:
- The primary reads
ClusterConfigfrom its localconf/current/on startup. If the file is missing, it is a fatal error — the cluster cannot start. - A standby that has no local config fetches it from the primary over the existing SSL/iomgr pipeline before completing initialisation. The received config is saved locally so the standby can restart without the primary being available.
- On every reconnect, the standby sends its current config's SHA-256 tag (first 8 bytes). The primary skips the push if the tag matches, avoiding unnecessary reinitialisation.
2. Goals and Non-Goals
Goals
Provisioning
- Single provisioning point: only the primary candidate node requires the config file.
- Detect and minimize config drift: standbys fetch from primary on every reconnect.
When the primary is unreachable, a saved config is used with staleness warning (§10.3).
Config drift is eliminated for the normal case (reconnect); the edge case where a
standby promotes while the primary is unreachable is mitigated by the staleness
warning and
config_staleflag but cannot be fully prevented without a quorum gate (deferred to §14.4). Residual risk accepted: if both the primary and all standbys with current config are simultaneously down, a standby with only a stale saved config will promote and serve with potentially wrong queue/resource definitions — the same failure mode described in §1.1. Theconfig_staleflag and log warnings make this discoverable (the current behaviour is silent), but do not prevent it. Operators who require hard prevention should implement the quorum gate (§14.4).
Integrity
- Atomic, crash-safe config update on the standby.
- SHA-256 as the token for both change detection and cryptographic integrity validation.
Observability
- Human-visible config version in the filesystem (SHA-256-named files + pointer file).
Remote config push
OP_upgradeConfigallows an admin tool to push a new config blob to the primary via the controller endpoint. The primary saves it, pushes it to connected standbys (best-effort, with ACK count in the reply), then restarts itself to apply the new config (~4–13s downtime; see §6.6 for timing decomposition). The admin tool that drivesOP_upgradeConfigis out of scope for this design but is a prerequisite for Phase 2 — it must encodeClusterConfigto Cap'n Proto binary, compute SHA-256, and send the request via the controller's internal endpoint (seeFUTURE_WORK.md).
Non-Goals
Scope
- Hot-reload of config without restart. Config drives kvmgr slot counts and resource definitions that are baked in at Phase 2 init time; a restart is required to apply them.
- YAML config parsing in the controller. The controller reads only the Cap'n Proto binary blob. A separate admin tool (out of scope) produces the blob.
Responsibility boundaries
- Semantic validation of
ClusterConfigcontent (e.g., non-zero queue count, valid port ranges, host list completeness). The controller validates structural correctness only (valid Cap'n Proto decode, size withinCONF_MAX_SIZE). Semantic correctness is the responsibility of the admin tool that produces the blob.
Compatibility
- Backward compatibility. This is the first version of the protocol.
Alternatives Considered
Config-management-based distribution: The simplest alternative is to distribute
the ClusterConfig blob via the same config management system (Ansible, Puppet,
Salt) already recommended for --peers distribution (§11.2). This avoids all new
controller code.
| Approach | Provisioning | Drift detection | Bootstrap without pre-seeding | Restart-free change | LOC |
|---|---|---|---|---|---|
| Config management only | Multi-node push | Manual (diff files) | No | No | 0 |
| Config-in-join-reply | Primary-only | Overwrite on every join | Yes (join reply) | No (restart required) | ~50 |
| This design | Primary-only | Automatic (SHA-256 tag) | Yes (pull from primary) | No (restart required) | ~310 |
Config management alone works for clusters where all controller nodes are managed by the same automation system. This design addresses the gap where that assumption breaks down: a newly provisioned standby with no pre-existing config file, or a standby rejoining after extended offline where its saved config may be stale. The HEAD-file + SHA-256 approach also provides cryptographic proof that two controllers run identical configs without parsing binary blobs — useful for operator verification.
Config-in-join-reply (minimal approach): A ~50 LOC alternative: include the raw
config bytes in the controller join reply. When a standby completes OP_join, the
primary attaches the current ClusterConfig blob. The standby saves it to
conf/current_config (overwriting any stale copy). This handles the bootstrap case
and the reconnect case (join reply always carries current config). It does not
provide: tag-based push skip (every join sends the full blob), cryptographic
integrity validation, version history/retention, or OP_upgradeConfig live push.
For clusters with infrequent joins and small configs (<100 KB), this is sufficient.
The full design is chosen for clusters where config integrity verification, change
auditability, and the push mechanism for online config updates are operational requirements.
WAL-based config replication (considered and rejected): XCE already has a WAL
replication path (src/controller/repl.c, ~340 lines) that streams job/queue/resource
mutations from primary to standbys. Extending it to carry serialised ClusterConfig
blobs would reuse the existing replication infrastructure. This approach is rejected
for three reasons: (1) Semantic mismatch — WAL replication is continuous and
ordered; config changes are one-shot, admin-initiated events that should not be
interleaved with job mutation records (a WAL ordering dependency between config records
and operation records creates unnecessary coupling). (2) Atomicity — config changes
require saving the full blob, computing SHA-256, updating the HEAD-file pointer, and
trimming old versions atomically; this is a self-contained operation with its own
integrity guarantees distinct from WAL append semantics. (3) Bootstrap ordering —
WAL replication requires persist_init and a replica connection to be established
(Phase 2), but the standby needs config before Phase 2 to initialise iomgr and kvmgr
(§7). A separate config fetch protocol (§5) avoids this circular dependency.
Why this design: The chosen approach layers on the full protocol not because the
bootstrap case demands it, but because the same SHA-256 infrastructure underpins:
(1) tag-based skip on reconnect (avoiding unnecessary deserialisation),
(2) cryptographic integrity validation of received blobs, (3) human-visible version
identity via conf/current pointer, and (4) the OP_upgradeConfig push mechanism
for remote admin-driven config changes. Each layer individually could be deferred;
together they form a coherent config management lifecycle that the simpler
approaches do not provide.
Phased Delivery
The design is organised into two phases, but both phases ship together in a single PR. The "phase" distinction refers to activation readiness, not separate deliveries:
Phase 1 — Pull-only config replication (active on merge; delivers Provisioning, Integrity, Observability goals):
OP_replConfigpull protocol (§5) with SHA-256 tag comparison- HEAD-file + versioned files storage layout (§3)
- Standby fetch with retry (§5.5)
am_i_primary()election probing (§9, shared with parent HA design)- ~310 LOC
Phase 2 — Remote config push (infrastructure ships now, feature activates when admin tool
and validate_host() are complete; see FUTURE_WORK.md):
OP_upgradeConfighandler (§6.4),OP_replConfigPushfan-out (§6.3)lost_and_foundqueue/host routing (§6.5)standby_endpointstracking,restart_requestedmechanism- WAL record bound (§13)
- ~170 LOC
The OP_upgradeConfig handler is gated behind a compile-time flag
CONF_ENABLE_UPGRADE_CONFIG (default: 0, disabled — see §5.3 safety gate). The
gate MUST NOT be set to 1 in any production build until BOTH prerequisites are met:
validate_host()is implemented with a real authorisation check (not a stub that returns success for all peers). Minimum viable check: verify the peer's certificate CN against an allowlist of authorised admin hosts. SeeFUTURE_WORK.md§6 for the implementation plan.- Admin config tool exists and has been tested end-to-end. It must encode
ClusterConfigto Cap'n Proto binary, compute SHA-256, connect via SSL, sendOP_upgradeConfig, display the reply (version tag, acked count, error), and pollOP_probeafter restart. SeeFUTURE_WORK.md§1 for the tool specification.
Prerequisite delivery commitment: Both prerequisites are tracked in
FUTURE_WORK.md and MUST be completed before the next production release that sets
CONF_ENABLE_UPGRADE_CONFIG=1. A release may ship with the gate at 0 (Phase 1
active, Phase 2 infrastructure compiled but inert). The gate is removed from the
codebase when both prerequisites are merged and integration-tested. Phase 1 alone
eliminates manual provisioning and config drift — the core problems stated in §1.
3. Config Storage Layout
3.1 Directory structure
The parent HA design (persistence-and-ha-design.md Part 5) specifies a config storage
layout under conf/. This design replaces that with the HEAD-file + versioned files
approach described below. This design replaces that with a
HEAD-file + versioned files layout that is atomically crash-safe. The
HEAD-file pattern avoids symlink(2), which is not atomic on all filesystems and
would require a separate atomic-rename technique anyway.
Local filesystem only — NFS is explicitly unsupported. The controller's working
directory (including conf/, data/wal/, and data/snap/) MUST reside on a local
filesystem. persist_init() (persist.c:217-222) detects and rejects NFS mounts at
startup with a FATAL error. Config replication assumes local rename(2) atomicity
and fsync(2) durability guarantees; neither holds on NFS. No effort is made to
support network or distributed filesystems for config storage. Cross-filesystem
portability is a non-goal.
${workdir}/${name}/
└── conf/
├── current ← plain text pointer: "cluster.bin.a1b2c3d4e5f6\n"
├── cluster.bin.0f1e2d3c4a5b ← previous version (retained)
└── cluster.bin.a1b2c3d4e5f6 ← current version
The filename suffix is the first 12 hex chars of the file's SHA-256 hash.
cat conf/current gives the active version filename at a glance — identical to Git's
HEAD file pattern. Two standbys with identical conf/current contents are provably
running identical configs without parsing any binary content.
Note
: The current code uses a flat path
conf/current_config(main.c:115). This is replaced by a two-step resolution: readconf/currentto get the versioned filename, then open that file. Theconffilefield inmyconf_tis a heap-allocated string set viaasprintf()during Phase 1 byload_primary_config()(primary path) orfetch_config_from_primary()(standby path) after reading theconf/currentpointer file and resolving it to the versioned file path (e.g.,${workdir}/${name}/conf/cluster.bin.a1b2c3d4e5f6). It is freed inmyconf_fini()alongside the other owned fields. The two-step path resolution is load-bearing for the atomicrename(2)semantics described in §3.2 — the pointer file swap mutates which versioned file is current, and all code that needs config bytes resolves throughconf/current→conffileto get the correct path.
3.2 Write procedure (standby receiving new config)
effective_dir throughout this section denotes ${workdir}/${name} — the controller instance's base directory. All conf/ paths are relative to it.
// 0. Ensure conf/ directory exists with restrictive permissions
snprintf(conf_dir, ..., "%s/conf", effective_dir);
mkdir(conf_dir, 0700); /* same as data/wal/ and data/snap/ — trusted LAN model.
* On EEXIST, verify existing permissions with fstat() and
* chmod() to 0700 if needed — mkdir() silently ignores
* pre-existing directories with wrong permissions. */
// 1. Write new versioned file atomically
// Thread-unique .tmp suffix avoids truncation under concurrent push+fetch (§10.11)
snprintf(tmp_path, ..., "%s/conf/cluster.bin.%s.%lu.tmp",
effective_dir, hex_tag, (unsigned long)pthread_self());
snprintf(ver_path, ..., "%s/conf/cluster.bin.%s", effective_dir, hex_tag);
fd = open(tmp_path, O_WRONLY|O_CREAT|O_TRUNC, 0600);
if (fd < 0) {
ERROR("open(%%s) failed: %%s", tmp_path, strerror(errno));
return -errno;
}
if (write(fd, bytes, len) != (ssize_t)len) {
ERROR("short write on %%s: %%s", tmp_path, strerror(errno));
close(fd); unlink(tmp_path);
return -EIO;
}
if (fsync(fd) != 0) {
ERROR("fsync(%%s) failed: %%s", tmp_path, strerror(errno));
close(fd); unlink(tmp_path);
return -EIO;
}
close(fd);
rename(tmp_path, ver_path); /* atomic: versioned file now on disk */
// 2. Atomically update the HEAD-style pointer file
// Thread-unique .tmp suffix avoids race under concurrent push+fetch (§10.11)
snprintf(ptr_tmp, ..., "%s/conf/current.%lu.tmp", effective_dir, (unsigned long)pthread_self());
snprintf(ptr_path, ..., "%s/conf/current", effective_dir);
char target[NAME_MAX];
snprintf(target, sizeof(target), "cluster.bin.%s\n", hex_tag);
fd = open(ptr_tmp, O_WRONLY|O_CREAT|O_TRUNC, 0600);
write(fd, target, strlen(target));
fsync(fd);
close(fd);
rename(ptr_tmp, ptr_path); /* atomic pointer swap — no symlink(2) required */
// 2.5. Durability: fsync the conf/ directory to persist the rename(2)
fd = open(conf_dir, O_RDONLY);
if (fd >= 0) {
fsync(fd);
close(fd);
}
// 3. Trim old versioned files beyond keep_n (default: 2)
conf_trim_old_versions(effective_dir, keep_n);
Read procedure — called at startup to resolve conffile:
static int conf_read_current_filename(const char *conf_dir,
char *out, size_t outsz)
{
char head_path[PATH_MAX];
snprintf(head_path, sizeof(head_path), "%s/current", conf_dir);
FILE *f = fopen(head_path, "r");
if (!f) return -ENOENT;
/* Distinguish empty file from EOF — fgets returns NULL for both.
* An empty pointer file means the controller crashed between open()
* and the first write()+fsync()+close() in §3.2 step 2. */
struct stat st;
if (fstat(fileno(f), &st) == 0 && st.st_size == 0) {
fclose(f);
ERROR("conf/current is empty — treating as missing config (§10.6a)");
return -ENODATA;
}
if (!fgets(out, outsz, f)) { fclose(f); return -EIO; }
fclose(f);
out[strcspn(out, "\n")] = '\0'; /* strip trailing newline */
/* Validate: reject path separators, require "cluster.bin." prefix,
* require exactly 12 hex chars in the suffix. */
if (strchr(out, '/')) {
ERROR("conf/current contains path separator — corrupted pointer file");
return -EINVAL;
}
if (strncmp(out, "cluster.bin.", 12) != 0) {
ERROR("conf/current filename '%s' does not start with 'cluster.bin.'", out);
return -EINVAL;
}
const char *hex = out + 12;
if (strlen(hex) != 12 || strspn(hex, "0123456789abcdef") != 12) {
ERROR("conf/current suffix '%s' is not 12 hex chars", hex);
return -EINVAL;
}
return 0;
}
After calling this, myconf->conffile is set via asprintf() to
"${conf_dir}/${out}" — a heap-allocated copy that survives past the caller's
stack frame. It is freed in myconf_fini() alongside the other owned fields.
Crash safety: if the process crashes after step 1 but before step 2, conf/current
still contains the old versioned filename — the standby boots on the previous config and
re-fetches on the next connection. If it crashes mid-step-1 (before the rename of the
.tmp file), the orphaned .tmp file is ignored on next boot.
3.3 Retention
Keep the last conf_keep_n versioned files on disk (default: 2 — current + one
previous). Trimming deletes the oldest by mtime after every successful update.
Stale .tmp files from interrupted writes are also cleaned during the trim pass.
conf_keep_n is a field in myconf_t — it is needed during config save (between
Phase 1 and Phase 2), before persist_init() runs and persist_config_t is available.
It is initialised from a hardcoded default (2) or the --conf-keep-n CLI flag during
Phase 1, before any config is loaded. It is not read from the cluster config, so it
is available before config fetch on standbys.
4. SHA-256 as Version and Integrity Token
4.1 Why SHA-256
configSeq and CRC32 were both considered and rejected:
| Property | configSeq | CRC32 | SHA-256 |
|---|---|---|---|
| Change detection | Compare ints | Compare CRC32 | Compare first 8 bytes |
| Integrity validation | Separate field | Same field (CRC) | Full 32-byte hash |
| Admin tool required | Yes | No | No |
| Push skip on match | No (always incr) | Yes (same content → same CRC) | Yes (same content → same hash) |
| Collision resistance | N/A | 2^-32 (vulnerable) | 2^-256 (none in practice) |
| Cryptographic strength | None | None (invertible) | Full (keyless integrity) |
| On-disk dependency | — | zlib crc32() |
OpenSSL (already in Phase 1) |
Why cryptographic hash in a trusted-LAN model? The trusted-LAN model (§5.3)
means we trust the network path between controllers, but config blobs traverse
additional boundaries where trust does not apply: admin workstations push configs
via OP_upgradeConfig (§6), config files may be hand-carried on USB drives for
air-gapped provisioning, and versioned files persist on disk across reboots where
bit rot or operator error (copying the wrong file) are risks. SHA-256 provides
end-to-end integrity across all of these paths without adding a separate checksum
mechanism — the same hash that detects version drift also validates integrity.
The implementation cost is negligible: OpenSSL is already initialised in Phase 1,
and SHA-256 of a <100 KB blob takes microseconds.
SHA-256 is computed over the raw bytes of the cluster.bin.* file — not over any
in-memory struct:
- The hash is never stored inside the Cap'n Proto payload (avoiding circularity).
- The first 8 bytes (UInt64, big-endian as raw SHA-256 bytes; Cap'n Proto wire format is little-endian — be64toh() converts from big-endian SHA-256 bytes to host order) serve as the fast "version tag" for change detection and the versioned filename suffix (first 12 hex chars).
- The full 32 bytes are sent in the protocol for the standby to verify cryptographic integrity of the received blob.
- Since config is always written to a new versioned file (never overwritten in place,
see §3.2), the file content is immutable after creation.
The primary's cached
config_sha256is always consistent with the bytes on disk. - The primary recomputes SHA-256 over the bytes read at request time and compares to the cached hash before sending — any discrepancy is logged as an error rather than silently sending mismatched data.
4.2 Zero-tag sentinel
A zero version_tag (UInt64) is the "no config" sentinel. A standby with no saved
config sends version_tag = 0. The probability of a legitimate config producing a
zero first 8 bytes of SHA-256 is ~2^-64 — negligible. No special-case FATAL is required.
4.3 SHA-256 API
Available via SHA256() from OpenSSL (<openssl/sha.h>), which is already initialised
in Phase 1 (initialize_openssl()). No new dependency. The hash is computed once at
startup and cached; the per-request recomputation is a safety double-check, not the
primary path.
4.4 Monotonic generation counter for auditing
SHA-256 alone prevents accidental mismatches but does not prevent downgrade — an
admin could push an older valid config blob (different SHA-256) and the primary would
accept it as an "upgrade." ClusterConfig carries a monotonic
configGeneration field (UInt64, default 1 on first save) as an audit
indicator, not an enforcement mechanism:
- On each successful
OP_upgradeConfig, the primary setsconfig_generation = c->myconf->config_generation + 1in the decoded struct, re-encodes the blob, and saves — so the generation strictly increases. - The admin tool does not need to manage the generation counter — the primary
is the sole authority and always increments. The
configGenerationfield in the incoming blob is ignored; the primary overwrites it withcurrent + 1before saving (§6.4 step 2b). The primary does not reject incoming blobs based on their embedded generation value —configGenerationprovides an audit trail (visible in the Cap'n Proto blob for post-hoc inspection) but does not mechanically prevent an operator from pushing an older config with a different SHA-256. Operators who need downgrade prevention should compare generation values out-of-band before callingOP_upgradeConfig. - Crash recovery: the generation is embedded in the
cluster.binCap'n Proto blob, so it survives restarts without a sidecar file. - Default value:
configGenerationis set to 1 byload_primary_config()on first load. If an existing config blob does not include the field (backward compat), the primary treats it as generation 0 and re-encodes with generation 1 on the next save.
Replay behaviour: a retried OP_upgradeConfig with identical config content
is accepted again — the primary re-encodes with an incremented generation, producing
a new SHA-256, and restarts. This is not idempotent: applying the same config
bytes twice produces different on-disk state (generation counter is incremented
both times) and triggers two restarts, not one. Systemd's StartLimitBurst=3
(§11.1) bounds restart storms to 3 within 60 seconds; the admin tool's retry
logic must implement deduplication (e.g., comparing the SHA-256 of the just-pushed
blob against the primary's reported versionTag in the reply) rather than blind
retry. The generation counter aids auditing of an operator accidentally pushing a stale
config fetched from another controller: the generation field embedded in the blob
provides a human-readable version indicator for auditing, even though the primary
does not reject blobs based on their generation value.
5. OP_replConfig Protocol
5.1 Schema additions
# controller.capnp
enum OPCode {
...existing entries...
replCatchUpDone @28;
replConfig @29; # ← new: standby → primary config pull
replConfigPush @30; # ← new: primary → standby config push (after OP_upgradeConfig)
upgradeConfig @31; # ← new: admin → primary live config upload
}
struct ReplConfigRequest $C.mapname("repl_config_req_t") {
versionTag @0: UInt64 $C.mapname("version_tag"); # 0 = no config saved
}
struct ReplConfigReply $C.mapname("repl_config_reply_t") {
upToDate @0: Bool $C.mapname("up_to_date"); # true = tag matched, no data
versionTag @1: UInt64 $C.mapname("version_tag"); # primary's tag (0 if upToDate)
sha256 @2: Data $C.mapname("sha256"); # full 32-byte SHA-256 (empty if upToDate)
bytes @3: Data $C.mapname("bytes"); # raw cluster.bin bytes (empty if upToDate)
}
struct ReplConfigPushRequest $C.mapname("repl_config_push_req_t") {
versionTag @0: UInt64 $C.mapname("version_tag"); # primary's new tag
sha256 @1: Data $C.mapname("sha256"); # full 32-byte SHA-256
bytes @2: Data $C.mapname("bytes"); # raw cluster.bin bytes
}
struct ReplConfigPushReply $C.mapname("repl_config_push_reply_t") {
accepted @0: Bool $C.mapname("accepted"); # true = saved and verified
versionTag @1: UInt64 $C.mapname("version_tag"); # standby's tag after save
}
struct UpgradeConfigRequest $C.mapname("upgrade_config_req_t") {
bytes @0: Data $C.mapname("bytes"); # raw cluster.bin blob
sha256 @1: Data $C.mapname("sha256"); # caller-computed SHA-256 (primary verifies)
}
struct UpgradeConfigReply $C.mapname("upgrade_config_reply_t") {
versionTag @0: UInt64 $C.mapname("version_tag"); # new tag if accepted; 0 on error
error @1: Text $C.mapname("error"); # non-empty if rejected
ackedCount @2: UInt32 $C.mapname("acked_count"); # number of standbys that ACKed push
}
Add union arms to Request.body and RequestReply.data with explicit field numbers:
replConfigRequestMsg @22: ReplConfigRequest,
replConfigReplyMsg @8: ReplConfigReply,
replConfigPushRequestMsg @23: ReplConfigPushRequest,
replConfigPushReplyMsg @9: ReplConfigPushReply,
upgradeConfigRequestMsg @24: UpgradeConfigRequest,
upgradeConfigReplyMsg @10: UpgradeConfigReply.
The field numbers follow the next free slots in each union
(after hostctrlmsg @21 in Request.body,
after clusterconf @7 in RequestReply.data).
Verified: The OPCode enum currently ends at replCatchUpDone @28.
The Request.body union ends at hostctrlmsg @21.
The RequestReply.data union ends at clusterconf @7.
All proposed field numbers are free as of the current controller.capnp schema.
5.2 Message flow
Pull (standby-initiated, at startup):
Standby Primary
│ │
│── OP_replConfig { version_tag=0xA3... } ─▶│
│ │ compare: 0xA3 != 0xF1
│◀── reply { up_to_date=false, │
│ versionTag=0xF1..., │
│ sha256=..., bytes=... } ───────│
│ │
│ validate: SHA256(bytes) == sha256 ✓ │
│ write cluster.bin.f1e2d3c4a5b │
│ update conf/current pointer file │
│── OP_replConfig { version_tag=0xF1 } ───▶│ (optional confirm)
│◀── reply { up_to_date=true } ─────────────│
The confirmation request is optional but provides assurance that the standby's saved copy matches. Implementations may skip it.
If tag matches on first request:
│── OP_replConfig { version_tag=0xF1 } ───▶│
│◀── reply { up_to_date=true } ─────────────│
│ proceed to cluster_init() phase 2 │
Push (primary-initiated, after OP_upgradeConfig — see §6):
Primary Standby
│ │
│── OP_replConfigPush { versionTag=0xF2, │
│ sha256=..., bytes=... } ─────────────▶│
│ │ validate SHA-256
│ │ write cluster.bin.f2...
│ │ update conf/current
│◀── reply { accepted=true, │
│ versionTag=0xF2 } ─────────────│
For the push direction the primary sends an OP_replConfigPush request (a
dedicated opcode, not a reused reply) to each connected standby after saving
a new config via OP_upgradeConfig. The standby validates, saves, and ACKs
with a ReplConfigPushReply confirming the new tag. This preserves the
existing request-reply dispatch pattern in myreq_handler().
5.3 Authentication model
OP_replConfig has no application-layer authentication beyond SSL transport-layer
encryption. Any controller that completes an SSL handshake (using a valid cert signed
by the shared CA) can request the full cluster config blob. This is consistent with the
trusted-LAN model adopted by the parent HA design — controller-to-controller sessions are
authenticated at the SSL layer, not at the application protocol layer. No additional
auth token, HMAC, or challenge-response is added here.
OP_upgradeConfig (§6) is more sensitive — it changes the cluster configuration and
triggers a primary restart. Access control for OP_upgradeConfig will be enforced
through the existing validate_host() mechanism (currently a stub) when it is
enhanced — the upgrade handler calls validate_host() to check whether the peer
is authorised before processing the request.
Safety gate until validate_host() is implemented: Since validate_host() is
a stub that returns success for all peers, OP_upgradeConfig is gated behind a
compile-time flag CONF_ENABLE_UPGRADE_CONFIG (default: 0, disabled). Set to 1
in the Meson build options (-Denable-upgrade-config=true) when the organisation
has implemented validate_host() and is ready for the feature. This prevents a
half-implemented admin endpoint from being reachable in production before its
authorisation check is complete. The compile-time gate is removed once
validate_host() has a production implementation (tracked in FUTURE_WORK.md).
5.4 Primary handler
The primary handles OP_replConfig via a new case in cluster.c:myreq_handler() (the
existing opcode dispatch switch). The handler is distinct from handle_controller_join()
which handles the join handshake.
Controller-controller join handshake (handle_controller_join() in join.c):
- The standby opens a TCP connection to
primary_addr:primary_port(from--peers) and completes an SSL handshake (TLS transport per §9.0ssl_ctx). - The standby sends
OPCode_joinwithHostRole_controllerand its own host identity (hostname from--peers). This identifies the session as a controller-controller (not client) connection. - The primary's join handler verifies the peer is a known controller (listed in
--peers) and returnsOPCode_joinreply with status 0. - After successful join, the primary appends the endpoint to
c->ha.standby_endpoints(replacing the current-ENOSYSstub — see §6.3). The standby proceeds to dispatchOP_replConfigon the same connection to fetch its config.
The primary must have completed Phase 2 (iomgr + endpoint up) before it can
serve OP_replConfig or accept join handshakes. The standby retry policy (§5.5)
handles this startup timing gap.
static int handle_config_request(cluster_t *c, endpoint_t *ep,
xce_request_t *req)
{
uint64_t their_tag = req->data.replConfigRequestMsg->version_tag;
uint64_t my_tag = c->myconf->config_version_tag;
xce_reply_t reply;
init_reply_by_req(&reply, req);
if (their_tag == my_tag) {
reply.status = 0;
reply.data.replConfigReplyMsg->up_to_date = true;
xce_reply(ep, c->myhost, &reply);
return 0;
}
/* Read raw bytes from conffile and recompute SHA-256 for double-check */
uint8_t *bytes;
size_t len;
int rc = conf_read_raw(c->myconf->conffile, &bytes, &len);
if (rc != 0) {
reply.status = rc;
xce_reply(ep, c->myhost, &reply);
return rc;
}
/* Recompute to guard against external pointer-file or versioned-file replacement */
uint8_t recomputed[32];
SHA256(bytes, len, recomputed);
if (memcmp(recomputed, c->myconf->config_sha256, 32) != 0) {
ERROR("Config file changed since startup — refusing to send stale hash");
free(bytes);
reply.status = -EIO;
xce_reply(ep, c->myhost, &reply);
return -EIO;
}
reply.status = 0;
reply.data.replConfigReplyMsg->up_to_date = false;
reply.data.replConfigReplyMsg->version_tag = my_tag;
reply.data.replConfigReplyMsg->sha256 = /* recomputed, 32 */;
reply.data.replConfigReplyMsg->bytes = /* alloc_and_copy(bytes, len) */;
xce_reply(ep, c->myhost, &reply);
free(bytes);
return 0;
}
The full 32-byte SHA-256 and raw config bytes are Data fields in the capnp schema.
The codecgen generates encode_ReplConfigReply_ptr() / decode_ReplConfigReply_ptr()
which internally allocate and manage the Data blob storage in the capn segment.
The C-side shim in src/lib/request.c calls these generated functions. See §12 for all
affected files.
conf_read_raw() and CONF_MAX_SIZE (defined in config_repl.c) are the canonical
entry points for reading config bytes — all callers that need raw config bytes (primary
handler, load_primary_config(), conf_save_config()) go through conf_read_raw(),
which enforces the size limit before allocating memory. conf_read_raw() returns
-EFBIG when the file exceeds CONF_MAX_SIZE — callers in request-handling context
(the primary handler) send an error reply to the peer; only load_primary_config()
(at startup) escalates this to FATAL.
c->myconf->config_version_tag and c->myconf->config_sha256 are computed once in
load_primary_config() and cached in myconf_t:
typedef struct {
cluster_config_t* cluster_config;
char* conffile;
char* certfile;
char* keyfile;
char* workdir;
char* name;
xce_cluster_config_t peers_cfg;
uint64_t config_version_tag; /* ← new: first 8 bytes of SHA-256 */
uint8_t config_sha256[32]; /* ← new: full SHA-256 of conffile bytes */
int conf_keep_n; /* ← new: config version retention count */
uint64_t config_generation; /* ← new: monotonic anti-downgrade guard (§4.4) */
} myconf_t;
5.5 Standby fetch
The standby makes a request using a standalone SSL_CTX created from the same cert/key
pair during Phase 1. A dedicated pthread performs a non-blocking connect() with
poll() to enable early exit when the election determines a role before the connect
completes. This is the same mechanism used by the election probes (§9):
- Create standalone
SSL_CTXfromcertfile/keyfile(already loaded in Phase 1). - Non-blocking
socket()+connect()to the primary candidate. poll()with timeoutprobe_connect_timeout_ms(500ms), pollingelection_donebetween attempts.- Once connected,
SSL_connect()withSSL_ERROR_WANT_READ/WANT_WRITEhandling. - Send
OP_replConfig, receive reply, join thread.
No iomgr dependency — this runs after am_i_primary() resolves to STANDBY and before
Phase 2 (which starts iomgr). The non-blocking connect + poll + SSL_connect approach is
the same mechanism used by the election probes (§9) — the config fetch and election probe
code paths share this pattern deliberately so both benefit from the same error-handling
and timeout logic.
Retry policy: if the primary is not yet ready (iomgr not up, still in Phase 2
initialisation including persist_recover()), the TCP connection is refused or the
request times out. The standby retries with exponential backoff:
initial_delay_ms = 500
max_delay_ms = 5000
max_attempts = 20
delay = min(initial_delay_ms * 2^attempt, max_delay_ms)
total window ≈ 88s (500+1k+2k+4k+16×5k = 87.5s)
The retry window is sized to accommodate the primary's Phase 2 startup time
(§8.5), specifically persist_recover() bounded by max_wal_records (see §13).
The primary must complete Phase 2 startup (iomgr, persist_recover, endpoint up)
before it can serve OP_replConfig; standbys retry during this window.
Note: iomgr_init and endpoint_init() complete early in Phase 2 (before
persist_recover()), so client connections can be dispatched before config
state is fully recovered. The OP_replConfig handler depends on config being
loaded and cached in myconf (populated before Phase 2), not on
persist_recover() completion — the handler reads from myconf, not from
WAL-recovered state. Post-recovery hooks (HOOK_AFTER_EVENT) do not affect
config fetch latency.
Each retry iteration checks atomic_load(&c->ha.role) == CTRL_STANDBY before
attempting the connect. If the HA watchdog has promoted this node to primary
(e.g., the original primary was detected as failed during the retry window),
the fetch thread logs INFO("Promoted to primary during config fetch — " "aborting fetch") and returns without error. The caller proceeds with
load_primary_config() on the primary path.
If all attempts fail, the standby checks whether conf/current points to a valid
versioned file:
- Yes (saved config exists): log WARN and proceed — see §10.3.
- No (no saved config): treat as FATAL — see §10.2.
6. OP_upgradeConfig Protocol
6.1 Purpose
OP_upgradeConfig allows an admin tool to upload a new ClusterConfig blob to the
running primary at runtime. The primary validates (SHA-256 transit integrity, Cap'n Proto
decode), increments the monotonic generation counter, saves, pushes to all connected
standbys (best-effort), then restarts itself to apply the new configuration.
6.2 Message flow
Admin tool Primary Standbys (connected)
│ │ │
│─ OP_upgradeConfig▶│ │
│ (bytes, sha256) │ 1. verify SHA256(bytes) │
│ │ 2. decode Cap'n Proto │
│ │ 3. set gen=current+1, re-encode│
│ │ 4. write cluster.bin.<newhex> │
│ │ 5. update conf/current │
│ │ 6. update config_version_tag │
│ │ 7. push to connected standbys │
│ │── OP_replConfigPush ──────────▶│
│ │ (versionTag, sha256, bytes) │ save to disk
│ │◀─ reply { accepted, tag } ─────│
│◀─ ACK (new tag, acked_count) ──│ │
│ │ │
│ │ │
│ │ 8. log "config upgraded, │
│ │ requesting restart" │
│ │ 9. set restart_requested=1 │
│ │ cluster_run() exits loop │
│ │ myapp_run() returns 0 │
│ ↓ │
│ [process exits; systemd/service manager │
│ restarts with same args and new config]│
6.3 Push semantics (hash-first, best-effort, parallel)
The primary pushes the new config's hash and tag to all connected standbys before
ACKing the admin tool. This reduces the silent-rollback window: the admin receives
the ACK only after at least attempting to notify all connected standbys. The push
is best-effort and parallel: all standbys are sent the config simultaneously,
and the primary waits at most CONF_PUSH_TIMEOUT_MS (default 500 ms, compile-time
constant in config_repl.c) total for all ACKs. This bounds the pre-restart delay
to a single timeout period regardless of standby count. The 500ms default is derived
from the expected push workload: a single SSL write (config blob < 1 MiB) + standby
SHA-256 verify + disk fsync + SSL ACK reply, bounded by two intra-LAN RTTs (~1ms
each) + one local fsync (~50ms). The per-standby worst case is ~55ms; 500ms
accommodates up to 9 concurrent standbys or a single degraded standby.
Push-then-ACK ordering: The primary saves the new config, pushes to standbys,
then ACKs the admin tool. The ACK includes the number of standbys that confirmed
the push (acked), so the admin tool can detect when zero standbys received the
update. The ACK reply is extended with:
struct UpgradeConfigReply $C.mapname("upgrade_config_reply_t") {
versionTag @0: UInt64 $C.mapname("version_tag"); # new tag if accepted; 0 on error
error @1: Text $C.mapname("error"); # non-empty if rejected
ackedCount @2: UInt32 $C.mapname("acked_count"); # number of standbys that ACKed push
}
If ackedCount == 0 and standbys are expected, the admin tool logs a WARNING
that the config may not have reached any standby.
/* Fan-out: launch one pthread per connected standby */
for (int i = 0; i < n_standbys; i++)
pthread_create(&push_threads[i], NULL, push_config_thread, &push_ctx[i]);
/* Single aggregate wait — 500ms total, not 500ms per standby. */
atomic_int push_done;
atomic_init(&push_done, 0);
int acked = wait_all_with_timeout(threads, ctxs, n,
CONF_PUSH_TIMEOUT_MS, &push_done);
LOG("Config pushed to %d/%d standbys; restarting", acked, n_standbys);
Implemented signature (in config_repl.c):
static int wait_all_with_timeout(pthread_t *threads, push_ctx_t *ctxs,
int n, uint32_t timeout_ms, atomic_int *done);
Uses a 5ms poll loop against ctx->done flags until the CLOCK_MONOTONIC deadline, then sets *done = 1 and joins all started threads. The done flag allows thread implementations that check it to exit early; push_thread sets ctx->done = 1 on natural completion, so joins return immediately for threads that finished within the timeout.
Offline standbys recover on their next reconnect via the normal pull mechanism
(§5.2) — their conf/current will have the old tag, the new primary will detect
the mismatch and push via OP_replConfig.
Failure mode — push not received + primary fails to restart: If push to all
standbys times out (e.g., all standbys offline) AND the primary fails to come back
after restart (e.g., the new config triggers a Phase 2 FATAL), the HA watchdog on the
standbys promotes one of them. The promoted standby boots with its saved config
— which is the old version, since it never received the push. The cluster silently
rolls back to the previous config. The admin tool now receives ackedCount == 0 in
the ACK reply, providing a clear signal that no standby received the update. If
ackedCount > 0, at least one standby has the new config and will have it available
when it promotes. The silent rollback scenario is limited to ackedCount == 0 cases
where all standbys were offline during the push. Future work may add a quorum-ACK
gate that refuses to restart unless a minimum number of standbys confirm (§14.4).
Standby endpoint tracking: The primary maintains a list of connected standby
controller endpoints in ha_state_t as a zarray_t *standby_endpoints (using the
existing zarray_t infrastructure, which already manages c->endpoints for
client connections). This is Phase 2 infrastructure — needed only for
OP_replConfigPush fan-out during live config upgrade. This list is populated
and pruned by the join/disconnect handlers:
-
Join:
handle_controller_join()(join.c) appends the new controller endpoint toc->ha.standby_endpointsafter completing the join handshake. This replaces the current-ENOSYSstub. -
Disconnect:
handle_endpoint_error()removes the endpoint fromstandby_endpointswhen a controller-controller connection drops. The existingHostRole_controllercase inhandle_endpoint_error()(cluster.c) must be updated to scanc->ha.standby_endpointsfor the matching endpoint and callzarray_del()before thebreak. -
Usage:
push_config_to_standbys()iteratesstandby_endpointsto obtainn_standbysand the per-standby context for the fan-out push threads.Thread safety:
push_config_to_standbys()takes a snapshot ofstandby_endpointsunderc->ha.lockbefore spawning push threads. This prevents a concurrent join or disconnect handler from modifying the array while threads are iterating it. The snapshot is a shallow copy of endpoint pointers into a stack-allocatedzarray_t; endpoints pinned by active push threads are protected fromendpoint_free()by the endpoint's internal refcount. After all push threads complete, the snapshot array is freed.
Only endpoints that completed the SSL handshake and join handshake are in this list — the SSL layer provides authentication (per §5.3 trusted-LAN model).
6.4 Handler (primary side)
static int handle_upgrade_config(cluster_t *c, endpoint_t *ep,
xce_request_t *req)
{
const uint8_t *bytes = req->data.upgradeConfigRequestMsg->bytes.ptr;
size_t len = req->data.upgradeConfigRequestMsg->bytes.len;
const uint8_t *sha256 = req->data.upgradeConfigRequestMsg->sha256.ptr;
/* 0. Reject oversized blobs before hashing */
if (len > CONF_MAX_SIZE) {
send_upgrade_error(ep, req, "Config blob exceeds CONF_MAX_SIZE");
return -EINVAL;
}
/* 0b. Validate input field lengths before accessing pointers.
* Cap'n Proto Data fields with default/empty value have ptr=NULL,
* len=0. Gated behind CONF_ENABLE_UPGRADE_CONFIG (default 0, §5.3). */
if (len == 0 || req->data.upgradeConfigRequestMsg->sha256.len != 32) {
send_upgrade_error(ep, req, "Invalid request: bytes or sha256 field missing");
return -EINVAL;
}
/* 0a. Authorisation check — gated behind CONF_ENABLE_UPGRADE_CONFIG
* (default 0, §5.3). validate_host() is a stub that returns
* success for all peers until enhanced with a real check.
* When the compile-time gate is removed, this call is mandatory. */
if (!validate_host(ep, c)) {
send_upgrade_error(ep, req, "Not authorised — peer not in allowed hosts");
return -EPERM;
}
/* 1. Verify caller-supplied SHA-256 */
uint8_t computed[32];
SHA256(bytes, len, computed);
if (memcmp(computed, sha256, 32) != 0) {
send_upgrade_error(ep, req, "SHA-256 mismatch — blob corrupted in transit");
return -EINVAL;
}
/* 2. Basic Cap'n Proto decode to catch malformed blobs */
cluster_config_t *new_cfg = decode_cluster_config(bytes, len);
if (!new_cfg) {
send_upgrade_error(ep, req, "Failed to decode ClusterConfig blob");
return -EINVAL;
}
/* 2.5 Minimum viability check — rejects trivially invalid configs
* that would cause Phase 2 FATAL on restart (§10.7 semantic validation
* is a non-goal; this only catches the most basic misconfiguration). */
if (new_cfg->queueCount == 0) {
free_cluster_config(new_cfg);
send_upgrade_error(ep, req, "Config requires at least one queue");
return -EINVAL;
}
if (new_cfg->port < 1024 || new_cfg->port > 65535) {
free_cluster_config(new_cfg);
send_upgrade_error(ep, req, "Config port out of valid range (1024–65535)");
return -EINVAL;
}
/* 3. Re-encode with incremented generation and recompute SHA-256.
* The primary is the sole authority for generation numbers — the admin
* tool's configGeneration field is ignored; we always use current+1. */
uint64_t new_generation = c->myconf->config_generation + 1;
new_cfg->config_generation = new_generation;
uint8_t *new_bytes = NULL;
size_t new_len = 0;
int rc = encode_cluster_config(new_cfg, &new_bytes, &new_len);
free_cluster_config(new_cfg);
if (rc != 0) {
send_upgrade_error(ep, req, "Failed to re-encode config");
return rc;
}
SHA256(new_bytes, new_len, computed);
/* Use computed (fresh SHA-256) for tag and save; caller-supplied sha256
* was validated for transit integrity above and is no longer needed. */
/* 4–6. Save and update pointer */
char hex_tag[13];
bytes_to_hex(computed, 6, hex_tag); /* 6 source bytes → 12 hex chars + NUL */
rc = conf_save_config(c->myconf, new_bytes, new_len, computed);
if (rc != 0) {
free(new_bytes);
send_upgrade_error(ep, req, "Failed to save new config");
return rc;
}
/* Update conffile to point to the newly saved versioned file.
* Without this, handle_config_request() (§5.4) would read the old
* versioned file and its recomputed SHA-256 would mismatch
* c->myconf->config_sha256 (already updated below), causing
* spurious -EIO replies to standbys during the restart window. */
free(c->myconf->conffile);
asprintf(&c->myconf->conffile, "%s/conf/cluster.bin.%s",
c->myconf->workdir, hex_tag);
uint64_t new_tag;
memcpy(&new_tag, computed, 8);
/* stored big-endian; swap to host order */
new_tag = be64toh(new_tag);
c->myconf->config_version_tag = new_tag;
memcpy(c->myconf->config_sha256, computed, 32);
c->myconf->config_generation = new_generation;
/* 7. Best-effort parallel push to connected standbys.
* Push runs BEFORE ACK so the admin tool knows how many standbys
* received the update. The reply includes acked_count. If the
* primary crashes before push, standbys catch up on their
* next reconnect via the normal pull path (§5.2). */
int acked = push_config_to_standbys(c, new_bytes, new_len, computed, new_tag);
free(new_bytes);
/* 8. ACK the admin — includes acked_count so the admin tool knows
* how many standbys confirmed the push. A zero acked_count with
* standbys expected is a WARNING condition (see §6.3). */
send_upgrade_reply(ep, req, new_tag, acked);
/* Flush the reply to the kernel TCP buffer before signaling restart.
* Without this, the iomgr shutdown path may close the endpoint before
* the ACK bytes are flushed, causing a connection reset. The admin
* tool would be unable to distinguish success from failure. */
endpoint_flush(ep); /* block until kernel send buffer is drained */
/* 9. Request restart — controller is managed by systemd or equivalent
* service manager. cluster_run() exits its event loop when this flag is
* set, myapp_run() returns cleanly, and the service manager restarts
* the process with the same arguments. On restart, conf/current points
* to the new config versioned file. No execve() or saved-argv needed.
*
* IMPORTANT: The main event loop blocks on cbuf_get(&c->wbuf, &stop)
* (mylock.c pthread_cond_wait). Setting restart_requested alone cannot
* unblock it. We push a stop signal via cbuf_put — the same pattern
* used by the existing OPCode_shutdown handler (cluster.c:152-157).
* cluster_run() checks restart_requested after the loop exits to
* distinguish a clean config-upgrade restart from a shutdown. */
/* peer_cn is the X.509 certificate Common Name — sanitise before logging
* to prevent log injection via crafted cert CN containing newlines or
* control characters. Limit to 64 printable ASCII chars. */
LOG("Config upgraded to tag=%016"PRIx64" by CN=%s from %s:%u — requesting restart",
new_tag, sanitise_cn(peer_cn, 64), peer_ip, peer_port);
c->restart_requested = 1;
int stop = 1;
cbuf_put(&c->wbuf, &stop); /* unblock cluster_run() main loop */
return 0;
}
6.5 WAL compatibility after config change
persist_recover() replays WAL records against the in-memory state built from the
new config. If the new config removes a queue or resource whose ID appears in
existing WAL records, replay encounters an unknown ID.
Distinction from existing skip-on-unknown-type behavior
The existing persist_recover() already skips WAL records whose type is unknown
(a forward-compatibility provision for format extensions). This design adds a
separate handling strategy for ID references within known record types:
| Condition | Existing behavior | This change |
|---|---|---|
| Unknown WAL record type | WARN + skip (safe — future format) | Unchanged |
| Known record type, unknown queue ID | Not checked today | Route to lost_and_found queue |
| Known record type, unknown resource/host ID | Not checked today | Route to lost_and_found host or PEND |
Unknown-type records are safe to skip because the type may be from a newer format. Unknown-ID references were previously treated as FATAL, but this made config upgrade operationally fragile — a single zombie job referencing a removed resource would brick the controller permanently (it cannot start because of FATAL, and it cannot upgrade config because that would also FATAL).
Routing strategy for unknown IDs
Instead of FATAL, apply_wal_entry() routes unknown-ID references as follows:
-
Unknown
queue_id: The job is placed in a built-inlost_and_foundqueue. Thelost_and_foundqueue is always present in the queue table regardless of config content. Operators can inspect it after recovery and reassign or cancel jobs at their discretion. -
Unknown
host(node name): The job's host is set tolost_and_found, a synthetic host entry that accepts jobs but marks them as requiring operator attention. -
Unknown
resource_id: The job's resource assignment is cleared and the job is set toPENDstate. The job will pend until an operator re-creates the resource in the config or cancels the job. A WARN log identifies the orphaned resource reference:
WARN("WAL record references unknown queue id=%u at offset %zu in %s — "
"routing job to lost_and_found queue",
queue_id, record_offset, wal_path)
WARN("WAL record references unknown resource id=%u at offset %zu in %s — "
"clearing resource assignment, job set to PEND",
resource_id, record_offset, wal_path)
WARN("WAL record references unknown host '%s' at offset %zu in %s — "
"routing to lost_and_found host",
host_name, record_offset, wal_path)
lost_and_found queue and host
These are synthetic entries created by cluster_init() Phase 2 after kvmgr makeInstance
and before persist_recover() (see startup diagram in §7). This ordering
is load-bearing: unknown queue/resource/host IDs encountered during persist_recover() must
have valid targets to route to. Consider adding a runtime assertion in create_lost_and_found()
that validates kvmgr is initialized and persist_recover() has not yet been called. They are not part of the admin-supplied
config blob — they are added by the controller itself. This ensures they always
exist, even after a config change that removes all user-defined queues or hosts.
Their existence is not persisted across restarts; they are re-created on each boot.
Implementation: create_lost_and_found() adds entries to kvmgr's queue table and
host table via the existing queue_insert() and host_insert() APIs, using reserved
names "lost_and_found". These are normal kvmgr entries — no special-casing in the
queue/host lookup paths is required. Name reservation: the name "lost_and_found"
is permanently reserved. The OP_upgradeConfig handler rejects any ClusterConfig
blob containing a queue or host named "lost_and_found" — the admin tool receives
an error: "Reserved name: 'lost_and_found' queue/host names are reserved by the controller. Rename the user-defined entry and retry." This prevents the ambiguity
between user-defined and synthetic entries at config ingestion time rather than at
WAL replay time. create_lost_and_found() always creates the synthetic queue and
host; if the name somehow exists (e.g., loaded from a pre-upgrade saved config),
it skips creation and logs ERROR. After creation, standard lookup functions
(get_queue_by_name(), get_host_by_name()) resolve them.
Enforcement at all entry points: Both load_primary_config() (primary at startup, §7.1)
and handle_upgrade_config() (live upgrade, §6.4) perform the same config_contains_reserved_name()
check. This ensures the synthetic lost_and_found queue and host cannot be aliased with a
user-defined entry regardless of how config enters the system — cold boot or live push.
The check is implemented once in config_repl.c and called from both paths.
Routing helper: A resolve_queue_or_lost_and_found(cluster_t *c, uint32_t queue_id)
helper in config_repl.c is called inside apply_wal_entry() (persist.c) and
restore_job() (job.c) when a WAL record references an unknown queue ID. The helper:
- Looks up
queue_idin the queue table. - If found, returns the queue pointer.
- If not found, logs a WARN (with record offset and WAL path, per §6.5 examples) and
returns the
lost_and_foundqueue pointer.
Analogous helpers for host resolution (resolve_host_or_lost_and_found()) and resource
resolution (clear_resource_and_pend()) follow the same pattern. All three are called
from the WAL apply path in persist_recover() → apply_wal_entry().
Observability: The lost_and_found queue and host are discoverable via
xcecli queue list and xcecli host list — they appear as normal entries with
the reserved name "lost_and_found". A metric counter lost_and_found_job_count
is incremented each time a job is routed to this queue during persist_recover().
If lost_and_found_job_count > 0 at startup, a WARN summary is logged. The
counter is stored in cluster_t as an atomic_int, initialised to 0 at
startup and incremented during persist_recover(). It is not persisted across
restarts. The counter is reset to 0 after a successful drain operation
(e.g., the future xcecli lost-and-found drain command reassigns or cancels
all sidelined jobs). Jobs accumulate across config upgrades (each upgrade writes new versioned files but
the WAL is truncated after checkpoint — see §13.4), so repeated upgrades do not
cause unbounded growth. Operators should inspect and drain lost_and_found after
a config upgrade; a future xcecli lost-and-found drain command can automate
reassignment or cancellation (§14.4 future work).
Job submission gate: The API server (src/frontend/apiserver) MUST reject job
submissions targeting the lost_and_found queue with a clear error: "Cannot submit jobs to 'lost_and_found' — this is an internal recovery queue." The
queue appears in xcecli queue list for operator visibility but does not accept
external job submissions. This prevents accidental routing of production jobs to
the synthetic recovery queue and keeps lost_and_found_job_count meaningful
(only incremented for genuine WAL-recovered orphans, not operator mistakes).
The check is enforced at the API layer (apiserver), not in the controller, so
standby-only deployments without the API server still have the controller-side
protection of §6.5's name reservation.
6.6 Downtime estimate for config upgrade
The primary is unavailable from when it sets restart_requested until Phase 2
completes on restart: ~4–7 seconds nominal, ~8–13 seconds with 1000 jobs (per
§8.5 timing table). This is the same duration as a cold start — the process exits
cleanly and systemd restarts it with the same arguments.
Standbys do not restart during config upgrade — they receive the new config via
OP_replConfigPush and save it to disk, but continue running with their current
in-memory state. If the primary fails to come back (e.g., the new config is corrupt),
the HA watchdog on the standbys will detect the primary's absence and trigger an
election. The new primary then boots with the already-saved config.
The admin tool receives the ACK before the restart begins (§6.4 step 6), so the
caller knows the config was accepted within probe_connect_timeout_ms (500ms for
push fan-out + network round-trip). The restart itself is asynchronous — the admin
tool should poll OP_probe or check the primary's log to confirm it's back online.
6.7 In-flight WAL checkpoint during config-upgrade restart
If an async WAL checkpoint (§13.4) is in progress when handle_upgrade_config()
triggers restart, the restart must not interrupt the checkpoint mid-write (which
could leave a partial snapshot file on disk).
Handling: before setting restart_requested, the handler waits for any
in-flight checkpoint to complete:
/* Wait for in-flight checkpoint before restarting.
* schedule_checkpoint() is async via vtaskmgr; the checkpoint task sets
* persist->checkpoint_done when finished (same atomic used by the
* idempotency guard in schedule_checkpoint()). */
struct timespec ckpt_wait_start, ckpt_wait_now;
clock_gettime(CLOCK_MONOTONIC, &ckpt_wait_start);
while (atomic_load(&c->persist->checkpoint_in_progress)) {
usleep(10000); /* 10ms spin — checkpoint is I/O-bound, completes quickly */
clock_gettime(CLOCK_MONOTONIC, &ckpt_wait_now);
if ((ckpt_wait_now.tv_sec - ckpt_wait_start.tv_sec) > 30) {
ERROR("Checkpoint did not complete within 30s — restarting anyway");
break;
}
}
This is safe because:
- The checkpoint is bounded: it flushes in-memory state and truncates the WAL, completing in well under 1 second on typical hardware.
- The spin-wait does not hold any locks —
handle_upgrade_config()runs on the admin connection's thread, not the main event loop. - If the checkpoint hangs (bug), the 30s timeout unblocks the restart; the controller is not permanently stuck. The admin tool's connection timeout provides a second bound.
- After the checkpoint completes, the WAL is empty (truncated at checkpoint),
so the next
persist_recover()on restart replays zero records — the fresh snapshot already contains all state.
7. Startup Sequence
The startup sequence for both roles is updated from the current monolithic
cluster_init() to a two-phase model.
Controller starts
│
▼
Parse --peers → my_priority, my_port
│
▼
Phase 1 — minimal init (no config, no iomgr)
┌─────────────────────────────────────────┐
│ xce_hook_init + xce_install_hooks │
│ initialize_openssl │
│ ssl_ctx = create_ssl_context(cert, key) │
│ ha_init(&c->ha, &ha_cfg, my_priority) │
│ c->ha.ssl_ctx = ssl_ctx /* stash */ │
│ (c->ha.ssl_ctx and c->iomgr.ssl_ctx │
│ are the same pointer; Phase 2 calls │
│ iomgr_set_ssl_ctx(c->iomgr, ssl_ctx) │
│ for the existing iomgr path. The │
│ ha.h "no new TLS" comment is removed; │
│ see §11 ha.h) │
│ ha_install_sigusr1 │
└─────────────────────────────────────────┘
│
▼
am_i_primary()? ← real election probing (§9)
│
┌────┴────────────────────────────────────┐
│ │
PRIMARY STANDBY
│ │
▼ ▼
load_primary_config() INFO("Starting as standby
← reads conf/current (priority %d); requesting
← computes SHA-256 config from primary %s:%u",
← FATAL if missing my_priority, host, port)
│ │
│ fetch_config_from_primary()
│ ← OP_replConfig exchange
│ ← save to conf/ (§3.2)
│ ← retry with backoff (§5.5)
│ ← FATAL if no config at all
│ │
└────────────────┬────────────────────────┘
│
▼
Phase 2 — full init (config available)
┌─────────────────────────────────────┐
│ iomgr_init │
│ endpoint_init(port from --peers) │
│ HOOK_BEFORE_EVENT → load resources │
│ (must check cluster_config!=NULL │
│ and early-return — config already │
│ loaded before Phase 2) │
│ kvmgr makeInstance (sizes from cfg) │
│ xstring_table_init, job_obj_init │
│ create_lost_and_found() │
│ ← lost_and_found queue + host │
│ ← always present regardless of │
│ config content (§6.5) │
│ vtaskmgr_init, delaymgr_init │
│ persist_init │
│ ha_promote (primary) or │
│ CTRL_STANDBY (standby) │
│ HOOK_AFTER_EVENT → build queues, │
│ groups, load jobs │
│ persist_recover │
└─────────────────────────────────────┘
│
▼
cluster_start()
7.1 Primary config load — fail-fast behaviour
load_primary_config() replaces get_default_conf() on the primary path:
static int load_primary_config(cluster_t *c)
{
if (access(c->myconf->conffile, F_OK) != 0) {
FATAL("Primary candidate (priority 0) has no config file '%s' — "
"provision the config before starting the primary controller",
c->myconf->conffile);
return -ENOENT;
}
/* read, decode Cap'n Proto, compute SHA-256, cache version_tag + full hash in myconf */
...
}
/* After decoding, check for reserved "lost_and_found" name — same check
* as OP_upgradeConfig handler (§6.5). Reject at boot rather than
* silently aliasing the synthetic queue/host with user-defined entries. */
if (config_contains_reserved_name(new_cfg)) {
FATAL("Config file '%s' contains reserved name 'lost_and_found' — "
"rename the user-defined queue or host and retry",
c->myconf->conffile);
return -EINVAL;
}
A priority-0 node with a missing config file must not start — it would boot with empty queues and resources, silently losing all cluster configuration.
7.2 Standby INFO log on startup
When a standby starts and its election probe detects a live higher-priority candidate:
INFO("HA: higher-priority candidate %s:%u is alive — starting as standby "
"(my_priority=%d)", primary_host, primary_port, my_priority)
INFO("HA: requesting cluster config from primary %s:%u (my_tag=%016"PRIx64")",
primary_host, primary_port, saved_tag)
After successful config fetch:
INFO("HA: config received and saved (tag=%016"PRIx64") — proceeding with full init",
new_tag)
8. cluster_init() Phase Split
8.1 What moves to phase 1 (before config)
From the current cluster_init() (cluster.c):
| Line(s) | Code | Phase |
|---|---|---|
| 544–546 | xce_hook_init, xce_install_hooks |
1 |
| 557–561 | initialize_openssl, create_ssl_context |
1 |
| 665–671 | ha_init, ha_install_sigusr1 |
1 |
| new | zarray_create(&c->ha.standby_endpoints, 4, NULL) — initialise the standby endpoint tracking array in ha_state_t before any join handler can append to it (§6.3). Uses NULL killer (endpoints are freed by endpoint_free() called by the iomgr disconnect path, not by zarray) |
1 |
8.2 What stays in phase 2 (after config)
| Line(s) | Code | Phase |
|---|---|---|
| new (before iomgr) | Stop heartbeat listener: set atomic_store(&c->heartbeat_listener_running, 0) and pthread_join(c->heartbeat_listener_tid, NULL) — the join must complete before iomgr_init to free the listening port and prevent EADDRINUSE |
2 |
| 580 | iomgr_init |
2 |
| 589–594 | endpoint_init (port from --peers) |
2 |
| 548–552 | xce_run_hooks HOOK_BEFORE_EVENT → in xce_config_init() (cluster.c:411-417), add if (c->myconf->cluster_config != NULL) return 0; guard before the existing get_default_conf(c) call. Config is already loaded by load_primary_config() or fetch_config_from_primary() before Phase 2. The HOOK_AFTER_EVENT case (cluster.c:419-446: cluster_build_config/resources/queues/groups/load_jobs) is unchanged. The static global cf in myconf.c and its associated HOOK_AFTER_EVENT SHUTDOWN cleanup become dead code — myconf_fini() takes over freeing cluster_config. |
2 |
| 563–579 | kvmgr makeInstance (sizes from config) |
2 |
| 584–585 | xstring_table_init, job_obj_init |
2 |
| 602–608 | vtaskmgr_init, delaymgr_init |
2 |
| 614–629 | persist_init |
2 |
| 674–679 | ha_promote_to_primary or CTRL_STANDBY |
2 |
| 681–683 | xce_run_hooks HOOK_AFTER_EVENT |
2 |
| new | create_lost_and_found() — synthetic queue + host (§6.5) — must complete before persist_recover() begins; see §6.5 ordering note |
2 |
| 686–691 | persist_recover |
2 |
8.3 kvmgr sizing with default values
The kvmgr hash slot counts (resmgr->nhash_slots, jobmgr->nhash_slots, etc.) come
from ClusterConfig. The slot count only affects hash collision rate — it has no
effect on correctness. A standby that has never received config can safely use the
same defaults as fill_default_cluster_config() while waiting, then re-initialize
with the correct sizes once config arrives.
In practice, the standby receives config before phase 2 begins, so it always uses the correct sizes. The default fallback is belt-and-suspenders only.
8.4 Listening port
endpoint_init() currently reads the port from cluster_config->port. After the
split, phase 2 derives the port from the --peers candidate list (available from
CLI parsing before any phase):
/* Find own entry in peers list by name match */
uint32_t my_port = 0;
for (int i = 0; i < pcfg->candidates_count; i++) {
if (strncmp(pcfg->candidates[i].name,
c->myconf->name, XCE_MAXNAME_LEN - 1) == 0) {
my_port = pcfg->candidates[i].port;
break;
}
}
if (my_port == 0) my_port = 9990; /* design default */
This is already available in myconf.peers_cfg before any config is loaded.
xce_candidate_t (defined in src/include/xce/cluster.h) carries a port field
populated by xce_parse_peers() from the name=host:port CLI syntax — confirmed in
the struct definition at line 21 of that header. No new parsing is needed.
8.5 Phase 2 timing decomposition
The standby retry window must be sized to survive Phase 2 startup on the primary. Each Phase 2 component contributes:
| Component | Nominal | With 1000 jobs |
|---|---|---|
iomgr_init + endpoint_init |
<50ms | <50ms |
kvmgr makeInstance ×5 |
<100ms | <100ms |
xstring_table_init + job_obj_init |
<50ms | <50ms |
HOOK_BEFORE_EVENT (load resources) |
<500ms | <2s |
vtaskmgr_init + delaymgr_init + apiserver_init |
<200ms | <500ms |
persist_init |
<100ms | <200ms |
persist_recover (bounded by §13, ≤10000 records) |
2–5s | 2–5s |
HOOK_AFTER_EVENT (queues, groups, jobs) |
<1s | <5s |
| Total | ~4–7s | ~8–13s |
The ~88-second standby retry window provides 7×–11× headroom over the worst case.
Operators of very large clusters (10000+ jobs) should verify Phase 2 duration from
the primary's startup log and tune max_attempts × max_delay_ms accordingly.
9. am_i_primary() Implementation
The current am_i_primary() stub (ha.c:213) returns c->ha.my_priority == 0,
which is never correct for a multi-node cluster — it never probes whether a
higher-priority candidate is actually alive.
9.0 Simultaneous-boot symmetry breaking
When all candidates start simultaneously, nobody is primary yet and nobody responds to election probes. Without symmetry breaking, every candidate would expire its deadline and self-declare primary — a multi-way split-brain.
The mechanism has two parts:
1. Dedicated heartbeat listener (all self-declared primaries)
After Phase 1, every candidate that am_i_primary() returns true for launches a
minimal heartbeat listener thread before entering Phase 2. This is not limited to
priority-0 — if a lower-priority candidate wins the jitter race and self-declares
primary, its heartbeat listener is essential for the higher-priority true primary
to detect the split-brain during its own election probe. The listener is started
in a dedicated pthread immediately after am_i_primary() returns:
if (am_i_primary()) {
/* Start a minimal SSL listener that responds to OP_heartbeat probes.
* This lets lower-priority candidates detect us while Phase 2 initializes. */
start_heartbeat_listener(c); /* dedicated pthread — non-blocking poll loop */
load_primary_config(c); /* FATAL if missing */
...
}
The heartbeat listener is fully specified in the parent HA design
(persistence-and-ha-design.md Part 5). This section only describes the
integration point relevant to config replication: the listener must be active
before the standby's fetch_config_from_primary() connects, so the standby can
reach a primary that is still in Phase 2 initialisation. Summary of the
listener's properties (see parent design for full specification):
- Binds the port from
--peersusingc->ha.ssl_ctx - Non-blocking poll loop; accepts
OP_heartbeatonly; rejects other opcodes - Does NOT depend on iomgr, vtaskmgr, persist, or any Phase 2 component
- Exits when the main thread clears
atomic_int heartbeat_listener_running(set to 0 beforeiomgr_initin Phase 2). The main thread callspthread_join(heartbeat_listener_tid, NULL)beforeendpoint_init()to prevent EADDRINUSE. - Registered in
cluster_t(c->heartbeat_listener_tid,c->heartbeat_listener_running) socluster_fini()can join the thread on any FATAL exit path before Phase 2
2. Priority-based base delay + randomized jitter (all candidates)
Each candidate's election timeout has two components: a deterministic priority-based base delay (higher priority = shorter delay) plus a small random jitter for tie-breaking:
/* Priority-based base delay: priority-0 waits 0ms, priority-1 waits
* election_priority_step_ms, priority-2 waits 2× that, etc.
* This ensures the highest-priority node nearly always wins without probing. */
long to_ms = c->ha.config.election_timeout_ms;
to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms;
/* Small random jitter for tie-breaking between same-step candidates */
long jitter_ms = (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms);
to_ms += jitter_ms;
With election_timeout_ms = 2000, election_priority_step_ms = 500, and
election_jitter_max_ms = 200, the effective deadlines are:
- Priority-0: 2000ms–2199ms (self-declares primary first)
- Priority-1: 2500ms–2699ms (probes priority-0 first; if alive, becomes standby)
- Priority-2: 3000ms–3199ms (probes priority-0 and priority-1 first)
The priority-based base delay all but eliminates simultaneous split-brain during normal boots: priority-0 declares primary and starts its heartbeat listener ~500ms before priority-1's deadline, so priority-1's probe finds a live primary. The priority-based step and jitter together make simultaneous split-brain vanishingly unlikely for N=3 in systemd-managed deployments:
- Priority-0 fires at t=0–199ms (election_timeout_ms + jitter only)
- Priority-1 fires at t=500–699ms (step + jitter, probes priority-0 first)
- Priority-2 fires at t=1000–1199ms (2×step + jitter)
Adjacent-priority candidates collide only if the lower-priority node starts so late that its jitter-extended deadline precedes the higher-priority node's first heartbeat response, which requires the 500ms step to be less than 2×J_max (400ms) — false for the configured values. For three nodes to all self-declare primary, all three must start within the same ~200ms jitter window, which is bounded by OS scheduler precision and systemd startup ordering.
The election_priority_step_ms is added to ha_config_t alongside the existing
election_timeout_ms and election_jitter_max_ms. The jitter-only fallback
(priority-0 with no step) retains the original behaviour when election_priority_step_ms = 0.
The window is bounded: in the worst case (all jitter values 0ms apart), all candidates declare primary. Once both reach Phase 2, the HA watchdog resolves the split via priority comparison in heartbeat exchange (see parent HA design). This is bounded by the time for the slower candidate to complete Phase 2 (~4–13s per §8.5) plus one heartbeat interval (~500ms).
Split-brain window during simultaneous boot: The heartbeat listener starts
immediately after am_i_primary() returns, but it is a minimal pre-iomgr
listener that only handles OP_heartbeat. All candidates that self-declare
primary launch a heartbeat listener, not just priority-0 — this is critical:
if a lower-priority candidate wins the jitter race and self-declares primary,
its heartbeat listener provides the response that resolves the split-brain
when the higher-priority (true) primary probes it.
During simultaneous boot, all candidates start with ha.term = 0
(atomic_store_explicit(&ha->term, 0ULL, ...) in ha_init(), ha.c:34).
Term-based priority comparison cannot break symmetry at this stage — symmetry
breaking during simultaneous boot relies entirely on the jitter mechanism
(part 2 below). Term-based step-down becomes effective only after the first
successful election, when the winning primary increments its term. The jitter
window is therefore the primary defence against multi-way split-brain during
cold start; the heartbeat listener provides the detection channel but term
comparison is not yet discriminating.
If a heartbeat listener fails to bind its port (EADDRINUSE), the candidate
logs a WARN indicating a likely split-brain condition (another candidate has
already bound the port) and continues initialization. This is expected during
simultaneous boot with overlapping jitter windows — the candidate that bound
the port first keeps it; the other continues to Phase 2 where the HA watchdog's
heartbeat exchange will resolve the split-brain.
If a lower-priority candidate also
self-declares primary and enters Phase 2, the HA watchdog heartbeat exchange
will resolve the split-brain once both candidates reach Phase 2 — but this
requires the HA watchdog to compare candidate priority (from --peers ordering)
in addition to term, since both nodes have term=0 during simultaneous boot.
This design implements ha_check_heartbeat_nocluster() (ha.c:101-126)
with a priority comparison at term=0 so that a lower-priority self-declared primary
steps down when it receives a heartbeat from a higher-priority node with the
same term. Without this enhancement, the current elapsed-time-only check cannot
distinguish a legitimate primary from a split-brain peer during cold start.
CRITICAL DEPENDENCY — implemented by this design: The ha_check_heartbeat_nocluster()
enhancement (priority comparison at term=0) is a hard prerequisite for correct
simultaneous-boot behavior. Without it, a lower-priority self-declared primary will
not step down when it receives a heartbeat from a higher-priority peer with the same
term — the split-brain is unbounded, not 4–13s. This enhancement is implemented in
this design (ha.c — see §12 files changed) alongside the election probing. The
parent HA design (persistence-and-ha-design.md Part 5) specifies the heartbeat
exchange requirements; this design provides the priority-comparison implementation
which the parent design references. Test coverage:
test_am_i_primary_simultaneous_boot in §14.1 validates priority-based step-down
at term=0 during simultaneous boot.
Implementation scope: This design adds a sender_priority field to the
OP_heartbeat request in controller.capnp. The heartbeat listener (§9.0)
populates this field from c->ha.my_priority. ha_check_heartbeat_nocluster()
(ha.c) compares sender_priority against ha->my_priority when term == 0:
if the sender has higher priority (lower index in --peers), the local node
steps down to standby. The data source is the heartbeat message, not ha_state_t,
so both the schema change and the handler change are part of this design.
The effective split-brain window is therefore the time for the
secondary-primary to complete Phase 2 and start processing heartbeats: ~4–13s
per §8.5. During this window both self-declared primaries have iomgr_init and
endpoint_init complete (per the Phase 2 ordering in §7), so both are accepting
client connections before ha_promote resolves the split-brain. Conflicting
client operations (job submissions, queue modifications) submitted to both
primaries during this window may diverge. This falls within the parent design's
acknowledged split_brain_risk_accepted model, with the caveat that the
window is longer than a single heartbeat interval during cold start.
Data divergence during split-brain: The losing primary's WAL is discarded when it
steps down to standby — any client operations it accepted during the split-brain
window are lost. This design does not add a reconciliation protocol; the scope is
bounded to the existing HA model. Operators who cannot tolerate any data loss during
cold start should not submit client operations until the primary's INFO("Promoted to primary") log message confirms Phase 2 is complete. The typical cold-start
window occurs only during initial cluster boot or full-cluster restart — not during
normal single-node failover where an existing primary with a term > 0 prevents the
split-brain scenario from occurring.
PRNG: The jitter calculation uses an xorshift64 PRNG seeded from
/dev/urandom. The PRNG state is stored in ha_state_t.jitter_seed (one per
node, initialised during Phase 1). The xorshift64 implementation should be a
shared inline function in ha.h (or a small prng.h utility header) so both
the election probe (§9.1) and the heartbeat listener (parent HA design) use the
same generator without code duplication. The seed is initialised from
/dev/urandom (read 8 bytes) during Phase 1.
If /dev/urandom is unavailable, fall back to
clock_gettime(CLOCK_MONOTONIC, &ts) nanoseconds XOR'd with
((uint64_t)priority << 56) ^ ((uint64_t)getpid() << 16) with a WARN log.
The shift mixing avoids the correlation problem where two systemd-started
processes in the same nanosecond produce jitter values differing only by
their priority index (a small integer). The time(NULL) ^ getpid() fallback
is rejected because it produces nearly identical values for processes started
simultaneously by systemd (same second, sequential PIDs) — exactly the scenario
where symmetry breaking matters most.
9.1 Algorithm (design doc §5.4)
int am_i_primary(cluster_t *c)
{
/* Priority 0: unconditionally primary — highest-priority node does not probe.
* Safe because all controllers receive the same --peers list, so ordering is
* deterministic and priority-0 is unique across the cluster. */
if (c->ha.my_priority == 0) return 1;
/* Launch one pthread per candidate with priority < my_priority.
* Each thread makes a non-blocking connect + poll loop to the candidate,
* sends OP_heartbeat. If ANY thread gets a live ACK before the shared
* election_timeout_ms deadline → return 0 (standby).
* If the deadline expires with no ACK → return 1 (primary). */
atomic_int election_done = 0;
probe_ctx_t probes[MAX_HA_CANDIDATES];
int n_probes = 0;
for (int i = 0; i < c->ha.my_priority; i++) {
probes[n_probes++] = (probe_ctx_t){
.candidate = &c->myconf->peers_cfg.candidates[i],
.ssl_ctx = c->ha.ssl_ctx, /* stashed in Phase 1 */
.timeout_ms = c->ha.config.probe_connect_timeout_ms,
.election_done = &election_done,
.got_ack = 0, /* set to 1 by probe_thread on success */
};
pthread_create(&probes[n_probes-1].thread, NULL,
probe_thread, &probes[n_probes-1]);
}
/* Wait for first ACK or deadline (with jitter for simultaneous-boot, see §9.0) */
long to_ms = c->ha.config.election_timeout_ms;
to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms;
to_ms += (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms);
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += to_ms / 1000;
deadline.tv_nsec += (to_ms % 1000) * 1000000L;
if (deadline.tv_nsec >= 1000000000L) {
deadline.tv_sec++;
deadline.tv_nsec -= 1000000000L;
}
int got_ack = wait_any_probe(probes, n_probes, &deadline);
/* Signal all threads to stop (no pthread_cancel — unsafe through OpenSSL) */
atomic_store(&election_done, 1);
for (int i = 0; i < n_probes; i++)
pthread_join(probes[i].thread, NULL);
return got_ack ? 0 : 1;
}
9.2 Thread coordination
pthread_cancel() is NOT used (cancellation through OpenSSL shutdown handlers
can deadlock). Each probe thread uses non-blocking connect() + poll():
socket()+fcntl(O_NONBLOCK)+connect()→ expectEINPROGRESSpoll()with timeoutprobe_connect_timeout_ms(default 500ms)- Between poll attempts, check
atomic_load(&election_done)— if set, exit cleanly - If
poll()returns -1 anderrno != EINTR, treat as probe failure — log at TRACE level and exit thread with*got_ack = 0 - After connect,
SSL_connect()withSSL_ERROR_WANT_READ/WANT_WRITEpolling; on hard SSL failures (certificate expired, protocol mismatch, peer disconnect), treat as probe failure — log at TRACE level and exit thread - If
send()orrecv()fails on the SSL socket, treat as probe failure - Send
OP_heartbeat, receive ACK, set*got_ackvia shared pointer
The per-attempt poll timeout bounds the worst-case exit delay to one poll interval
per thread (~500ms). TCP connect timeout (kernel SYN retry) is not reached because
the thread checks election_done between each poll iteration and exits early.
Probe failures are expected when peers are down — only TRACE-level logging is
appropriate; ERROR logs would generate noise during normal operation.
9.3 Election outcome logging
/* Higher-priority candidate alive: */
INFO("HA election: candidate %s:%u responded — starting as standby (my_priority=%d)",
winner_host, winner_port, my_priority)
/* No higher-priority candidate alive: */
INFO("HA election: no higher-priority candidate responded within %ums — "
"starting as primary (my_priority=%d)",
election_timeout_ms, my_priority)
10. Error Handling
10.1 Primary with missing config
Condition: am_i_primary() returns 1 (this node is primary) AND
conf/current does not exist or does not point to a readable versioned file.
Action: FATAL log + return error from load_primary_config(), which propagates
to myapp_run() → cluster_fini() → exit.
FATAL("Primary candidate (priority 0) has no config file '%s' — "
"provision the config before starting the primary controller. "
"Run: xce-config --workdir %s --name %s --config cluster.yaml",
conffile, workdir, name)
10.2 Standby with no primary and no saved config
Condition: am_i_primary() returns 0 (standby) AND all config fetch retries
fail AND conf/current does not exist.
Action: FATAL log + exit. A standby cannot operate without any config — it
cannot initialise kvmgrs, load queues, or accept connections.
FATAL("Standby (priority %d) could not fetch config from primary and has no "
"saved config — start the primary controller first, or provision "
"the config file manually at '%s'",
my_priority, conffile)
10.3 Standby with no primary but saved config
Condition: am_i_primary() returns 0 AND all config fetch retries fail AND
conf/current exists and is valid.
Action: WARN log, proceed with saved config. This allows a standby to restart
during a primary outage without being permanently blocked. A conf_stale_timeout
(default: 24h, configurable via --conf-stale-timeout CLI flag) soft-limits
staleness: configs older than the threshold are flagged but not rejected (see below).
WARN("Standby (priority %d) could not reach primary for config fetch — "
"using saved config (tag=%016"PRIx64"). Config may be stale.",
my_priority, saved_tag)
A conf_stale_timeout field (default: 24h, configurable via --conf-stale-timeout
CLI flag) is stored in myconf_t alongside conf_keep_n. The 24h default is chosen to
match the typical maintenance window: if a primary is down for less than 24h, the
config on any standby that was connected before the outage is at most 24h old, which
is within the expected change cadence for HPC cluster configuration. Operators may
increase this for clusters with infrequent config changes, or set it to 0 to disable
staleness checking entirely. When a standby boots with
a saved config older than conf_stale_timeout seconds (determined by comparing the
versioned file's mtime against CLOCK_REALTIME), the controller logs a WARN and
proceeds, but sets a config_stale flag in myconf_t. The primary checks this flag
on the first OP_replConfig exchange after the standby reconnects: if the stale
config's tag does not match the primary's current tag, the primary pushes the new
config as normal. If a standby promotes while its config is stale (older than
conf_stale_timeout), the promotion proceeds — the stale timeout is a soft
indication, not a hard block, because the config may still be functionally valid.
Operators can set conf_stale_timeout to 0 to disable staleness checking.
The config_stale flag is recomputed each time the controller boots (comparing
the versioned file's mtime to the current time). It is not persisted across
restarts. If a standby reconnects to the primary and successfully fetches a
new config matching the primary's current tag, the flag is cleared (the saved
config is now fresh). If the standby promotes while config_stale is set, it
proceeds to Phase 2 as primary with the acknowledgment that the config may be
stale — operators should monitor logs and trigger a config upgrade via
OP_upgradeConfig if necessary to synchronise the cluster.
10.4 SHA-256 mismatch on receipt
Condition: standby recomputes SHA-256 of received bytes ≠ reply.sha256.
Action: discard the received bytes, log ERROR, retry the fetch (up to
max_attempts). If all retries fail with SHA-256 mismatches, treat as §10.2.
ERROR("Config fetch SHA-256 mismatch from primary — expected %s, got %s. "
"Retrying (%d/%d)", expected_hex, actual_hex, attempt, max_attempts)
10.5 Zero-tag in received config (redundant check)
Condition: primary sends reply.version_tag = 0.
Action: FATAL on the standby. With SHA-256 the probability of a legitimate
config producing version_tag = 0 is ~2^-64 — negligible. A zero tag in the
reply indicates a primary-side bug (not a hash collision). The error message
distinguishes this: "BUG: primary sent zero version_tag for config. Primary conffile " "may be empty or corrupted: '%s'".
10.6 Empty config file
Condition: config file exists but is 0 bytes.
Action: FATAL. SHA256("") produces a non-zero hash, but an empty file has no
valid Cap'n Proto content. The error message reads: "Config file '%s' is empty — not a " "valid ClusterConfig blob".
10.6a Empty pointer file
Condition: conf/current exists but is 0 bytes (zero-length pointer file).
Action: WARN on primary, treat as missing config (escalate to §10.1). Standby
treats as trigger for full config fetch from primary (the saved config is unreachable
because the pointer file was not fully written — typically a crash between open() and
the first write()+fsync()+close() in §3.2 step 2). The error message reads:
"Config pointer file 'conf/current' exists but is empty — treating as missing config. " "This can occur if the controller crashed during a config write."
A zero-length conf/current is distinct from §10.6 (empty versioned file) —
fgets() returns NULL on an empty file without setting errno, and the
caller receives -EIO from conf_read_current_filename(). Adding a dedicated
diagnostic before the fgets() call (checking file size via fstat() or reading
one byte with fread()) makes the failure cause explicit rather than opaque.
10.7 Config file size limit
Condition: config file exceeds CONF_MAX_SIZE (default 1 MiB).
Action: FATAL. A Cap'n Proto cluster config should be small (< 100 KB); a
larger file indicates corruption or a misconfiguration. The conf_read_raw()
function rejects files over this limit before allocating memory.
10.8 Standby with corrupt saved config
Condition: standby has conf/current pointing to a valid file, but its SHA-256
differs from the primary's cached hash.
Action: log WARN, discard saved config after receiving the correct one from
primary. This is normal — the saved config is stale, not necessarily bit-corrupt.
If the mismatch is detected during fetch_config_from_primary() (the standby's tag
≠ the primary's tag), the primary pushes the correct config and the standby overwrites:
WARN("Standby saved config tag=%016"PRIx64" differs from primary "
"tag=%016"PRIx64" — saved config may be stale, updating from primary",
saved_tag, primary_tag)
10.9 Trim crash safety
Condition: controller crashes during conf_trim_old_versions().
Action: no recovery needed. Trimming is best-effort — if a crash leaves more
than conf_keep_n versioned files, the excess files are removed on the next
successful config update. Stale .tmp files from interrupted writes are also
removed during the next trim pass (which runs after every conf_save_config() call).
There is no boot-time cleanup of orphaned files; the trim function is the sole
cleanup mechanism.
conf_trim_old_versions() always preserves the file named by the current conf/current
pointer: it reads the pointer file before building the deletion list and excludes that
filename regardless of mtime ordering.
10.10 Duplicate OP_upgradeConfig during restart window
Condition: a second OP_upgradeConfig arrives while the primary's cluster_run()
event loop is exiting (between restart_requested = 1 and cluster_fini()).
Action: iomgr is still accepting connections during this window, so the second
upgrade request is dispatched and may write a second config version, update
conf/current, and set restart_requested = 1 (already set — a no-op). The write
procedure (§3.2) is crash-safe regardless. The first config write's versioned file
remains on disk for the trim pass to clean up later. WAL/data consistency is
preserved: both configs are valid ClusterConfig blobs (validated before save).
The process exits cleanly with whichever config conf/current names at the time
of the exec. No FATAL is needed — concurrent upgrades are benign.
On restart via systemd, conf/current reflects the last saved config, which is
the one the primary will apply in Phase 2.
10.11 Concurrent push and fetch on conf/current
Condition: a standby receives an unsolicited push (§5.2 push direction) from the primary while it is concurrently executing its own retry-fetch thread (§5.5).
Action: both the push handler and the fetch thread call conf_save_config() which
ends with rename(ptr_tmp, ptr_path). To avoid temp-file collisions, each call
uses a thread-unique temp filename based on the caller's pthread ID:
snprintf(ptr_tmp, ..., "%s/conf/current.%lu.tmp", effective_dir, (unsigned long)pthread_self());
Each thread writes to its own .tmp file, fsyncs it, then renames it to conf/current.
POSIX rename() is atomic — both renames target the same destination (conf/current),
so whichever completes last wins, and the result is always a valid pointer file. Both
writes are for the same config version (the primary sends the same blob in both paths),
so either result is correct. Orphaned .tmp files from the losing thread are cleaned
by the next conf_trim_old_versions() pass.
The same pattern applies to the versioned .tmp files in step 1 of the write procedure
(§3.2): each caller appends its thread ID to the temp filename to avoid truncating
another thread's in-progress write.
Implementation note:
conf_save_config()must not open any temp file withO_EXCL(which would fail if another thread somehow created the same name).O_CREAT|O_TRUNCis correct for the thread-unique name.
10.12 OP_upgradeConfig directed at a standby
Condition: an admin tool sends OP_upgradeConfig to a standby controller's
endpoint.
Action: the standby's myreq_handler() checks its HA role before dispatching:
case OPCode_upgradeConfig:
if (!ha_is_primary(&c->ha)) {
send_upgrade_error(ep, req, "Not primary — redirect to primary");
/* Send primary address from ha_state if known */
return -EAGAIN;
}
return handle_upgrade_config(c, ep, req);
The standby logs WARN and sends an error reply with the primary's address
(if known from the HA watchdog state). The admin tool is expected to retry against
the primary.
11. Service Management and Deployment
11.1 systemd unit configuration
The controller is managed by a service manager (systemd or equivalent). The design
assumes the service manager handles process restart after cluster_run() exits
(whether from OP_upgradeConfig restart or crash recovery).
Certificate provisioning: The SSL transport layer (§5.3) requires each controller
node to have a certificate signed by the shared CA before it can participate in config
replication or election probing. Certificate provisioning and lifecycle management
(renewal, revocation, CA rotation) are external to this design — they are assumed to
be handled by the same configuration management system that distributes --peers
(§11.2). At minimum, each node needs certfile and keyfile on disk before the
controller starts. A controller whose certificate has expired cannot complete the SSL
handshake and will be unable to fetch config or participate in elections — the
operational impact is equivalent to a network partition. Operators should monitor
certificate expiry and integrate renewal into their standard PKI workflows.
Certificate expiry warning: At startup (Phase 1, after create_ssl_context),
the controller checks X509_get0_notAfter() on its own certificate. If the
certificate expires within CERT_WARN_DAYS (compile-time constant, default 30),
a WARN is logged:
WARN("Controller certificate expires in %d days (%s) — renew before expiry "
"to avoid HA election exclusion", days_remaining, asn1_time_str)
A controller whose certificate has expired cannot complete the SSL handshake and will be unable to fetch config or participate in elections — the operational impact is equivalent to a network partition. The startup warning gives operators advance notice before the hard failure.
A recommended systemd unit file:
[Unit]
Description=XCE Controller
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
Type=notify
NotifyAccess=all
ExecStart=/usr/local/bin/xcecontroller --peers=node0=host0:9990,node1=host1:9990,node2=host2:9990
Restart=always
RestartSec=5
# Hardening
User=xce
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateDevices=yes
ReadWritePaths=/var/lib/xce
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
RestrictNamespaces=yes
SystemCallFilter=@system-service
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
RestrictAddressFamilies=AF_INET AF_INET6
ProtectHostname=yes
LockPersonality=yes
[Install]
WantedBy=multi-user.target
Key points:
After=network-online.targetensures network interfaces are up before the controller starts, reducing the probability of connection failures during the initial config fetch.RestartSec=5withStartLimitBurst=3bounds restart storms. A longerRestartSec(e.g., 15–30s) can be used to stagger controller restarts across nodes in a simultaneous-boot scenario, reducing the probability that all candidates enter election probing at the same time.Type=notifyallows the controller to signal readiness to systemd after Phase 2 completes, preventing the service manager from considering the process "started" before it can serve requests.
11.2 --peers consistency
All controller candidates in a cluster must receive the same --peers argument.
The priority ordering (positional: candidate 0 is highest priority, candidate N-1 is
lowest) must be identical across all nodes. Inconsistent --peers arguments can cause
two nodes to self-declare primary (both with my_priority == 0) and never detect
each other.
Relationship to provisioning goal: The §2 provisioning goal ("single provisioning
point: only the primary requires the config file") eliminates manual config distribution
but does not eliminate manual --peers distribution. Every candidate still needs the
--peers argument provisioned consistently — this is a prerequisite for correct HA
election behaviour. The --peers list is a bootstrap concern (it tells controllers
how to find each other) and cannot be bootstrapped from the config itself (circular
dependency: you need --peers to know who to fetch config from). Mitigations:
(1) the --peers list changes far less frequently than cluster config — it only
changes when nodes are added or removed from the HA cluster, which is rare compared
to queue/resource config changes that drive this design; (2) the future
peers-consistency cross-validation in OP_heartbeat (§11.2 below) will detect
mismatches at runtime; (3) the existing config management tooling already distributes
--peers for the HA foundation.
Recommended deployment: distribute the --peers argument via a shared configuration
management system (Ansible, Puppet, Salt) or a template-generated unit file. This
ensures all nodes receive the same candidate list and ordering. The --peers argument
can alternatively be read from a local config file (xce.conf or equivalent) managed
by the same configuration system.
Validation: xce_parse_peers() should log a WARN at startup if my_priority is
not in the range [0, MAX_CANDIDATES-1] or if any candidate entry has an empty
host or port zero — these indicate a malformed --peers string.
Future cross-validation: A lightweight peers-consistency check in OP_heartbeat
(exchanging a hash of the local --peers string) is deferred to a follow-up release.
See FUTURE_WORK.md.
12. Files Changed
| File | Change | Complexity |
|---|---|---|
src/schema/controller.capnp |
Add replConfig @29, replConfigPush @30, upgradeConfig @31 to OPCode; add ReplConfigRequest, ReplConfigReply, ReplConfigPushRequest, ReplConfigPushReply, UpgradeConfigRequest, UpgradeConfigReply structs; add configGeneration @20 :UInt64 to ClusterConfig (§4.4); add sender_priority @0 :UInt32 to Heartbeat struct (populated by heartbeat listener, compared by ha_check_heartbeat_nocluster() — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to Request and RequestReply |
Low |
src/controller/main.c |
Update conf_file resolution: read conf/current pointer file, then open named versioned file (replaces flat conf/current_config path); remove stack-allocated char conf_file[] and the myconf.conffile = conf_file assignment — conffile is now heap-allocated via asprintf() in load_primary_config() or fetch_config_from_primary() and freed in myconf_fini(); add --conf-stale-timeout and --conf-keep-n CLI flag parsing; add --peers validation (WARN on empty host or port 0) |
Trivial |
src/include/myconf.h |
Add config_version_tag (uint64_t), config_sha256[32], conf_keep_n, config_generation (uint64_t, §4.4), conf_stale_timeout (time_t, default 86400), and config_stale (bool) fields to myconf_t |
Trivial |
src/include/xce/cluster_def.h |
Add restart_requested (atomic_int, for config-change restart), heartbeat_listener_tid, heartbeat_listener_running, and lost_and_found_job_count (atomic_int, incremented during persist_recover() when jobs are routed to the synthetic queue, §6.5) fields to cluster_t |
Trivial |
src/controller/myconf.c |
Replace get_default_conf() with load_primary_config() (fail-fast + SHA-256 compute on primary path); config save/trim/validation functions moved to config_repl.c; add myconf_fini() to free conffile (now heap-allocated via asprintf() per §3.2), cluster_config, config_version_tag, and config_sha256 |
Low |
src/controller/cluster.c |
Split cluster_init() into phase 1 and phase 2; gate config load on role; move iomgr_init and endpoint_init to phase 2; add create_lost_and_found() call before persist_recover; add OP_replConfig, OP_replConfigPush, and OP_upgradeConfig cases to myreq_handler() opcode dispatch; cluster_run() event loop checks c->restart_requested and exits cleanly when set; cluster_fini() joins heartbeat listener thread on any exit path and frees standby_endpoints |
Medium |
src/controller/myapp.c |
Restructure myapp_run(): call phase 1, then am_i_primary(), then config load/fetch, then phase 2. When cluster_run() returns cleanly (restart_requested=1), myapp_run() returns 0 to main() which exits; the systemd service manager restarts the process. Call myconf_fini() in the cleanup path before cluster_fini() |
Medium |
src/controller/config_repl.c |
Phase 1: conf_read_current_filename() (read HEAD-style pointer), conf_read_raw() (read + size-check, enforcing CONF_MAX_SIZE), conf_save_config() (write + fsync + thread-unique pointer-file swap + trim), conf_trim_old_versions(), compute_config_sha256(), check_conf_stale() (mtime-based staleness check, §10.3), sanitise_cn() (cert CN log sanitisation, §6.4), config_contains_reserved_name() (lost_and_found name collision check, §7.1). Phase 2: handle_upgrade_config() (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), push_config_to_standbys(), create_lost_and_found() (synthetic queue + host, §6.5), resolve_queue_or_lost_and_found(), resolve_host_or_lost_and_found(), clear_resource_and_pend(). Compile-time gate: CONF_ENABLE_UPGRADE_CONFIG (default 0, §5.3) |
Medium |
src/include/byteorder.h |
New file: be64toh() fallback macro for non-glibc or pre-2.25 targets (§6.4 portability note). The Meson build checks for be64toh() at configure time; if unavailable, this header provides a portable implementation via __builtin_bswap64() or le64toh()/ntohll() |
Trivial |
src/controller/meson.build |
Add -DCONF_PUSH_TIMEOUT_MS=500 and -DCONF_ENABLE_UPGRADE_CONFIG=0 compile definitions (defaults, overridable via Meson options); add config_repl.c, config_repl.h, byteorder.h to sources |
Trivial |
src/controller/ha.c |
Replace am_i_primary() stub with real election probing; add probe_thread(), wait_any_probe(); add start_heartbeat_listener() / stop_heartbeat_listener(); add xorshift64_next() (or use shared ha.h inline); enhance ha_check_heartbeat_nocluster() with priority comparison for simultaneous-boot symmetry breaking (§9.0) |
High |
src/controller/ha.h |
Add ssl_ctx field to ha_state_t (use void* to avoid pulling in <openssl/ssl.h> — cast to SSL_CTX* at use sites in election probe and heartbeat listener); add jitter_seed (uint64_t, for xorshift64 PRNG) and xorshift64_next() inline function to ha_state_t; add zarray_t *standby_endpoints to ha_state_t for tracking connected standby controllers; add probe_ctx_t struct; add election_jitter_max_ms and election_priority_step_ms to ha_config_t with defaults HA_DEFAULT_ELECTION_JITTER_MAX_MS 200 and HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500 (§9.0 priority-based base delay); update "no new TLS" comment to note SSL transport is sufficient for trusted-LAN model; update am_i_primary() declaration; declare ha_is_primary() for runtime role check (§10.12) |
Low |
src/controller/join.c |
Update handle_controller_join(): remove -ENOSYS (controller-controller sessions are valid — config exchange is dispatched from myreq_handler()); register connected standby endpoints in c->ha.standby_endpoints after successful join handshake; handle endpoint removal on disconnect |
Low |
src/include/typemap.h |
Add replConfigRequestMsg, replConfigReplyMsg, replConfigPushRequestMsg, replConfigPushReplyMsg, upgradeConfigRequestMsg, upgradeConfigReplyMsg union arms to xce_request_t.body and xce_reply_t.body |
Low |
src/lib/request.c |
Add encode/decode shims for ReplConfigRequest, ReplConfigReply, ReplConfigPushRequest, ReplConfigPushReply, UpgradeConfigRequest, UpgradeConfigReply; Data blobs managed by codecgen encode_*_ptr() functions |
Medium |
src/include/persist.h |
Add max_wal_records field to persist_config_t (for WAL bound, see §13); add wal_record_count (atomic_uint) and checkpoint_in_progress (atomic_int) fields to persist_t for the count-based checkpoint trigger (§13.4) |
Trivial |
New files: src/controller/config_repl.c, src/controller/config_repl.h, and
src/include/byteorder.h — config save/load/trim, SHA-256 computation, upgrade handler,
standby push, and be64toh() portability fallback. The config replication logic lives in
its own translation unit rather than cluttering myconf.c.
Note: the existing src/controller/repl.c (WAL replication — standby slot management,
ACK tracking, catch-up) is a separate module with a similar name but different domain.
repl.c uses replicator_t.standbys[] for per-connection WAL ACK tracking;
config_repl.c uses ha_state_t.standby_endpoints for config push fan-out.
See src/controller/repl.c (340 lines) for the existing WAL replication module.
Total estimated LOC: ~510 net new lines across all files (updated from ~480 to
account for compile-time gate, sanitise_cn(), config_contains_reserved_name(),
byteorder.h, and CONF_PUSH_TIMEOUT_MS).
Of these, ~170 are the am_i_primary() election probing and heartbeat listener
infrastructure. This infrastructure is implemented by this design (the current
stub at ha.c:213 returns c->ha.my_priority == 0 without probing). The parent
HA design (persistence-and-ha-design.md Part 5) specifies the requirements
(HA Requirements 3 and 4: election probing and heartbeat listener); this design
provides the implementation, which the parent design references. The config
replication contribution is ~310 LOC: ~250 for the pull protocol, storage layout,
and SHA-256 infrastructure; ~60 for the OP_upgradeConfig handler and
push_config_to_standbys() logic.
13. WAL Record Bound
13.1 Motivation
persist_recover() runs in Phase 2 on the primary, before iomgr is ready to serve
OP_replConfig. The standby retry window is ~88 seconds (§5.5). Without a bound on
persist_recover() time, the standby may exhaust retries and FATAL even though the
primary is healthy and making progress.
13.2 Mechanism
Add max_wal_records to persist_config_t (default: 10,000). After every
persist_append() call in persist.c, if the active WAL segment exceeds
max_wal_records, trigger an asynchronous checkpoint:
persist_append(wal, entry):
append entry to WAL segment
if n_records(wal) > max_wal_records:
schedule_checkpoint() /* async — queues a vtask */
The checkpoint flushes all in-memory state (jobs, queues, resources) to a fresh
snapshot file, truncates the WAL, and updates snap_ctx to reference the new
snapshot. Subsequent persist_recover() calls only replay records since the last
completed checkpoint.
13.3 Time bound
With max_wal_records = 10000, each record being a Cap'n Proto decode + in-memory
apply, persist_recover() replays at most 10,000 records. Benchmarked estimate:
~2-5 seconds on typical HPC controller hardware. The ~88-second retry window provides
10× headroom.
13.4 Checkpoint scheduling
Checkpoints are scheduled via the delaymgr (available in Phase 2 after
delaymgr_init) rather than vtaskmgr directly. Using delaymgr avoids
saturating the vtaskmgr thread pool with repeated checkpoint requests during
burst-append scenarios — delaymgr coalesces multiple requests into a single
delayed dispatch.
schedule_checkpoint() specification:
/* schedule_checkpoint — request an async WAL checkpoint via delaymgr.
*
* Idempotent: if a checkpoint is already scheduled or in progress,
* this is a no-op (guarded by checkpoint_in_progress atomic).
*
* @param persist The persist_t instance (has access to cluster_t via
* persist->cluster for delaymgr and vtaskmgr handles).
*
* Thread-safety: safe to call from any thread. Uses atomic CAS on
* persist->checkpoint_in_progress to ensure only one checkpoint
* is queued at a time.
*/
static void schedule_checkpoint(persist_t *persist)
{
/* CAS: 0 → 1. If already 1 (in-progress or queued), no-op. */
int expected = 0;
if (!atomic_compare_exchange_strong(&persist->checkpoint_in_progress,
&expected, 1)) {
return; /* already scheduled or running */
}
/* Schedule via delaymgr with a short delay to coalesce
* burst-append requests into a single checkpoint dispatch.
* delay_task() uses time_t (seconds) granularity — see src/lib/delay.c:187. The actual API is delay_task(delaymgr_t*, int period, int runcount,
* delay_task_entry_fn_t, void*). */
delay_task(&persist->cluster->delaymgr,
1 /* 1s — coalesce burst appends; delay_task() takes seconds */,
1 /* runcount — fire once */,
checkpoint_task_fn,
persist);
atomic_store(&persist->wal_record_count, 0); /* reset counter after checkpoint */
}
/* checkpoint_task_fn — executed by vtaskmgr worker, dispatched by delaymgr.
* Returns int per delay_task_entry_fn_t signature. */
static int checkpoint_task_fn(void *arg)
{
persist_t *persist = (persist_t *)arg;
persist_checkpoint(persist); /* existing synchronous checkpoint */
atomic_store(&persist->checkpoint_in_progress, 0);
return 0;
}
The WAL record-count trigger in persist_wal_sync_append() and
persist_wal_async() (the actual WAL write functions in persist.h:109-111)
calls schedule_checkpoint() after appending each record when the record count
exceeds max_wal_records:
/* After appending WAL record: */
if (atomic_fetch_add(&persist->wal_record_count, 1) >= persist->config.max_wal_records) {
schedule_checkpoint(persist);
}
persist->wal_record_count is an atomic_uint in persist_t, reset to 0
after each checkpoint completes. The primary continues serving requests while
the checkpoint progresses. Multiple persist_append() calls may queue redundant
checkpoints — schedule_checkpoint() is idempotent via the atomic CAS guard.
Records appended to the WAL during an in-progress checkpoint may push
wal_record_count above max_wal_records again before the checkpoint
completes. This is intentional — wal_record_count is a soft, lossy
signal whose sole purpose is to trigger checkpoint scheduling. It is
reset to 0 after persist_checkpoint() returns, so any records appended
during the checkpoint will be counted in the next trigger window. This
does not affect correctness: the checkpoint always flushes all in-memory
state regardless of the counter value.
13.5 Configuration
max_wal_records is set in persist_config_t alongside existing WAL parameters
(wal_max_size, snap_keep_n). Administrators can tune it per workload profile.
Larger values mean longer persist_recover() but fewer checkpoints (less I/O).
Smaller values bound recovery time more tightly at the cost of more frequent
snapshot writes. The minimum allowed value is 100 (checked at startup in persist_init());
values below this are rejected with a FATAL log to prevent checkpoint storms.
max_wal_records is a soft trigger for checkpoint scheduling, not a hard
limit on record count. After persist_append() appends the new entry, if the
WAL segment now contains more than max_wal_records entries,
schedule_checkpoint() is called; subsequent appends proceed immediately
without waiting. The WAL may temporarily exceed max_wal_records during a
burst of appends if the asynchronous checkpoint is slow to complete.
The parent design (persistence-and-ha-design.md §2.3) is the authoritative
source for WAL growth controls. This section adds only the record-count-based
trigger needed to bound persist_recover() time for the standby retry window
(§5.5). The parent design provides: wal_max_size_mb hard limit (2 GB default)
as an emergency trigger, checkpoint_in_progress atomic guard, and WAL growth
warning at 2 * snap_wal_size_mb during in-progress checkpoint. These
mechanisms are complementary — the size-based limit catches large-record workloads
while the count-based limit bounds replay time regardless of record size.
14. Test Plan
14.1 Unit tests (no live cluster required)
Phase 1 tests:
| Test | File | Covers |
|---|---|---|
test_conf_sha256_change_detection |
test_config_repl.c |
Tag match → no push; tag differ → push |
test_conf_sha256_full_integrity |
test_config_repl.c |
Full SHA-256 mismatch → integrity failure → retry |
test_conf_save_pointer_atomic |
test_config_repl.c |
Write versioned file + fsync + HEAD-file swap; verify cat conf/current names new file after update |
test_conf_save_crash_after_fsync |
test_config_repl.c |
Crash after step 1 (versioned file on disk, pointer file not updated) → next boot reads old pointer file |
test_conf_read_current_filename |
test_config_repl.c |
conf_read_current_filename() parses filename, strips newline; returns -ENOENT when file absent |
test_conf_zero_tag_sentinel |
test_config_repl.c |
Standby with no saved config sends tag=0 → primary correctly pushes config (no FATAL for zero-tag sentinel; FATAL only when primary sends zero in reply per §10.5) |
test_conf_trim_keep_n |
test_config_repl.c |
After 3 updates with keep_n=2, only 2 versioned files remain; stale .tmp cleaned |
test_am_i_primary_no_peers |
test_ha_role.c (extend) |
Priority-0 with no peers → primary without probing |
test_am_i_primary_higher_alive |
test_ha_role.c (extend) |
Mock probe ACK from priority-0 → standby |
test_am_i_primary_deadline_expires |
test_ha_role.c (extend) |
All probes time out → primary |
test_am_i_primary_simultaneous_boot |
test_ha_role.c (extend) |
Three priorities start simultaneously → transient split-brain resolves; priority-0 is primary after Phase 2 watchdog exchange (§9.0) |
test_primary_missing_config_fatal |
test_config_repl.c |
Priority-0 + no conffile → load_primary_config() returns -ENOENT |
test_standby_no_config_no_primary_fatal |
test_config_repl.c |
Priority-1 + fetch fails + no saved config → fatal error path |
test_standby_saved_config_no_primary |
test_config_repl.c |
Priority-1 + fetch fails + saved config exists → WARN + proceed |
test_conf_stale_timeout |
test_config_repl.c |
Saved config file mtime older than conf_stale_timeout → WARN logged, config_stale flag set; saved config within timeout → no WARN, flag clear |
test_cert_expiry_warning |
test_ha_role.c (extend) |
Certificate expiring within CERT_WARN_DAYS → WARN logged; certificate far in future → no WARN (§11.1) |
Phase 2 tests (require cluster_init() Phase 2 to be complete — iomgr, kvmgr, persist):
| Test | File | Covers |
|---|---|---|
test_upgrade_config_sha256_mismatch |
test_config_repl.c |
OP_upgradeConfig with wrong caller SHA-256 → rejected before disk write |
test_upgrade_config_invalid_capnp |
test_config_repl.c |
OP_upgradeConfig with corrupt blob → Cap'n Proto decode fails → rejected |
test_upgrade_config_generation_incremented |
test_config_repl.c |
OP_upgradeConfig with any generation value → primary ignores incoming generation, sets config_generation = current + 1 (§4.4, §6.4 step 3) |
test_upgrade_config_replay_harmless |
test_config_repl.c |
Duplicate OP_upgradeConfig (same bytes) → accepted again, generation incremented, restart triggered (§4.4) |
test_upgrade_config_wal_unknown_id |
test_config_repl.c |
WAL replay with unknown queue ID → job routed to lost_and_found queue; unknown resource ID → job resource cleared, set to PEND; unknown host → routed to lost_and_found host |
test_lost_and_found_queue |
test_config_repl.c |
persist_recover() with job referencing removed queue ID → job placed in lost_and_found queue; unknown resource ID → job resource cleared, state set to PEND |
test_lost_and_found_counter |
test_config_repl.c |
persist_recover() routes jobs to lost_and_found → lost_and_found_job_count incremented; startup WARN summary logged if counter > 0 (§6.5) |
14.2 Integration test (requires two controller processes)
| Test | Covers |
|---|---|
test_standby_receives_config_on_first_boot |
Fresh standby with no conf/ → receives config → conf/current pointer file created, SHA-256 verified |
test_standby_skips_fetch_on_tag_match |
Standby reconnects with matching tag → up_to_date=true → no file I/O |
test_standby_updates_config_on_tag_mismatch |
Standby reconnects with stale tag → receives new config → pointer file updated → previous version retained |
test_primary_missing_config_aborts |
Start primary with no conf/ → process exits non-zero |
test_primary_recomputes_hash_before_send |
Primary's handler double-checks SHA-256 before sending; mismatch detected |
test_upgrade_config_pushes_to_standby |
Admin sends OP_upgradeConfig → primary ACKs → connected standby receives push → both have new tag before primary restarts |
test_upgrade_config_offline_standby_catches_up |
Admin upgrades config while standby is offline → standby restarts → detects tag mismatch on reconnect → pulls new config |
test_upgrade_config_no_standbys |
Admin upgrades config on single-node cluster (no standbys) → save succeeds → restart applies new config |
14.3 Regression
All existing HA tests (test_ha_cluster.c, test_ha_role.c, test_ha_redirect.c,
test_p1_integration.c) must continue to pass unchanged. These tests manually
zero-init cluster_t and call ha_init() directly, bypassing cluster_init() — the
phase split does not affect them.
14.4 Future Work
Tracked in FUTURE_WORK.md. Summary:
| Test | Covers |
|---|---|
test_peers_cross_validation |
OP_heartbeat includes --peers hash → mismatch between two candidates → ERROR logged (§11.2 future work) |
test_lost_and_found_drain |
xcecli lost-and-found drain command → reassign or cancel sidelined jobs → lost_and_found_job_count reset to 0 (§6.5 observability) |
test_upgradeconfig_redirect |
OP_upgradeConfig sent to standby → error reply with primary address → admin tool retries (§10.12) |