diff --git a/src/docs/FUTURE_WORK.md b/src/docs/FUTURE_WORK.md new file mode 100644 index 0000000..ba0812f --- /dev/null +++ b/src/docs/FUTURE_WORK.md @@ -0,0 +1,52 @@ +# XCE Controller: Future Work Items + +> **Status**: Tracked from config-replication-design.md review +> **Last Updated**: 2026-05-12 + +--- + +## 1. Admin Config Tool (prerequisite for Phase 2) + +`OP_upgradeConfig` requires an admin tool to drive it. The tool must: + +- Encode a `ClusterConfig` YAML/JSON file to Cap'n Proto binary +- Compute SHA-256 over the binary bytes +- Connect to the primary controller's endpoint (SSL, same CA as controllers) +- Send `OP_upgradeConfig` with the binary blob and SHA-256 +- Display the reply: new version tag, acked count, and any error message +- Poll `OP_probe` after restart to confirm the primary is back online with the new config + +Suggested interface: `xce-config push --controller --config cluster.yaml` + +--- + +## 2. Quorum-ACK Gate Before Restart (§14.4 extension) + +Current `OP_upgradeConfig` restarts the primary after best-effort push to standbys. +A quorum-ACK gate would refuse to restart unless a minimum number of standbys +acknowledge the push, preventing silent rollback when all standbys are offline. + +Requires: `OP_upgradeConfig` reply to include `--require-standby-ack=N` flag. + +--- + +## 3. OP_heartbeat Peers Cross-Validation (§11.2) + +Include a hash of the local `--peers` string in the heartbeat request. If two +candidates disagree on the peers list, log ERROR. Detects misconfigured nodes +at election time rather than silently producing dual-primary split-brain. + +--- + +## 4. Lost and Found Drain Command (§6.5 extension) + +`xcecli lost-and-found drain` — reassign or cancel sidelined jobs, reset +`lost_and_found_job_count` to 0. Automates operator cleanup after config upgrade. + +--- + +## 5. OP_upgradeConfig Standby Redirect (§10.12 verification) + +Test that a standby correctly returns an error reply with the primary's address +when an admin tool sends `OP_upgradeConfig` to a standby, and the admin tool +retries against the primary. \ No newline at end of file diff --git a/src/docs/config-replication-design.md b/src/docs/config-replication-design.md new file mode 100644 index 0000000..7e74d10 --- /dev/null +++ b/src/docs/config-replication-design.md @@ -0,0 +1,2321 @@ +# XCE Controller: Configuration Replication Design + +> **Status**: Draft +> **Last Updated**: 2026-05-12 +> **Target**: `src/controller/` (C daemon) +> **Depends on**: `persistence-and-ha-design.md` Part 5 (HA Failover) + +--- + +## Table of Contents + +1. [Problem Statement](#1-problem-statement) +2. [Goals and Non-Goals](#2-goals-and-non-goals) +3. [Config Storage Layout](#3-config-storage-layout) +4. [SHA-256 as Version and Integrity Token](#4-sha-256-as-version-and-integrity-token) +5. [OP_replConfig Protocol](#5-op_replconfig-protocol) +6. [OP_upgradeConfig Protocol](#6-op_upgradeconfig-protocol) +7. [Startup Sequence](#7-startup-sequence) +8. [cluster_init() Phase Split](#8-cluster_init-phase-split) +9. [am_i_primary() Implementation](#9-am_i_primary-implementation) +10. [Error Handling](#10-error-handling) +11. [Service Management and Deployment](#11-service-management-and-deployment) +12. [Files Changed](#12-files-changed) +13. [WAL Record Bound](#13-wal-record-bound) +14. [Test Plan](#14-test-plan) + +--- + +## 1. Problem Statement + +### 1.1 Current behaviour + +Every controller candidate reads its cluster configuration (`ClusterConfig` Cap'n Proto blob) +from a local file at `${workdir}/${name}/conf/current_config` on startup. This means: + +- Every candidate node must be independently provisioned with a copy of the config file. +- Any config change (new queue, resource definition update) must be applied to every + candidate separately. There is no enforcement mechanism — a missed copy is silent. +- A standby running a stale config promotes with wrong queue/resource definitions. + The mismatch is only discovered when the promoted standby starts serving clients. + +### 1.2 Proposed behaviour + +The **primary** (highest-priority candidate) is the sole config authority: + +- The primary reads `ClusterConfig` from its local `conf/current/` on startup. If the file + is missing, it is a fatal error — the cluster cannot start. +- A **standby** that has no local config fetches it from the primary over the existing + SSL/iomgr pipeline before completing initialisation. The received config is saved locally + so the standby can restart without the primary being available. +- On every reconnect, the standby sends its current config's SHA-256 tag (first 8 bytes). + The primary skips the push if the tag matches, avoiding unnecessary reinitialisation. + +--- + +## 2. Goals and Non-Goals + +### Goals + +**Provisioning** + +- Single provisioning point: only the primary candidate node requires the config file. +- Detect and minimize config drift: standbys fetch from primary on every reconnect. + When the primary is unreachable, a saved config is used with staleness warning (§10.3). + Config drift is eliminated for the normal case (reconnect); the edge case where a + standby promotes while the primary is unreachable is mitigated by the staleness + warning and `config_stale` flag but cannot be fully prevented without a quorum gate + (deferred to §14.4). **Residual risk accepted**: if both the primary and all + standbys with current config are simultaneously down, a standby with only a stale + saved config will promote and serve with potentially wrong queue/resource definitions + — the same failure mode described in §1.1. The `config_stale` flag and log warnings + make this discoverable (the current behaviour is silent), but do not prevent it. + Operators who require hard prevention should implement the quorum gate (§14.4). + +**Integrity** + +- Atomic, crash-safe config update on the standby. +- SHA-256 as the token for both change detection and cryptographic integrity validation. + +**Observability** + +- Human-visible config version in the filesystem (SHA-256-named files + pointer file). + +**Remote config push** + +- `OP_upgradeConfig` allows an admin tool to push a new config blob to the primary + via the controller endpoint. The primary saves it, pushes it to connected standbys + (best-effort, with ACK count in the reply), then restarts itself to apply the + new config (~4–13s downtime; see §6.6 for timing decomposition). The admin tool + that drives `OP_upgradeConfig` is out of scope for this design but is a prerequisite + for Phase 2 — it must encode `ClusterConfig` to Cap'n Proto binary, compute SHA-256, + and send the request via the controller's internal endpoint (see `FUTURE_WORK.md`). + +### Non-Goals + +**Scope** +- Hot-reload of config without restart. Config drives kvmgr slot counts and resource + definitions that are baked in at Phase 2 init time; a restart is required to apply them. +- YAML config parsing in the controller. The controller reads only the Cap'n Proto binary + blob. A separate admin tool (out of scope) produces the blob. + +**Responsibility boundaries** +- Semantic validation of `ClusterConfig` content (e.g., non-zero queue count, valid port + ranges, host list completeness). The controller validates structural correctness only + (valid Cap'n Proto decode, size within `CONF_MAX_SIZE`). Semantic correctness is the + responsibility of the admin tool that produces the blob. + +**Compatibility** +- Backward compatibility. This is the first version of the protocol. + +### Alternatives Considered + +**Config-management-based distribution**: The simplest alternative is to distribute +the `ClusterConfig` blob via the same config management system (Ansible, Puppet, +Salt) already recommended for `--peers` distribution (§11.2). This avoids all new +controller code. + +| Approach | Provisioning | Drift detection | Bootstrap without pre-seeding | Restart-free change | LOC | +|---|---|---|---|---|---| +| Config management only | Multi-node push | Manual (diff files) | No | No | 0 | +| Config-in-join-reply | Primary-only | Overwrite on every join | Yes (join reply) | No (restart required) | ~50 | +| This design | Primary-only | Automatic (SHA-256 tag) | Yes (pull from primary) | No (restart required) | ~310 | + +Config management alone works for clusters where all controller nodes are managed +by the same automation system. This design addresses the gap where that assumption +breaks down: a newly provisioned standby with no pre-existing config file, or a +standby rejoining after extended offline where its saved config may be stale. The +HEAD-file + SHA-256 approach also provides cryptographic proof that two controllers +run identical configs without parsing binary blobs — useful for operator verification. + +**Config-in-join-reply (minimal approach)**: A ~50 LOC alternative: include the raw +config bytes in the controller join reply. When a standby completes `OP_join`, the +primary attaches the current `ClusterConfig` blob. The standby saves it to +`conf/current_config` (overwriting any stale copy). This handles the bootstrap case +and the reconnect case (join reply always carries current config). It does not +provide: tag-based push skip (every join sends the full blob), cryptographic +integrity validation, version history/retention, or `OP_upgradeConfig` live push. +For clusters with infrequent joins and small configs (<100 KB), this is sufficient. +The full design is chosen for clusters where config integrity verification, change +auditability, and the push mechanism for online config updates are operational requirements. + +**WAL-based config replication (considered and rejected)**: XCE already has a WAL +replication path (`src/controller/repl.c`, ~340 lines) that streams job/queue/resource +mutations from primary to standbys. Extending it to carry serialised `ClusterConfig` +blobs would reuse the existing replication infrastructure. This approach is rejected +for three reasons: (1) **Semantic mismatch** — WAL replication is continuous and +ordered; config changes are one-shot, admin-initiated events that should not be +interleaved with job mutation records (a WAL ordering dependency between config records +and operation records creates unnecessary coupling). (2) **Atomicity** — config changes +require saving the full blob, computing SHA-256, updating the HEAD-file pointer, and +trimming old versions atomically; this is a self-contained operation with its own +integrity guarantees distinct from WAL append semantics. (3) **Bootstrap ordering** — +WAL replication requires `persist_init` and a replica connection to be established +(Phase 2), but the standby needs config before Phase 2 to initialise iomgr and kvmgr +(§7). A separate config fetch protocol (§5) avoids this circular dependency. + +**Why this design**: The chosen approach layers on the full protocol not because the +bootstrap case demands it, but because the same SHA-256 infrastructure underpins: +(1) tag-based skip on reconnect (avoiding unnecessary deserialisation), +(2) cryptographic integrity validation of received blobs, (3) human-visible version +identity via `conf/current` pointer, and (4) the `OP_upgradeConfig` push mechanism +for remote admin-driven config changes. Each layer individually could be deferred; +together they form a coherent config management lifecycle that the simpler +approaches do not provide. + +### Phased Delivery + +The design is organised into two phases, but **both phases ship together in a single PR**. +The "phase" distinction refers to activation readiness, not separate deliveries: + +**Phase 1 — Pull-only config replication** (active on merge; delivers Provisioning, Integrity, +Observability goals): +- `OP_replConfig` pull protocol (§5) with SHA-256 tag comparison +- HEAD-file + versioned files storage layout (§3) +- Standby fetch with retry (§5.5) +- `am_i_primary()` election probing (§9, shared with parent HA design) +- ~310 LOC + +**Phase 2 — Remote config push** (infrastructure ships now, feature activates when admin tool +and `validate_host()` are complete; see `FUTURE_WORK.md`): +- `OP_upgradeConfig` handler (§6.4), `OP_replConfigPush` fan-out (§6.3) +- `lost_and_found` queue/host routing (§6.5) +- `standby_endpoints` tracking, `restart_requested` mechanism +- WAL record bound (§13) +- ~170 LOC + +The `OP_upgradeConfig` handler is gated behind a compile-time flag +`CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled — see §5.3 safety gate) until +two prerequisites are met: (1) `validate_host()` is implemented with a real +authorisation check, and (2) an admin tool exists to drive the endpoint (encoding +`ClusterConfig` to Cap'n Proto binary, computing SHA-256, and sending the request +via the controller's internal endpoint). Both prerequisites are tracked in +`FUTURE_WORK.md`. Phase 1 alone eliminates manual provisioning and config drift — +the core problems stated in §1. + +--- + +## 3. Config Storage Layout + +### 3.1 Directory structure + +The parent HA design (`persistence-and-ha-design.md` Part 5) specifies a config storage +layout under `conf/`. This design replaces that with the HEAD-file + versioned files +approach described below. This design replaces that with a +**HEAD-file + versioned files** layout that is atomically crash-safe. The +HEAD-file pattern avoids symlink(2), which is not atomic on all filesystems and +would require a separate atomic-rename technique anyway. + +**Local filesystem only — NFS is explicitly unsupported.** The controller's working +directory (including `conf/`, `data/wal/`, and `data/snap/`) MUST reside on a local +filesystem. `persist_init()` (persist.c:217-222) detects and rejects NFS mounts at +startup with a FATAL error. Config replication assumes local `rename(2)` atomicity +and `fsync(2)` durability guarantees; neither holds on NFS. No effort is made to +support network or distributed filesystems for config storage. Cross-filesystem +portability is a non-goal. + +``` +${workdir}/${name}/ +└── conf/ + ├── current ← plain text pointer: "cluster.bin.a1b2c3d4e5f6\n" + ├── cluster.bin.0f1e2d3c4a5b ← previous version (retained) + └── cluster.bin.a1b2c3d4e5f6 ← current version +``` + +The filename suffix is the **first 12 hex chars of the file's SHA-256 hash**. +`cat conf/current` gives the active version filename at a glance — identical to Git's +`HEAD` file pattern. Two standbys with identical `conf/current` contents are provably +running identical configs without parsing any binary content. + +> **Note**: The current code uses a flat path `conf/current_config` +> (`main.c:115`). This is replaced by a two-step resolution: read `conf/current` to +> get the versioned filename, then open that file. The `conffile` field in `myconf_t` +> is a **heap-allocated string** set via `asprintf()` during Phase 1 by +> `load_primary_config()` (primary path) or `fetch_config_from_primary()` (standby +> path) after reading the `conf/current` pointer file and resolving it to the +> versioned file path (e.g., `${workdir}/${name}/conf/cluster.bin.a1b2c3d4e5f6`). +> It is freed in `myconf_fini()` alongside the other owned fields. The two-step +> path resolution is load-bearing for the atomic `rename(2)` semantics described +> in §3.2 — the pointer file swap mutates which versioned file is current, and +> all code that needs config bytes resolves through `conf/current` → `conffile` +> to get the correct path. + +### 3.2 Write procedure (standby receiving new config) + +```c +// 0. Ensure conf/ directory exists with restrictive permissions +snprintf(conf_dir, ..., "%s/conf", effective_dir); +mkdir(conf_dir, 0700); /* same as data/wal/ and data/snap/ — trusted LAN model. + * On EEXIST, verify existing permissions with fstat() and + * chmod() to 0700 if needed — mkdir() silently ignores + * pre-existing directories with wrong permissions. */ + +// 1. Write new versioned file atomically +// Thread-unique .tmp suffix avoids truncation under concurrent push+fetch (§10.11) +snprintf(tmp_path, ..., "%s/conf/cluster.bin.%s.%lu.tmp", + effective_dir, hex_tag, (unsigned long)pthread_self()); +snprintf(ver_path, ..., "%s/conf/cluster.bin.%s", effective_dir, hex_tag); + +fd = open(tmp_path, O_WRONLY|O_CREAT|O_TRUNC, 0600); +if (fd < 0) { + ERROR("open(%%s) failed: %%s", tmp_path, strerror(errno)); + return -errno; +} +if (write(fd, bytes, len) != (ssize_t)len) { + ERROR("short write on %%s: %%s", tmp_path, strerror(errno)); + close(fd); unlink(tmp_path); + return -EIO; +} +if (fsync(fd) != 0) { + ERROR("fsync(%%s) failed: %%s", tmp_path, strerror(errno)); + close(fd); unlink(tmp_path); + return -EIO; +} +close(fd); +rename(tmp_path, ver_path); /* atomic: versioned file now on disk */ + +// 2. Atomically update the HEAD-style pointer file +// Thread-unique .tmp suffix avoids race under concurrent push+fetch (§10.11) +snprintf(ptr_tmp, ..., "%s/conf/current.%lu.tmp", effective_dir, (unsigned long)pthread_self()); +snprintf(ptr_path, ..., "%s/conf/current", effective_dir); +char target[NAME_MAX]; +snprintf(target, sizeof(target), "cluster.bin.%s\n", hex_tag); + +fd = open(ptr_tmp, O_WRONLY|O_CREAT|O_TRUNC, 0600); +write(fd, target, strlen(target)); +fsync(fd); +close(fd); +rename(ptr_tmp, ptr_path); /* atomic pointer swap — no symlink(2) required */ + +// 2.5. Durability: fsync the conf/ directory to persist the rename(2) +fd = open(conf_dir, O_RDONLY); +if (fd >= 0) { + fsync(fd); + close(fd); +} + +// 3. Trim old versioned files beyond keep_n (default: 2) +conf_trim_old_versions(effective_dir, keep_n); +``` + +**Read procedure** — called at startup to resolve `conffile`: + +```c +static int conf_read_current_filename(const char *conf_dir, + char *out, size_t outsz) +{ + char head_path[PATH_MAX]; + snprintf(head_path, sizeof(head_path), "%s/current", conf_dir); + FILE *f = fopen(head_path, "r"); + if (!f) return -ENOENT; + + /* Distinguish empty file from EOF — fgets returns NULL for both. + * An empty pointer file means the controller crashed between open() + * and the first write()+fsync()+close() in §3.2 step 2. */ + struct stat st; + if (fstat(fileno(f), &st) == 0 && st.st_size == 0) { + fclose(f); + ERROR("conf/current is empty — treating as missing config (§10.6a)"); + return -ENODATA; + } + if (!fgets(out, outsz, f)) { fclose(f); return -EIO; } + fclose(f); + out[strcspn(out, "\n")] = '\0'; /* strip trailing newline */ + + /* Validate: reject path separators, require "cluster.bin." prefix, + * require exactly 12 hex chars in the suffix. */ + if (strchr(out, '/')) { + ERROR("conf/current contains path separator — corrupted pointer file"); + return -EINVAL; + } + if (strncmp(out, "cluster.bin.", 12) != 0) { + ERROR("conf/current filename '%s' does not start with 'cluster.bin.'", out); + return -EINVAL; + } + const char *hex = out + 12; + if (strlen(hex) != 12 || strspn(hex, "0123456789abcdef") != 12) { + ERROR("conf/current suffix '%s' is not 12 hex chars", hex); + return -EINVAL; + } + return 0; +} +``` + +After calling this, `myconf->conffile` is set via `asprintf()` to +`"${conf_dir}/${out}"` — a heap-allocated copy that survives past the caller's +stack frame. It is freed in `myconf_fini()` alongside the other owned fields. + +**Crash safety**: if the process crashes after step 1 but before step 2, `conf/current` +still contains the old versioned filename — the standby boots on the previous config and +re-fetches on the next connection. If it crashes mid-step-1 (before the rename of the +`.tmp` file), the orphaned `.tmp` file is ignored on next boot. + +### 3.3 Retention + +Keep the last `conf_keep_n` versioned files on disk (default: 2 — current + one +previous). Trimming deletes the oldest by mtime after every successful update. +Stale `.tmp` files from interrupted writes are also cleaned during the trim pass. + +`conf_keep_n` is a field in `myconf_t` — it is needed during config save (between +Phase 1 and Phase 2), before `persist_init()` runs and `persist_config_t` is available. +It is initialised from a hardcoded default (2) or the `--conf-keep-n` CLI flag during +Phase 1, before any config is loaded. It is not read from the cluster config, so it +is available before config fetch on standbys. + +--- + +## 4. SHA-256 as Version and Integrity Token + +### 4.1 Why SHA-256 + +`configSeq` and CRC32 were both considered and rejected: + +| Property | configSeq | CRC32 | SHA-256 | +|---|---|---|---| +| Change detection | Compare ints | Compare CRC32 | Compare first 8 bytes | +| Integrity validation | Separate field | Same field (CRC) | Full 32-byte hash | +| Admin tool required | Yes | No | No | +| Push skip on match | No (always incr) | Yes (same content → same CRC) | Yes (same content → same hash) | +| Collision resistance | N/A | 2^-32 (vulnerable) | 2^-256 (none in practice) | +| Cryptographic strength | None | None (invertible) | Full (keyless integrity) | +| On-disk dependency | — | zlib `crc32()` | OpenSSL (already in Phase 1) | + +**Why cryptographic hash in a trusted-LAN model?** The trusted-LAN model (§5.3) +means we trust the network path between controllers, but config blobs traverse +additional boundaries where trust does not apply: admin workstations push configs +via `OP_upgradeConfig` (§6), config files may be hand-carried on USB drives for +air-gapped provisioning, and versioned files persist on disk across reboots where +bit rot or operator error (copying the wrong file) are risks. SHA-256 provides +end-to-end integrity across all of these paths without adding a separate checksum +mechanism — the same hash that detects version drift also validates integrity. +The implementation cost is negligible: OpenSSL is already initialised in Phase 1, +and SHA-256 of a <100 KB blob takes microseconds. + +SHA-256 is computed over the **raw bytes of the `cluster.bin.*` file** — not over any +in-memory struct: + +- The hash is **never stored inside** the Cap'n Proto payload (avoiding circularity). +- The **first 8 bytes** (UInt64, big-endian as raw SHA-256 bytes; Cap'n Proto wire format is little-endian — be64toh() converts from big-endian SHA-256 bytes to host order) serve + as the fast "version tag" for change detection and the versioned filename suffix + (first 12 hex chars). +- The **full 32 bytes** are sent in the protocol for the standby to verify + cryptographic integrity of the received blob. +- Since config is always written to a **new versioned file** (never overwritten in place, + see §3.2), the file content is **immutable** after creation. + The primary's cached `config_sha256` is always consistent with the bytes on disk. +- The primary recomputes SHA-256 over the bytes read at request time and compares to + the cached hash before sending — any discrepancy is logged as an error rather than + silently sending mismatched data. + +### 4.2 Zero-tag sentinel + +A **zero `version_tag` (UInt64)** is the "no config" sentinel. A standby with no saved +config sends `version_tag = 0`. The probability of a legitimate config producing a +zero first 8 bytes of SHA-256 is ~2^-64 — negligible. No special-case FATAL is required. + +### 4.3 SHA-256 API + +Available via `SHA256()` from OpenSSL (``), which is already initialised +in Phase 1 (`initialize_openssl()`). No new dependency. The hash is computed once at +startup and cached; the per-request recomputation is a safety double-check, not the +primary path. + +### 4.4 Monotonic generation counter for auditing + +SHA-256 alone prevents accidental mismatches but does not prevent **downgrade** — an +admin could push an older valid config blob (different SHA-256) and the primary would +accept it as an "upgrade." `ClusterConfig` carries a monotonic +`configGeneration` field (`UInt64`, default 1 on first save) as an **audit +indicator**, not an enforcement mechanism: + +- On each successful `OP_upgradeConfig`, the primary sets + `config_generation = c->myconf->config_generation + 1` in the decoded struct, + re-encodes the blob, and saves — so the generation strictly increases. +- The admin tool does not need to manage the generation counter — the primary + is the sole authority and always increments. The `configGeneration` field in + the incoming blob is ignored; the primary overwrites it with `current + 1` + before saving (§6.4 step 2b). The primary does **not** reject incoming blobs + based on their embedded generation value — `configGeneration` provides an + audit trail (visible in the Cap'n Proto blob for post-hoc inspection) but + does not mechanically prevent an operator from pushing an older config with + a different SHA-256. Operators who need downgrade prevention should compare + generation values out-of-band before calling `OP_upgradeConfig`. +- Crash recovery: the generation is embedded in the `cluster.bin` Cap'n Proto blob, + so it survives restarts without a sidecar file. +- Default value: `configGeneration` is set to 1 by `load_primary_config()` on first + load. If an existing config blob does not include the field (backward compat), the + primary treats it as generation 0 and re-encodes with generation 1 on the next save. + +Replay behaviour: a retried `OP_upgradeConfig` with identical config content +is accepted again — the primary re-encodes with an incremented generation, producing +a new SHA-256, and restarts. **This is not idempotent:** applying the same config +bytes twice produces different on-disk state (generation counter is incremented +both times) and triggers two restarts, not one. Systemd's `StartLimitBurst=3` +(§11.1) bounds restart storms to 3 within 60 seconds; the admin tool's retry +logic must implement deduplication (e.g., comparing the SHA-256 of the just-pushed +blob against the primary's reported `versionTag` in the reply) rather than blind +retry. The generation counter aids auditing of an operator accidentally pushing a stale +config fetched from another controller: the generation field embedded in the blob +provides a human-readable version indicator for auditing, even though the primary +does not reject blobs based on their generation value. + +--- + +## 5. OP_replConfig Protocol + +### 5.1 Schema additions + +```capnp +# controller.capnp + +enum OPCode { + ...existing entries... + replCatchUpDone @28; + replConfig @29; # ← new: standby → primary config pull + replConfigPush @30; # ← new: primary → standby config push (after OP_upgradeConfig) + upgradeConfig @31; # ← new: admin → primary live config upload +} + +struct ReplConfigRequest $C.mapname("repl_config_req_t") { + versionTag @0: UInt64 $C.mapname("version_tag"); # 0 = no config saved +} + +struct ReplConfigReply $C.mapname("repl_config_reply_t") { + upToDate @0: Bool $C.mapname("up_to_date"); # true = tag matched, no data + versionTag @1: UInt64 $C.mapname("version_tag"); # primary's tag (0 if upToDate) + sha256 @2: Data $C.mapname("sha256"); # full 32-byte SHA-256 (empty if upToDate) + bytes @3: Data $C.mapname("bytes"); # raw cluster.bin bytes (empty if upToDate) +} + +struct ReplConfigPushRequest $C.mapname("repl_config_push_req_t") { + versionTag @0: UInt64 $C.mapname("version_tag"); # primary's new tag + sha256 @1: Data $C.mapname("sha256"); # full 32-byte SHA-256 + bytes @2: Data $C.mapname("bytes"); # raw cluster.bin bytes +} + +struct ReplConfigPushReply $C.mapname("repl_config_push_reply_t") { + accepted @0: Bool $C.mapname("accepted"); # true = saved and verified + versionTag @1: UInt64 $C.mapname("version_tag"); # standby's tag after save +} + +struct UpgradeConfigRequest $C.mapname("upgrade_config_req_t") { + bytes @0: Data $C.mapname("bytes"); # raw cluster.bin blob + sha256 @1: Data $C.mapname("sha256"); # caller-computed SHA-256 (primary verifies) +} + +struct UpgradeConfigReply $C.mapname("upgrade_config_reply_t") { + versionTag @0: UInt64 $C.mapname("version_tag"); # new tag if accepted; 0 on error + error @1: Text $C.mapname("error"); # non-empty if rejected + ackedCount @2: UInt32 $C.mapname("acked_count"); # number of standbys that ACKed push +} +``` + +Add union arms to `Request.body` and `RequestReply.data` with explicit field numbers: +`replConfigRequestMsg @22: ReplConfigRequest`, +`replConfigReplyMsg @8: ReplConfigReply`, +`replConfigPushRequestMsg @23: ReplConfigPushRequest`, +`replConfigPushReplyMsg @9: ReplConfigPushReply`, +`upgradeConfigRequestMsg @24: UpgradeConfigRequest`, +`upgradeConfigReplyMsg @10: UpgradeConfigReply`. +The field numbers follow the next free slots in each union +(after `hostctrlmsg @21` in `Request.body`, +after `clusterconf @7` in `RequestReply.data`). +**Verified**: The OPCode enum currently ends at `replCatchUpDone @28`. +The `Request.body` union ends at `hostctrlmsg @21`. +The `RequestReply.data` union ends at `clusterconf @7`. +All proposed field numbers are free as of the current `controller.capnp` schema. + +### 5.2 Message flow + +**Pull (standby-initiated, at startup):** + +``` +Standby Primary + │ │ + │── OP_replConfig { version_tag=0xA3... } ─▶│ + │ │ compare: 0xA3 != 0xF1 + │◀── reply { up_to_date=false, │ + │ versionTag=0xF1..., │ + │ sha256=..., bytes=... } ───────│ + │ │ + │ validate: SHA256(bytes) == sha256 ✓ │ + │ write cluster.bin.f1e2d3c4a5b │ + │ update conf/current pointer file │ + │── OP_replConfig { version_tag=0xF1 } ───▶│ (optional confirm) + │◀── reply { up_to_date=true } ─────────────│ +``` + +The confirmation request is optional but provides assurance that the standby's +saved copy matches. Implementations may skip it. + +**If tag matches on first request:** + +``` + │── OP_replConfig { version_tag=0xF1 } ───▶│ + │◀── reply { up_to_date=true } ─────────────│ + │ proceed to cluster_init() phase 2 │ +``` + +**Push (primary-initiated, after OP_upgradeConfig — see §6):** + +``` +Primary Standby + │ │ + │── OP_replConfigPush { versionTag=0xF2, │ + │ sha256=..., bytes=... } ─────────────▶│ + │ │ validate SHA-256 + │ │ write cluster.bin.f2... + │ │ update conf/current + │◀── reply { accepted=true, │ + │ versionTag=0xF2 } ─────────────│ +``` + +For the push direction the primary sends an `OP_replConfigPush` request (a +dedicated opcode, not a reused reply) to each connected standby after saving +a new config via `OP_upgradeConfig`. The standby validates, saves, and ACKs +with a `ReplConfigPushReply` confirming the new tag. This preserves the +existing request-reply dispatch pattern in `myreq_handler()`. + +### 5.3 Authentication model + +`OP_replConfig` has no application-layer authentication beyond SSL transport-layer +encryption. Any controller that completes an SSL handshake (using a valid cert signed +by the shared CA) can request the full cluster config blob. This is consistent with the +trusted-LAN model adopted by the parent HA design — controller-to-controller sessions are +authenticated at the SSL layer, not at the application protocol layer. No additional +auth token, HMAC, or challenge-response is added here. + +`OP_upgradeConfig` (§6) is more sensitive — it changes the cluster configuration and +triggers a primary restart. Access control for `OP_upgradeConfig` will be enforced +through the existing `validate_host()` mechanism (currently a stub) when it is +enhanced — the upgrade handler calls `validate_host()` to check whether the peer +is authorised before processing the request. + +**Safety gate until `validate_host()` is implemented**: Since `validate_host()` is +a stub that returns success for all peers, `OP_upgradeConfig` is gated behind a +compile-time flag `CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled). Set to 1 +in the Meson build options (`-Denable-upgrade-config=true`) when the organisation +has implemented `validate_host()` and is ready for the feature. This prevents a +half-implemented admin endpoint from being reachable in production before its +authorisation check is complete. The compile-time gate is removed once +`validate_host()` has a production implementation (tracked in `FUTURE_WORK.md`). + +### 5.4 Primary handler + +The primary handles `OP_replConfig` via a new case in `cluster.c:myreq_handler()` (the +existing opcode dispatch switch). The handler is distinct from `handle_controller_join()` +which handles the join handshake. + +**Controller-controller join handshake** (`handle_controller_join()` in `join.c`): + +1. The standby opens a TCP connection to `primary_addr:primary_port` (from `--peers`) + and completes an SSL handshake (TLS transport per §9.0 `ssl_ctx`). +2. The standby sends `OPCode_join` with `HostRole_controller` and its own host + identity (hostname from `--peers`). This identifies the session as a + controller-controller (not client) connection. +3. The primary's join handler verifies the peer is a known controller (listed in + `--peers`) and returns `OPCode_join` reply with status 0. +4. After successful join, the primary appends the endpoint to + `c->ha.standby_endpoints` (replacing the current `-ENOSYS` stub — see + §6.3). The standby proceeds to dispatch `OP_replConfig` on the same + connection to fetch its config. + +The primary must have completed Phase 2 (iomgr + endpoint up) before it can +serve `OP_replConfig` or accept join handshakes. The standby retry policy (§5.5) +handles this startup timing gap. + +```c +static int handle_config_request(cluster_t *c, endpoint_t *ep, + xce_request_t *req) +{ + uint64_t their_tag = req->data.replConfigRequestMsg->version_tag; + uint64_t my_tag = c->myconf->config_version_tag; + + xce_reply_t reply; + init_reply_by_req(&reply, req); + + if (their_tag == my_tag) { + reply.status = 0; + reply.data.replConfigReplyMsg->up_to_date = true; + xce_reply(ep, c->myhost, &reply); + return 0; + } + + /* Read raw bytes from conffile and recompute SHA-256 for double-check */ + uint8_t *bytes; + size_t len; + int rc = conf_read_raw(c->myconf->conffile, &bytes, &len); + if (rc != 0) { + reply.status = rc; + xce_reply(ep, c->myhost, &reply); + return rc; + } + + /* Recompute to guard against external pointer-file or versioned-file replacement */ + uint8_t recomputed[32]; + SHA256(bytes, len, recomputed); + if (memcmp(recomputed, c->myconf->config_sha256, 32) != 0) { + ERROR("Config file changed since startup — refusing to send stale hash"); + free(bytes); + reply.status = -EIO; + xce_reply(ep, c->myhost, &reply); + return -EIO; + } + + reply.status = 0; + reply.data.replConfigReplyMsg->up_to_date = false; + reply.data.replConfigReplyMsg->version_tag = my_tag; + reply.data.replConfigReplyMsg->sha256 = /* recomputed, 32 */; + reply.data.replConfigReplyMsg->bytes = /* alloc_and_copy(bytes, len) */; + xce_reply(ep, c->myhost, &reply); + free(bytes); + return 0; +} +``` + +The full 32-byte SHA-256 and raw config bytes are `Data` fields in the capnp schema. +The codecgen generates `encode_ReplConfigReply_ptr()` / `decode_ReplConfigReply_ptr()` +which internally allocate and manage the `Data` blob storage in the capn segment. +The C-side shim in `src/lib/request.c` calls these generated functions. See §12 for all +affected files. + +`conf_read_raw()` and `CONF_MAX_SIZE` (defined in `config_repl.c`) are the canonical +entry points for reading config bytes — all callers that need raw config bytes (primary +handler, `load_primary_config()`, `conf_save_config()`) go through `conf_read_raw()`, +which enforces the size limit before allocating memory. `conf_read_raw()` returns +`-EFBIG` when the file exceeds `CONF_MAX_SIZE` — callers in request-handling context +(the primary handler) send an error reply to the peer; only `load_primary_config()` +(at startup) escalates this to `FATAL`. + +`c->myconf->config_version_tag` and `c->myconf->config_sha256` are computed once in +`load_primary_config()` and cached in `myconf_t`: + +```c +typedef struct { + cluster_config_t* cluster_config; + char* conffile; + char* certfile; + char* keyfile; + char* workdir; + char* name; + xce_cluster_config_t peers_cfg; + uint64_t config_version_tag; /* ← new: first 8 bytes of SHA-256 */ + uint8_t config_sha256[32]; /* ← new: full SHA-256 of conffile bytes */ + int conf_keep_n; /* ← new: config version retention count */ + uint64_t config_generation; /* ← new: monotonic anti-downgrade guard (§4.4) */ +} myconf_t; +``` + +### 5.5 Standby fetch + +The standby makes a request using a standalone `SSL_CTX` created from the same cert/key +pair during Phase 1. A dedicated pthread performs a **non-blocking `connect()`** with +`poll()` to enable early exit when the election determines a role before the connect +completes. This is the same mechanism used by the election probes (§9): + +1. Create standalone `SSL_CTX` from `certfile`/`keyfile` (already loaded in Phase 1). +2. Non-blocking `socket()` + `connect()` to the primary candidate. +3. `poll()` with timeout `probe_connect_timeout_ms` (500ms), polling + `election_done` between attempts. +4. Once connected, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` handling. +5. Send `OP_replConfig`, receive reply, join thread. + +No iomgr dependency — this runs after `am_i_primary()` resolves to STANDBY and before +Phase 2 (which starts iomgr). The non-blocking connect + poll + SSL_connect approach is +the same mechanism used by the election probes (§9) — the config fetch and election probe +code paths share this pattern deliberately so both benefit from the same error-handling +and timeout logic. + +**Retry policy**: if the primary is not yet ready (iomgr not up, still in Phase 2 +initialisation including `persist_recover()`), the TCP connection is refused or the +request times out. The standby retries with exponential backoff: + +``` +initial_delay_ms = 500 +max_delay_ms = 5000 +max_attempts = 20 +delay = min(initial_delay_ms * 2^attempt, max_delay_ms) +total window ≈ 88s (500+1k+2k+4k+16×5k = 87.5s) +``` + +The retry window is sized to accommodate the primary's Phase 2 startup time +(§8.5), specifically `persist_recover()` bounded by `max_wal_records` (see §13). +The primary must complete Phase 2 startup (iomgr, persist_recover, endpoint up) +before it can serve `OP_replConfig`; standbys retry during this window. +Note: `iomgr_init` and `endpoint_init()` complete early in Phase 2 (before +`persist_recover()`), so client connections can be dispatched before config +state is fully recovered. The `OP_replConfig` handler depends on config being +loaded and cached in `myconf` (populated before Phase 2), not on +`persist_recover()` completion — the handler reads from `myconf`, not from +WAL-recovered state. Post-recovery hooks (`HOOK_AFTER_EVENT`) do not affect +config fetch latency. + +Each retry iteration checks `atomic_load(&c->ha.role) == CTRL_STANDBY` before +attempting the connect. If the HA watchdog has promoted this node to primary +(e.g., the original primary was detected as failed during the retry window), +the fetch thread logs `INFO("Promoted to primary during config fetch — " +"aborting fetch")` and returns without error. The caller proceeds with +`load_primary_config()` on the primary path. + +If all attempts fail, the standby checks whether `conf/current` points to a valid +versioned file: +- **Yes** (saved config exists): log WARN and proceed — see §10.3. +- **No** (no saved config): treat as FATAL — see §10.2. + +--- + +## 6. OP_upgradeConfig Protocol + +### 6.1 Purpose + +`OP_upgradeConfig` allows an admin tool to upload a new `ClusterConfig` blob to the +running primary at runtime. The primary validates (SHA-256 transit integrity, Cap'n Proto +decode), increments the monotonic generation counter, saves, pushes to all connected +standbys (best-effort), then restarts itself to apply the new configuration. + +### 6.2 Message flow + +``` +Admin tool Primary Standbys (connected) + │ │ │ + │─ OP_upgradeConfig▶│ │ + │ (bytes, sha256) │ 1. verify SHA256(bytes) │ + │ │ 2. decode Cap'n Proto │ + │ │ 3. set gen=current+1, re-encode│ + │ │ 4. write cluster.bin. │ + │ │ 5. update conf/current │ + │ │ 6. update config_version_tag │ + │ │ 7. push to connected standbys │ + │ │── OP_replConfigPush ──────────▶│ + │ │ (versionTag, sha256, bytes) │ save to disk + │ │◀─ reply { accepted, tag } ─────│ + │◀─ ACK (new tag, acked_count) ──│ │ + │ │ │ + │ │ │ + │ │ 8. log "config upgraded, │ + │ │ requesting restart" │ + │ │ 9. set restart_requested=1 │ + │ │ cluster_run() exits loop │ + │ │ myapp_run() returns 0 │ + │ ↓ │ + │ [process exits; systemd/service manager │ + │ restarts with same args and new config]│ +``` + +### 6.3 Push semantics (hash-first, best-effort, parallel) + +The primary pushes the new config's hash and tag to all connected standbys **before** +ACKing the admin tool. This reduces the silent-rollback window: the admin receives +the ACK only after at least attempting to notify all connected standbys. The push +is best-effort and **parallel**: all standbys are sent the config simultaneously, +and the primary waits at most `CONF_PUSH_TIMEOUT_MS` (default 500 ms, compile-time +constant in `config_repl.c`) total for all ACKs. This bounds the pre-restart delay +to a single timeout period regardless of standby count. The 500ms default is derived +from the expected push workload: a single SSL write (config blob < 1 MiB) + standby +SHA-256 verify + disk fsync + SSL ACK reply, bounded by two intra-LAN RTTs (~1ms +each) + one local fsync (~50ms). The per-standby worst case is ~55ms; 500ms +accommodates up to 9 concurrent standbys or a single degraded standby. + +**Push-then-ACK ordering**: The primary saves the new config, pushes to standbys, +then ACKs the admin tool. The ACK includes the number of standbys that confirmed +the push (`acked`), so the admin tool can detect when zero standbys received the +update. The ACK reply is extended with: +```capnp +struct UpgradeConfigReply $C.mapname("upgrade_config_reply_t") { + versionTag @0: UInt64 $C.mapname("version_tag"); # new tag if accepted; 0 on error + error @1: Text $C.mapname("error"); # non-empty if rejected + ackedCount @2: UInt32 $C.mapname("acked_count"); # number of standbys that ACKed push +} +``` +If `ackedCount == 0` and standbys are expected, the admin tool logs a WARNING +that the config may not have reached any standby. + +```c +/* Fan-out: launch one pthread per connected standby */ +for (int i = 0; i < n_standbys; i++) + pthread_create(&push_threads[i], NULL, push_config_thread, &push_ctx[i]); + +/* Single aggregate wait — 500ms total, not 500ms per standby. + * Uses CONF_PUSH_TIMEOUT_MS (default 500ms), distinct from probe_connect_timeout_ms + * used in election probing (§9.2). */ +wait_all_with_timeout(push_threads, n_standbys, CONF_PUSH_TIMEOUT_MS); + +/* Cancel still-running threads via atomic stop flag (same pattern as probe_thread) */ +atomic_store(&push_done, 1); +for (int i = 0; i < n_standbys; i++) + pthread_join(push_threads[i], NULL); + +LOG("Config pushed to %d/%d standbys; restarting", acked, n_standbys); +``` + +Offline standbys recover on their next reconnect via the normal pull mechanism +(§5.2) — their `conf/current` will have the old tag, the new primary will detect +the mismatch and push via `OP_replConfig`. + +**Failure mode — push not received + primary fails to restart**: If push to all +standbys times out (e.g., all standbys offline) AND the primary fails to come back +after restart (e.g., the new config triggers a Phase 2 FATAL), the HA watchdog on the +standbys promotes one of them. The promoted standby boots with its **saved config** +— which is the old version, since it never received the push. The cluster silently +rolls back to the previous config. The admin tool now receives `ackedCount == 0` in +the ACK reply, providing a clear signal that no standby received the update. If +`ackedCount > 0`, at least one standby has the new config and will have it available +when it promotes. The silent rollback scenario is limited to `ackedCount == 0` cases +where all standbys were offline during the push. Future work may add a quorum-ACK +gate that refuses to restart unless a minimum number of standbys confirm (§14.4). + +**Standby endpoint tracking**: The primary maintains a list of connected standby +controller endpoints in `ha_state_t` as a `zarray_t *standby_endpoints` (using the +existing `zarray_t` infrastructure, which already manages `c->endpoints` for +client connections). This is Phase 2 infrastructure — needed only for +`OP_replConfigPush` fan-out during live config upgrade. This list is populated +and pruned by the join/disconnect handlers: + +- **Join**: `handle_controller_join()` (join.c) appends the new controller + endpoint to `c->ha.standby_endpoints` after completing the join handshake. + This replaces the current `-ENOSYS` stub. +- **Disconnect**: `handle_endpoint_error()` removes the endpoint from + `standby_endpoints` when a controller-controller connection drops. + The existing `HostRole_controller` case in `handle_endpoint_error()` + (cluster.c) must be updated to scan `c->ha.standby_endpoints` for the + matching endpoint and call `zarray_del()` before the `break`. +- **Usage**: `push_config_to_standbys()` iterates `standby_endpoints` to obtain + `n_standbys` and the per-standby context for the fan-out push threads. + + **Thread safety**: `push_config_to_standbys()` takes a **snapshot** of + `standby_endpoints` under `c->ha.lock` before spawning push threads. This + prevents a concurrent join or disconnect handler from modifying the array + while threads are iterating it. The snapshot is a shallow copy of endpoint + pointers into a stack-allocated `zarray_t`; endpoints pinned by active + push threads are protected from `endpoint_free()` by the endpoint's + internal refcount. After all push threads complete, the snapshot array + is freed. + +Only endpoints that completed the SSL handshake and join handshake are in this +list — the SSL layer provides authentication (per §5.3 trusted-LAN model). + +### 6.4 Handler (primary side) + +```c +static int handle_upgrade_config(cluster_t *c, endpoint_t *ep, + xce_request_t *req) +{ + const uint8_t *bytes = req->data.upgradeConfigRequestMsg->bytes.ptr; + size_t len = req->data.upgradeConfigRequestMsg->bytes.len; + const uint8_t *sha256 = req->data.upgradeConfigRequestMsg->sha256.ptr; + + /* 0. Reject oversized blobs before hashing */ + if (len > CONF_MAX_SIZE) { + send_upgrade_error(ep, req, "Config blob exceeds CONF_MAX_SIZE"); + return -EINVAL; + } + + /* 0b. Validate input field lengths before accessing pointers. + * Cap'n Proto Data fields with default/empty value have ptr=NULL, + * len=0. Gated behind CONF_ENABLE_UPGRADE_CONFIG (default 0, §5.3). */ + if (len == 0 || req->data.upgradeConfigRequestMsg->sha256.len != 32) { + send_upgrade_error(ep, req, "Invalid request: bytes or sha256 field missing"); + return -EINVAL; + } + + /* 0a. Authorisation check — gated behind CONF_ENABLE_UPGRADE_CONFIG + * (default 0, §5.3). validate_host() is a stub that returns + * success for all peers until enhanced with a real check. + * When the compile-time gate is removed, this call is mandatory. */ + if (!validate_host(ep, c)) { + send_upgrade_error(ep, req, "Not authorised — peer not in allowed hosts"); + return -EPERM; + } + + /* 1. Verify caller-supplied SHA-256 */ + uint8_t computed[32]; + SHA256(bytes, len, computed); + if (memcmp(computed, sha256, 32) != 0) { + send_upgrade_error(ep, req, "SHA-256 mismatch — blob corrupted in transit"); + return -EINVAL; + } + + /* 2. Basic Cap'n Proto decode to catch malformed blobs */ + cluster_config_t *new_cfg = decode_cluster_config(bytes, len); + if (!new_cfg) { + send_upgrade_error(ep, req, "Failed to decode ClusterConfig blob"); + return -EINVAL; + } + + /* 2.5 Minimum viability check — rejects trivially invalid configs + * that would cause Phase 2 FATAL on restart (§10.7 semantic validation + * is a non-goal; this only catches the most basic misconfiguration). */ + if (new_cfg->queueCount == 0) { + free_cluster_config(new_cfg); + send_upgrade_error(ep, req, "Config requires at least one queue"); + return -EINVAL; + } + if (new_cfg->port < 1024 || new_cfg->port > 65535) { + free_cluster_config(new_cfg); + send_upgrade_error(ep, req, "Config port out of valid range (1024–65535)"); + return -EINVAL; + } + + /* 3. Re-encode with incremented generation and recompute SHA-256. + * The primary is the sole authority for generation numbers — the admin + * tool's configGeneration field is ignored; we always use current+1. */ + uint64_t new_generation = c->myconf->config_generation + 1; + new_cfg->config_generation = new_generation; + uint8_t *new_bytes = NULL; + size_t new_len = 0; + int rc = encode_cluster_config(new_cfg, &new_bytes, &new_len); + free_cluster_config(new_cfg); + if (rc != 0) { + send_upgrade_error(ep, req, "Failed to re-encode config"); + return rc; + } + SHA256(new_bytes, new_len, computed); + /* Use computed (fresh SHA-256) for tag and save; caller-supplied sha256 + * was validated for transit integrity above and is no longer needed. */ + + /* 4–6. Save and update pointer */ + char hex_tag[13]; + bytes_to_hex(computed, 6, hex_tag); /* 6 source bytes → 12 hex chars + NUL */ + rc = conf_save_config(c->myconf, new_bytes, new_len, computed); + if (rc != 0) { + free(new_bytes); + send_upgrade_error(ep, req, "Failed to save new config"); + return rc; + } + /* Update conffile to point to the newly saved versioned file. + * Without this, handle_config_request() (§5.4) would read the old + * versioned file and its recomputed SHA-256 would mismatch + * c->myconf->config_sha256 (already updated below), causing + * spurious -EIO replies to standbys during the restart window. */ + free(c->myconf->conffile); + asprintf(&c->myconf->conffile, "%s/conf/cluster.bin.%s", + c->myconf->workdir, hex_tag); + uint64_t new_tag; + memcpy(&new_tag, computed, 8); + /* stored big-endian; swap to host order */ + new_tag = be64toh(new_tag); + c->myconf->config_version_tag = new_tag; + memcpy(c->myconf->config_sha256, computed, 32); + c->myconf->config_generation = new_generation; + + /* 7. Best-effort parallel push to connected standbys. + * Push runs BEFORE ACK so the admin tool knows how many standbys + * received the update. The reply includes acked_count. If the + * primary crashes before push, standbys catch up on their + * next reconnect via the normal pull path (§5.2). */ + int acked = push_config_to_standbys(c, new_bytes, new_len, computed, new_tag); + free(new_bytes); + + /* 8. ACK the admin — includes acked_count so the admin tool knows + * how many standbys confirmed the push. A zero acked_count with + * standbys expected is a WARNING condition (see §6.3). */ + send_upgrade_reply(ep, req, new_tag, acked); + + /* Flush the reply to the kernel TCP buffer before signaling restart. + * Without this, the iomgr shutdown path may close the endpoint before + * the ACK bytes are flushed, causing a connection reset. The admin + * tool would be unable to distinguish success from failure. */ + endpoint_flush(ep); /* block until kernel send buffer is drained */ + + /* 9. Request restart — controller is managed by systemd or equivalent + * service manager. cluster_run() exits its event loop when this flag is + * set, myapp_run() returns cleanly, and the service manager restarts + * the process with the same arguments. On restart, conf/current points + * to the new config versioned file. No execve() or saved-argv needed. + * + * IMPORTANT: The main event loop blocks on cbuf_get(&c->wbuf, &stop) + * (mylock.c pthread_cond_wait). Setting restart_requested alone cannot + * unblock it. We push a stop signal via cbuf_put — the same pattern + * used by the existing OPCode_shutdown handler (cluster.c:152-157). + * cluster_run() checks restart_requested after the loop exits to + * distinguish a clean config-upgrade restart from a shutdown. */ + /* peer_cn is the X.509 certificate Common Name — sanitise before logging + * to prevent log injection via crafted cert CN containing newlines or + * control characters. Limit to 64 printable ASCII chars. */ + LOG("Config upgraded to tag=%016"PRIx64" by CN=%s from %s:%u — requesting restart", + new_tag, sanitise_cn(peer_cn, 64), peer_ip, peer_port); + c->restart_requested = 1; + int stop = 1; + cbuf_put(&c->wbuf, &stop); /* unblock cluster_run() main loop */ + return 0; +} +``` + +### 6.5 WAL compatibility after config change + +`persist_recover()` replays WAL records against the in-memory state built from the +new config. If the new config **removes** a queue or resource whose ID appears in +existing WAL records, replay encounters an unknown ID. + +**Distinction from existing skip-on-unknown-type behavior** + +The existing `persist_recover()` already skips WAL records whose *type* is unknown +(a forward-compatibility provision for format extensions). This design adds a +separate handling strategy for *ID* references within known record types: + +| Condition | Existing behavior | This change | +|---|---|---| +| Unknown WAL record type | WARN + skip (safe — future format) | Unchanged | +| Known record type, unknown queue ID | Not checked today | **Route to `lost_and_found` queue** | +| Known record type, unknown resource/host ID | Not checked today | **Route to `lost_and_found` host or PEND** | + +Unknown-type records are safe to skip because the type may be from a newer format. +Unknown-ID references were previously treated as FATAL, but this made config upgrade +operationally fragile — a single zombie job referencing a removed resource would brick +the controller permanently (it cannot start because of FATAL, and it cannot upgrade +config because that would also FATAL). + +**Routing strategy for unknown IDs** + +Instead of FATAL, `apply_wal_entry()` routes unknown-ID references as follows: + +- **Unknown `queue_id`**: The job is placed in a built-in `lost_and_found` queue. + The `lost_and_found` queue is always present in the queue table regardless of + config content. Operators can inspect it after recovery and reassign or cancel + jobs at their discretion. + +- **Unknown `host` (node name)**: The job's host is set to `lost_and_found`, + a synthetic host entry that accepts jobs but marks them as requiring operator + attention. + +- **Unknown `resource_id`**: The job's resource assignment is cleared and the + job is set to `PEND` state. The job will pend until an operator re-creates + the resource in the config or cancels the job. A WARN log identifies the + orphaned resource reference: + +``` +WARN("WAL record references unknown queue id=%u at offset %zu in %s — " + "routing job to lost_and_found queue", + queue_id, record_offset, wal_path) + +WARN("WAL record references unknown resource id=%u at offset %zu in %s — " + "clearing resource assignment, job set to PEND", + resource_id, record_offset, wal_path) + +WARN("WAL record references unknown host '%s' at offset %zu in %s — " + "routing to lost_and_found host", + host_name, record_offset, wal_path) +``` + +**`lost_and_found` queue and host** + +These are synthetic entries created by `cluster_init()` Phase 2 after `kvmgr makeInstance` +and **before `persist_recover()`** (see startup diagram in §7). This ordering +is load-bearing: unknown queue/resource/host IDs encountered during `persist_recover()` must +have valid targets to route to. Consider adding a runtime assertion in `create_lost_and_found()` +that validates `kvmgr` is initialized and `persist_recover()` has not yet been called. They are not part of the admin-supplied +config blob — they are added by the controller itself. This ensures they always +exist, even after a config change that removes all user-defined queues or hosts. +Their existence is not persisted across restarts; they are re-created on each boot. + +**Implementation**: `create_lost_and_found()` adds entries to kvmgr's queue table and +host table via the existing `queue_insert()` and `host_insert()` APIs, using reserved +names `"lost_and_found"`. These are normal kvmgr entries — no special-casing in the +queue/host lookup paths is required. **Name reservation**: the name `"lost_and_found"` +is permanently reserved. The `OP_upgradeConfig` handler rejects any `ClusterConfig` +blob containing a queue or host named `"lost_and_found"` — the admin tool receives +an error: `"Reserved name: 'lost_and_found' queue/host names are reserved by the +controller. Rename the user-defined entry and retry."` This prevents the ambiguity +between user-defined and synthetic entries at config ingestion time rather than at +WAL replay time. `create_lost_and_found()` always creates the synthetic queue and +host; if the name somehow exists (e.g., loaded from a pre-upgrade saved config), +it skips creation and logs ERROR. After creation, standard lookup functions +(`get_queue_by_name()`, `get_host_by_name()`) resolve them. + +**Enforcement at all entry points**: Both `load_primary_config()` (primary at startup, §7.1) +and `handle_upgrade_config()` (live upgrade, §6.4) perform the same `config_contains_reserved_name()` +check. This ensures the synthetic `lost_and_found` queue and host cannot be aliased with a +user-defined entry regardless of how config enters the system — cold boot or live push. +The check is implemented once in `config_repl.c` and called from both paths. + +**Routing helper**: A `resolve_queue_or_lost_and_found(cluster_t *c, uint32_t queue_id)` +helper in `config_repl.c` is called inside `apply_wal_entry()` (persist.c) and +`restore_job()` (job.c) when a WAL record references an unknown queue ID. The helper: + +1. Looks up `queue_id` in the queue table. +2. If found, returns the queue pointer. +3. If not found, logs a WARN (with record offset and WAL path, per §6.5 examples) and + returns the `lost_and_found` queue pointer. + +Analogous helpers for host resolution (`resolve_host_or_lost_and_found()`) and resource +resolution (`clear_resource_and_pend()`) follow the same pattern. All three are called +from the WAL apply path in `persist_recover()` → `apply_wal_entry()`. + +**Observability**: The `lost_and_found` queue and host are discoverable via +`xcecli queue list` and `xcecli host list` — they appear as normal entries with +the reserved name `"lost_and_found"`. A metric counter `lost_and_found_job_count` +is incremented each time a job is routed to this queue during `persist_recover()`. +If `lost_and_found_job_count > 0` at startup, a WARN summary is logged. The +counter is stored in `cluster_t` as an `atomic_int`, initialised to 0 at +startup and incremented during `persist_recover()`. It is not persisted across +restarts. The counter is reset to 0 after a successful drain operation +(e.g., the future `xcecli lost-and-found drain` command reassigns or cancels +all sidelined jobs). Jobs accumulate across config upgrades (each upgrade writes new versioned files but +the WAL is truncated after checkpoint — see §13.4), so repeated upgrades do not +cause unbounded growth. Operators should inspect and drain `lost_and_found` after +a config upgrade; a future `xcecli lost-and-found drain` command can automate +reassignment or cancellation (§14.4 future work). + +### 6.6 Downtime estimate for config upgrade + +The primary is unavailable from when it sets `restart_requested` until Phase 2 +completes on restart: **~4–7 seconds nominal, ~8–13 seconds with 1000 jobs** (per +§8.5 timing table). This is the same duration as a cold start — the process exits +cleanly and systemd restarts it with the same arguments. + +Standbys do **not** restart during config upgrade — they receive the new config via +`OP_replConfigPush` and save it to disk, but continue running with their current +in-memory state. If the primary fails to come back (e.g., the new config is corrupt), +the HA watchdog on the standbys will detect the primary's absence and trigger an +election. The new primary then boots with the already-saved config. + +The admin tool receives the ACK before the restart begins (§6.4 step 6), so the +caller knows the config was accepted within `probe_connect_timeout_ms` (500ms for +push fan-out + network round-trip). The restart itself is asynchronous — the admin +tool should poll `OP_probe` or check the primary's log to confirm it's back online. + +### 6.7 In-flight WAL checkpoint during config-upgrade restart + +If an async WAL checkpoint (§13.4) is in progress when `handle_upgrade_config()` +triggers restart, the restart must not interrupt the checkpoint mid-write (which +could leave a partial snapshot file on disk). + +**Handling**: before setting `restart_requested`, the handler waits for any +in-flight checkpoint to complete: + +```c +/* Wait for in-flight checkpoint before restarting. + * schedule_checkpoint() is async via vtaskmgr; the checkpoint task sets + * persist->checkpoint_done when finished (same atomic used by the + * idempotency guard in schedule_checkpoint()). */ +struct timespec ckpt_wait_start, ckpt_wait_now; +clock_gettime(CLOCK_MONOTONIC, &ckpt_wait_start); +while (atomic_load(&c->persist->checkpoint_in_progress)) { + usleep(10000); /* 10ms spin — checkpoint is I/O-bound, completes quickly */ + clock_gettime(CLOCK_MONOTONIC, &ckpt_wait_now); + if ((ckpt_wait_now.tv_sec - ckpt_wait_start.tv_sec) > 30) { + ERROR("Checkpoint did not complete within 30s — restarting anyway"); + break; + } +} +``` + +This is safe because: +- The checkpoint is bounded: it flushes in-memory state and truncates the WAL, + completing in well under 1 second on typical hardware. +- The spin-wait does not hold any locks — `handle_upgrade_config()` runs on the + admin connection's thread, not the main event loop. +- If the checkpoint hangs (bug), the 30s timeout unblocks the restart; the + controller is not permanently stuck. The admin tool's connection timeout + provides a second bound. +- After the checkpoint completes, the WAL is empty (truncated at checkpoint), + so the next `persist_recover()` on restart replays zero records — the fresh + snapshot already contains all state. + +--- + +## 7. Startup Sequence + +The startup sequence for both roles is updated from the current monolithic +`cluster_init()` to a two-phase model. + +``` +Controller starts + │ + ▼ + Parse --peers → my_priority, my_port + │ + ▼ + Phase 1 — minimal init (no config, no iomgr) + ┌─────────────────────────────────────────┐ + │ xce_hook_init + xce_install_hooks │ + │ initialize_openssl │ + │ ssl_ctx = create_ssl_context(cert, key) │ + │ ha_init(&c->ha, &ha_cfg, my_priority) │ + │ c->ha.ssl_ctx = ssl_ctx /* stash */ │ + │ (c->ha.ssl_ctx and c->iomgr.ssl_ctx │ + │ are the same pointer; Phase 2 calls │ + │ iomgr_set_ssl_ctx(c->iomgr, ssl_ctx) │ + │ for the existing iomgr path. The │ + │ ha.h "no new TLS" comment is removed; │ + │ see §11 ha.h) │ + │ ha_install_sigusr1 │ + └─────────────────────────────────────────┘ + │ + ▼ + am_i_primary()? ← real election probing (§9) + │ + ┌────┴────────────────────────────────────┐ + │ │ +PRIMARY STANDBY + │ │ + ▼ ▼ +load_primary_config() INFO("Starting as standby + ← reads conf/current (priority %d); requesting + ← computes SHA-256 config from primary %s:%u", + ← FATAL if missing my_priority, host, port) + │ │ + │ fetch_config_from_primary() + │ ← OP_replConfig exchange + │ ← save to conf/ (§3.2) + │ ← retry with backoff (§5.5) + │ ← FATAL if no config at all + │ │ + └────────────────┬────────────────────────┘ + │ + ▼ + Phase 2 — full init (config available) + ┌─────────────────────────────────────┐ + │ iomgr_init │ + │ endpoint_init(port from --peers) │ + │ HOOK_BEFORE_EVENT → load resources │ + │ (must check cluster_config!=NULL │ + │ and early-return — config already │ + │ loaded before Phase 2) │ + │ kvmgr makeInstance (sizes from cfg) │ + │ xstring_table_init, job_obj_init │ + │ create_lost_and_found() │ + │ ← lost_and_found queue + host │ + │ ← always present regardless of │ + │ config content (§6.5) │ + │ vtaskmgr_init, delaymgr_init │ + │ persist_init │ + │ ha_promote (primary) or │ + │ CTRL_STANDBY (standby) │ + │ HOOK_AFTER_EVENT → build queues, │ + │ groups, load jobs │ + │ persist_recover │ + └─────────────────────────────────────┘ + │ + ▼ + cluster_start() +``` + +### 7.1 Primary config load — fail-fast behaviour + +`load_primary_config()` replaces `get_default_conf()` on the primary path: + +```c +static int load_primary_config(cluster_t *c) +{ + if (access(c->myconf->conffile, F_OK) != 0) { + FATAL("Primary candidate (priority 0) has no config file '%s' — " + "provision the config before starting the primary controller", + c->myconf->conffile); + return -ENOENT; + } + /* read, decode Cap'n Proto, compute SHA-256, cache version_tag + full hash in myconf */ + ... +} +``` + /* After decoding, check for reserved "lost_and_found" name — same check + * as OP_upgradeConfig handler (§6.5). Reject at boot rather than + * silently aliasing the synthetic queue/host with user-defined entries. */ + if (config_contains_reserved_name(new_cfg)) { + FATAL("Config file '%s' contains reserved name 'lost_and_found' — " + "rename the user-defined queue or host and retry", + c->myconf->conffile); + return -EINVAL; + } + +A priority-0 node with a missing config file must not start — it would boot with +empty queues and resources, silently losing all cluster configuration. + +### 7.2 Standby INFO log on startup + +When a standby starts and its election probe detects a live higher-priority candidate: + +``` +INFO("HA: higher-priority candidate %s:%u is alive — starting as standby " + "(my_priority=%d)", primary_host, primary_port, my_priority) +INFO("HA: requesting cluster config from primary %s:%u (my_tag=%016"PRIx64")", + primary_host, primary_port, saved_tag) +``` + +After successful config fetch: + +``` +INFO("HA: config received and saved (tag=%016"PRIx64") — proceeding with full init", + new_tag) +``` + +--- + +## 8. cluster_init() Phase Split + +### 8.1 What moves to phase 1 (before config) + +From the current `cluster_init()` (`cluster.c`): + +| Line(s) | Code | Phase | +|---|---|---| +| 544–546 | `xce_hook_init`, `xce_install_hooks` | 1 | +| 557–561 | `initialize_openssl`, `create_ssl_context` | 1 | +| 665–671 | `ha_init`, `ha_install_sigusr1` | 1 | +| new | `zarray_create(&c->ha.standby_endpoints, 4, NULL)` — initialise the standby endpoint tracking array in `ha_state_t` before any join handler can append to it (§6.3). Uses NULL killer (endpoints are freed by `endpoint_free()` called by the iomgr disconnect path, not by zarray) | 1 | + +### 8.2 What stays in phase 2 (after config) + +| Line(s) | Code | Phase | +|---|---|---| +| new (before iomgr) | Stop heartbeat listener: set `atomic_store(&c->heartbeat_listener_running, 0)` and `pthread_join(c->heartbeat_listener_tid, NULL)` — the join must complete before `iomgr_init` to free the listening port and prevent `EADDRINUSE` | 2 | +| 580 | `iomgr_init` | 2 | +| 589–594 | `endpoint_init` (port from `--peers`) | 2 | +| 548–552 | `xce_run_hooks HOOK_BEFORE_EVENT` → in `xce_config_init()` (cluster.c:411-417), add `if (c->myconf->cluster_config != NULL) return 0;` guard before the existing `get_default_conf(c)` call. Config is already loaded by `load_primary_config()` or `fetch_config_from_primary()` before Phase 2. The HOOK_AFTER_EVENT case (cluster.c:419-446: `cluster_build_config`/resources/queues/groups/`load_jobs`) is unchanged. The static global `cf` in myconf.c and its associated HOOK_AFTER_EVENT SHUTDOWN cleanup become dead code — `myconf_fini()` takes over freeing `cluster_config`. | 2 | +| 563–579 | `kvmgr makeInstance` (sizes from config) | 2 | +| 584–585 | `xstring_table_init`, `job_obj_init` | 2 | +| 602–608 | `vtaskmgr_init`, `delaymgr_init` | 2 | +| 614–629 | `persist_init` | 2 | +| 674–679 | `ha_promote_to_primary` or `CTRL_STANDBY` | 2 | +| 681–683 | `xce_run_hooks HOOK_AFTER_EVENT` | 2 | +| new | `create_lost_and_found()` — synthetic queue + host (§6.5) — **must complete before `persist_recover()` begins; see §6.5 ordering note** | 2 | +| 686–691 | `persist_recover` | 2 | + +### 8.3 kvmgr sizing with default values + +The kvmgr hash slot counts (`resmgr->nhash_slots`, `jobmgr->nhash_slots`, etc.) come +from `ClusterConfig`. The slot count only affects hash collision rate — it has no +effect on correctness. A standby that has never received config can safely use the +same defaults as `fill_default_cluster_config()` while waiting, then re-initialize +with the correct sizes once config arrives. + +In practice, the standby receives config before phase 2 begins, so it always uses +the correct sizes. The default fallback is belt-and-suspenders only. + +### 8.4 Listening port + +`endpoint_init()` currently reads the port from `cluster_config->port`. After the +split, phase 2 derives the port from the `--peers` candidate list (available from +CLI parsing before any phase): + +```c +/* Find own entry in peers list by name match */ +uint32_t my_port = 0; +for (int i = 0; i < pcfg->candidates_count; i++) { + if (strncmp(pcfg->candidates[i].name, + c->myconf->name, XCE_MAXNAME_LEN - 1) == 0) { + my_port = pcfg->candidates[i].port; + break; + } +} +if (my_port == 0) my_port = 9990; /* design default */ +``` + +This is already available in `myconf.peers_cfg` before any config is loaded. +`xce_candidate_t` (defined in `src/include/xce/cluster.h`) carries a `port` field +populated by `xce_parse_peers()` from the `name=host:port` CLI syntax — confirmed in +the struct definition at line 21 of that header. No new parsing is needed. + +### 8.5 Phase 2 timing decomposition + +The standby retry window must be sized to survive Phase 2 startup on the primary. +Each Phase 2 component contributes: + +| Component | Nominal | With 1000 jobs | +|---|---|---| +| `iomgr_init` + `endpoint_init` | <50ms | <50ms | +| `kvmgr makeInstance` ×5 | <100ms | <100ms | +| `xstring_table_init` + `job_obj_init` | <50ms | <50ms | +| `HOOK_BEFORE_EVENT` (load resources) | <500ms | <2s | +| `vtaskmgr_init` + `delaymgr_init` + `apiserver_init` | <200ms | <500ms | +| `persist_init` | <100ms | <200ms | +| `persist_recover` (bounded by §13, ≤10000 records) | 2–5s | 2–5s | +| `HOOK_AFTER_EVENT` (queues, groups, jobs) | <1s | <5s | +| **Total** | **~4–7s** | **~8–13s** | + +The ~88-second standby retry window provides 7×–11× headroom over the worst case. +Operators of very large clusters (10000+ jobs) should verify Phase 2 duration from +the primary's startup log and tune `max_attempts` × `max_delay_ms` accordingly. + +--- + +## 9. am_i_primary() Implementation + +The current `am_i_primary()` stub (`ha.c:213`) returns `c->ha.my_priority == 0`, +which is never correct for a multi-node cluster — it never probes whether a +higher-priority candidate is actually alive. + +### 9.0 Simultaneous-boot symmetry breaking + +When all candidates start simultaneously, nobody is primary yet and nobody responds +to election probes. Without symmetry breaking, every candidate would expire its +deadline and self-declare primary — a multi-way split-brain. + +The mechanism has two parts: + +**1. Dedicated heartbeat listener (all self-declared primaries)** + +After Phase 1, **every candidate that `am_i_primary()` returns true for** launches a +minimal heartbeat listener thread before entering Phase 2. This is not limited to +priority-0 — if a lower-priority candidate wins the jitter race and self-declares +primary, its heartbeat listener is essential for the higher-priority true primary +to detect the split-brain during its own election probe. The listener is started +in a dedicated pthread immediately after `am_i_primary()` returns: + +```c +if (am_i_primary()) { + /* Start a minimal SSL listener that responds to OP_heartbeat probes. + * This lets lower-priority candidates detect us while Phase 2 initializes. */ + start_heartbeat_listener(c); /* dedicated pthread — non-blocking poll loop */ + load_primary_config(c); /* FATAL if missing */ + ... +} +``` + +The heartbeat listener is fully specified in the parent HA design +(`persistence-and-ha-design.md` Part 5). This section only describes the +integration point relevant to config replication: the listener must be active +before the standby's `fetch_config_from_primary()` connects, so the standby can +reach a primary that is still in Phase 2 initialisation. Summary of the +listener's properties (see parent design for full specification): + +- Binds the port from `--peers` using `c->ha.ssl_ctx` +- Non-blocking poll loop; accepts `OP_heartbeat` only; rejects other opcodes +- Does NOT depend on iomgr, vtaskmgr, persist, or any Phase 2 component +- Exits when the main thread clears `atomic_int heartbeat_listener_running` + (set to 0 before `iomgr_init` in Phase 2). The main thread calls + `pthread_join(heartbeat_listener_tid, NULL)` before `endpoint_init()` to + prevent EADDRINUSE. +- Registered in `cluster_t` (`c->heartbeat_listener_tid`, + `c->heartbeat_listener_running`) so `cluster_fini()` can join the thread + on any FATAL exit path before Phase 2 + +**2. Priority-based base delay + randomized jitter (all candidates)** + +Each candidate's election timeout has two components: a deterministic +**priority-based base delay** (higher priority = shorter delay) plus a small +random jitter for tie-breaking: + +```c +/* Priority-based base delay: priority-0 waits 0ms, priority-1 waits + * election_priority_step_ms, priority-2 waits 2× that, etc. + * This ensures the highest-priority node nearly always wins without probing. */ +long to_ms = c->ha.config.election_timeout_ms; +to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms; + +/* Small random jitter for tie-breaking between same-step candidates */ +long jitter_ms = (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms); +to_ms += jitter_ms; +``` + +With `election_timeout_ms = 2000`, `election_priority_step_ms = 500`, and +`election_jitter_max_ms = 200`, the effective deadlines are: +- Priority-0: 2000ms–2199ms (self-declares primary first) +- Priority-1: 2500ms–2699ms (probes priority-0 first; if alive, becomes standby) +- Priority-2: 3000ms–3199ms (probes priority-0 and priority-1 first) + +The priority-based base delay all but eliminates simultaneous split-brain during +normal boots: priority-0 declares primary and starts its heartbeat listener +~500ms before priority-1's deadline, so priority-1's probe finds a live primary. +The priority-based step and jitter together make simultaneous split-brain +vanishingly unlikely for N=3 in systemd-managed deployments: + +- Priority-0 fires at t=0–199ms (election_timeout_ms + jitter only) +- Priority-1 fires at t=500–699ms (step + jitter, probes priority-0 first) +- Priority-2 fires at t=1000–1199ms (2×step + jitter) + +Adjacent-priority candidates collide only if the lower-priority node starts so +late that its jitter-extended deadline precedes the higher-priority node's first +heartbeat response, which requires the 500ms step to be less than 2×J_max +(400ms) — false for the configured values. For three nodes to all self-declare +primary, all three must start within the same ~200ms jitter window, which is +bounded by OS scheduler precision and systemd startup ordering. + +The `election_priority_step_ms` is added to `ha_config_t` alongside the existing +`election_timeout_ms` and `election_jitter_max_ms`. The jitter-only fallback +(priority-0 with no step) retains the original behaviour when `election_priority_step_ms = 0`. + +The window is bounded: in the worst case (all jitter values 0ms apart), all +candidates declare primary. Once both reach Phase 2, the HA watchdog resolves +the split via priority comparison in heartbeat exchange (see parent HA design). +This is bounded by the time for the slower candidate to complete Phase 2 +(~4–13s per §8.5) plus one heartbeat interval (~500ms). + +**Split-brain window during simultaneous boot**: The heartbeat listener starts +immediately after `am_i_primary()` returns, but it is a minimal pre-iomgr +listener that only handles `OP_heartbeat`. **All candidates that self-declare +primary launch a heartbeat listener**, not just priority-0 — this is critical: +if a lower-priority candidate wins the jitter race and self-declares primary, +its heartbeat listener provides the response that resolves the split-brain +when the higher-priority (true) primary probes it. + +During simultaneous boot, all candidates start with `ha.term = 0` +(`atomic_store_explicit(&ha->term, 0ULL, ...)` in `ha_init()`, ha.c:34). +Term-based priority comparison cannot break symmetry at this stage — symmetry +breaking during simultaneous boot relies entirely on the jitter mechanism +(part 2 below). Term-based step-down becomes effective only after the first +successful election, when the winning primary increments its term. The jitter +window is therefore the primary defence against multi-way split-brain during +cold start; the heartbeat listener provides the detection channel but term +comparison is not yet discriminating. + +If a heartbeat listener fails to bind its port (`EADDRINUSE`), the candidate +logs a WARN indicating a likely split-brain condition (another candidate has +already bound the port) and continues initialization. This is expected during +simultaneous boot with overlapping jitter windows — the candidate that bound +the port first keeps it; the other continues to Phase 2 where the HA watchdog's +heartbeat exchange will resolve the split-brain. + +If a lower-priority candidate also +self-declares primary and enters Phase 2, the HA watchdog heartbeat exchange +will resolve the split-brain once both candidates reach Phase 2 — but this +requires the HA watchdog to compare candidate priority (from `--peers` ordering) +in addition to term, since both nodes have term=0 during simultaneous boot. +**This design implements** `ha_check_heartbeat_nocluster()` (ha.c:101-126) +with a priority comparison at term=0 so that a lower-priority self-declared primary +steps down when it receives a heartbeat from a higher-priority node with the +same term. Without this enhancement, the current elapsed-time-only check cannot +distinguish a legitimate primary from a split-brain peer during cold start. + +**CRITICAL DEPENDENCY — implemented by this design**: The `ha_check_heartbeat_nocluster()` +enhancement (priority comparison at term=0) is a **hard prerequisite** for correct +simultaneous-boot behavior. Without it, a lower-priority self-declared primary will +not step down when it receives a heartbeat from a higher-priority peer with the same +term — the split-brain is unbounded, not 4–13s. This enhancement is implemented in +this design (`ha.c` — see §12 files changed) alongside the election probing. The +parent HA design (`persistence-and-ha-design.md` Part 5) specifies the heartbeat +exchange requirements; this design provides the priority-comparison implementation +which the parent design references. Test coverage: +`test_am_i_primary_simultaneous_boot` in §14.1 validates priority-based step-down +at term=0 during simultaneous boot. + +**Implementation scope**: This design adds a `sender_priority` field to the +`OP_heartbeat` request in `controller.capnp`. The heartbeat listener (§9.0) +populates this field from `c->ha.my_priority`. `ha_check_heartbeat_nocluster()` +(ha.c) compares `sender_priority` against `ha->my_priority` when `term == 0`: +if the sender has higher priority (lower index in `--peers`), the local node +steps down to standby. The data source is the heartbeat message, not `ha_state_t`, +so both the schema change and the handler change are part of this design. + +The effective split-brain window is therefore the time for the +secondary-primary to complete Phase 2 and start processing heartbeats: ~4–13s +per §8.5. During this window both self-declared primaries have `iomgr_init` and +`endpoint_init` complete (per the Phase 2 ordering in §7), so both are accepting +client connections before `ha_promote` resolves the split-brain. Conflicting +client operations (job submissions, queue modifications) submitted to both +primaries during this window may diverge. This falls within the parent design's +acknowledged `split_brain_risk_accepted` model, with the caveat that the +window is longer than a single heartbeat interval during cold start. + +**Data divergence during split-brain**: The losing primary's WAL is discarded when it +steps down to standby — any client operations it accepted during the split-brain +window are lost. This design does not add a reconciliation protocol; the scope is +bounded to the existing HA model. Operators who cannot tolerate any data loss during +cold start should not submit client operations until the primary's `INFO("Promoted +to primary")` log message confirms Phase 2 is complete. The typical cold-start +window occurs only during initial cluster boot or full-cluster restart — not during +normal single-node failover where an existing primary with a term > 0 prevents the +split-brain scenario from occurring. + +**PRNG**: The jitter calculation uses an xorshift64 PRNG seeded from +`/dev/urandom`. The PRNG state is stored in `ha_state_t.jitter_seed` (one per +node, initialised during Phase 1). The xorshift64 implementation should be a +shared inline function in `ha.h` (or a small `prng.h` utility header) so both +the election probe (§9.1) and the heartbeat listener (parent HA design) use the +same generator without code duplication. The seed is initialised from +`/dev/urandom` (read 8 bytes) during Phase 1. +If `/dev/urandom` is unavailable, fall back to +`clock_gettime(CLOCK_MONOTONIC, &ts)` nanoseconds XOR'd with +`((uint64_t)priority << 56) ^ ((uint64_t)getpid() << 16)` with a `WARN` log. +The shift mixing avoids the correlation problem where two systemd-started +processes in the same nanosecond produce jitter values differing only by +their priority index (a small integer). The `time(NULL) ^ getpid()` fallback +is rejected because it produces nearly identical values for processes started +simultaneously by systemd (same second, sequential PIDs) — exactly the scenario +where symmetry breaking matters most. + +### 9.1 Algorithm (design doc §5.4) + +```c +int am_i_primary(cluster_t *c) +{ + /* Priority 0: unconditionally primary — highest-priority node does not probe. + * Safe because all controllers receive the same --peers list, so ordering is + * deterministic and priority-0 is unique across the cluster. */ + if (c->ha.my_priority == 0) return 1; + + /* Launch one pthread per candidate with priority < my_priority. + * Each thread makes a non-blocking connect + poll loop to the candidate, + * sends OP_heartbeat. If ANY thread gets a live ACK before the shared + * election_timeout_ms deadline → return 0 (standby). + * If the deadline expires with no ACK → return 1 (primary). */ + + atomic_int election_done = 0; + probe_ctx_t probes[MAX_HA_CANDIDATES]; + int n_probes = 0; + + for (int i = 0; i < c->ha.my_priority; i++) { + probes[n_probes++] = (probe_ctx_t){ + .candidate = &c->myconf->peers_cfg.candidates[i], + .ssl_ctx = c->ha.ssl_ctx, /* stashed in Phase 1 */ + .timeout_ms = c->ha.config.probe_connect_timeout_ms, + .election_done = &election_done, + .got_ack = 0, /* set to 1 by probe_thread on success */ + }; + pthread_create(&probes[n_probes-1].thread, NULL, + probe_thread, &probes[n_probes-1]); + } + + /* Wait for first ACK or deadline (with jitter for simultaneous-boot, see §9.0) */ + long to_ms = c->ha.config.election_timeout_ms; + to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms; + to_ms += (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms); + struct timespec deadline; + clock_gettime(CLOCK_MONOTONIC, &deadline); + deadline.tv_sec += to_ms / 1000; + deadline.tv_nsec += (to_ms % 1000) * 1000000L; + if (deadline.tv_nsec >= 1000000000L) { + deadline.tv_sec++; + deadline.tv_nsec -= 1000000000L; + } + int got_ack = wait_any_probe(probes, n_probes, &deadline); + + /* Signal all threads to stop (no pthread_cancel — unsafe through OpenSSL) */ + atomic_store(&election_done, 1); + for (int i = 0; i < n_probes; i++) + pthread_join(probes[i].thread, NULL); + + return got_ack ? 0 : 1; +} +``` + +### 9.2 Thread coordination + +`pthread_cancel()` is **NOT** used (cancellation through OpenSSL shutdown handlers +can deadlock). Each probe thread uses **non-blocking `connect()` + `poll()`**: + +1. `socket()` + `fcntl(O_NONBLOCK)` + `connect()` → expect `EINPROGRESS` +2. `poll()` with timeout `probe_connect_timeout_ms` (default 500ms) +3. Between poll attempts, check `atomic_load(&election_done)` — if set, exit cleanly +4. If `poll()` returns -1 and `errno != EINTR`, treat as probe failure — log at TRACE + level and exit thread with `*got_ack = 0` +5. After connect, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` polling; + on hard SSL failures (certificate expired, protocol mismatch, peer disconnect), + treat as probe failure — log at TRACE level and exit thread +6. If `send()` or `recv()` fails on the SSL socket, treat as probe failure +7. Send `OP_heartbeat`, receive ACK, set `*got_ack` via shared pointer + +The per-attempt poll timeout bounds the worst-case exit delay to one poll interval +per thread (~500ms). TCP connect timeout (kernel SYN retry) is not reached because +the thread checks `election_done` between each poll iteration and exits early. +Probe failures are expected when peers are down — only TRACE-level logging is +appropriate; ERROR logs would generate noise during normal operation. + +### 9.3 Election outcome logging + +``` +/* Higher-priority candidate alive: */ +INFO("HA election: candidate %s:%u responded — starting as standby (my_priority=%d)", + winner_host, winner_port, my_priority) + +/* No higher-priority candidate alive: */ +INFO("HA election: no higher-priority candidate responded within %ums — " + "starting as primary (my_priority=%d)", + election_timeout_ms, my_priority) +``` + +--- + +## 10. Error Handling + +### 10.1 Primary with missing config + +**Condition**: `am_i_primary()` returns 1 (this node is primary) AND +`conf/current` does not exist or does not point to a readable versioned file. + +**Action**: `FATAL` log + return error from `load_primary_config()`, which propagates +to `myapp_run()` → `cluster_fini()` → exit. + +``` +FATAL("Primary candidate (priority 0) has no config file '%s' — " + "provision the config before starting the primary controller. " + "Run: xce-config --workdir %s --name %s --config cluster.yaml", + conffile, workdir, name) +``` + +### 10.2 Standby with no primary and no saved config + +**Condition**: `am_i_primary()` returns 0 (standby) AND all config fetch retries +fail AND `conf/current` does not exist. + +**Action**: `FATAL` log + exit. A standby cannot operate without any config — it +cannot initialise kvmgrs, load queues, or accept connections. + +``` +FATAL("Standby (priority %d) could not fetch config from primary and has no " + "saved config — start the primary controller first, or provision " + "the config file manually at '%s'", + my_priority, conffile) +``` + +### 10.3 Standby with no primary but saved config + +**Condition**: `am_i_primary()` returns 0 AND all config fetch retries fail AND +`conf/current` exists and is valid. + +**Action**: `WARN` log, proceed with saved config. This allows a standby to restart +during a primary outage without being permanently blocked. A `conf_stale_timeout` +(default: 24h, configurable via `--conf-stale-timeout` CLI flag) soft-limits +staleness: configs older than the threshold are flagged but not rejected (see below). + +``` +WARN("Standby (priority %d) could not reach primary for config fetch — " + "using saved config (tag=%016"PRIx64"). Config may be stale.", + my_priority, saved_tag) +``` + +A `conf_stale_timeout` field (default: 24h, configurable via `--conf-stale-timeout` +CLI flag) is stored in `myconf_t` alongside `conf_keep_n`. The 24h default is chosen to +match the typical maintenance window: if a primary is down for less than 24h, the +config on any standby that was connected before the outage is at most 24h old, which +is within the expected change cadence for HPC cluster configuration. Operators may +increase this for clusters with infrequent config changes, or set it to 0 to disable +staleness checking entirely. When a standby boots with +a saved config older than `conf_stale_timeout` seconds (determined by comparing the +versioned file's mtime against `CLOCK_REALTIME`), the controller logs a WARN and +proceeds, but sets a `config_stale` flag in `myconf_t`. The primary checks this flag +on the first `OP_replConfig` exchange after the standby reconnects: if the stale +config's tag does not match the primary's current tag, the primary pushes the new +config as normal. If a standby promotes while its config is stale (older than +`conf_stale_timeout`), the promotion proceeds — the stale timeout is a soft +indication, not a hard block, because the config may still be functionally valid. +Operators can set `conf_stale_timeout` to 0 to disable staleness checking. + +The `config_stale` flag is recomputed each time the controller boots (comparing +the versioned file's mtime to the current time). It is not persisted across +restarts. If a standby reconnects to the primary and successfully fetches a +new config matching the primary's current tag, the flag is cleared (the saved +config is now fresh). If the standby promotes while `config_stale` is set, it +proceeds to Phase 2 as primary with the acknowledgment that the config may be +stale — operators should monitor logs and trigger a config upgrade via +`OP_upgradeConfig` if necessary to synchronise the cluster. + +### 10.4 SHA-256 mismatch on receipt + +**Condition**: standby recomputes SHA-256 of received bytes ≠ `reply.sha256`. + +**Action**: discard the received bytes, log `ERROR`, retry the fetch (up to +`max_attempts`). If all retries fail with SHA-256 mismatches, treat as §10.2. + +``` +ERROR("Config fetch SHA-256 mismatch from primary — expected %s, got %s. " + "Retrying (%d/%d)", expected_hex, actual_hex, attempt, max_attempts) +``` + +### 10.5 Zero-tag in received config (redundant check) + +**Condition**: primary sends `reply.version_tag = 0`. + +**Action**: `FATAL` on the standby. With SHA-256 the probability of a legitimate +config producing `version_tag = 0` is ~2^-64 — negligible. A zero tag in the +reply indicates a primary-side bug (not a hash collision). The error message +distinguishes this: `"BUG: primary sent zero version_tag for config. Primary conffile " +"may be empty or corrupted: '%s'"`. + +### 10.6 Empty config file + +**Condition**: config file exists but is 0 bytes. + +**Action**: `FATAL`. `SHA256("")` produces a non-zero hash, but an empty file has no +valid Cap'n Proto content. The error message reads: `"Config file '%s' is empty — not a " +"valid ClusterConfig blob"`. + +### 10.6a Empty pointer file + +**Condition**: `conf/current` exists but is 0 bytes (zero-length pointer file). + +**Action**: `WARN` on primary, treat as missing config (escalate to §10.1). Standby +treats as trigger for full config fetch from primary (the saved config is unreachable +because the pointer file was not fully written — typically a crash between `open()` and +the first `write()`+`fsync()`+`close()` in §3.2 step 2). The error message reads: +`"Config pointer file 'conf/current' exists but is empty — treating as missing config. " +"This can occur if the controller crashed during a config write."` + +A zero-length `conf/current` is distinct from §10.6 (empty versioned file) — +`fgets()` returns `NULL` on an empty file without setting `errno`, and the +caller receives `-EIO` from `conf_read_current_filename()`. Adding a dedicated +diagnostic before the `fgets()` call (checking file size via `fstat()` or reading +one byte with `fread()`) makes the failure cause explicit rather than opaque. + +### 10.7 Config file size limit + +**Condition**: config file exceeds `CONF_MAX_SIZE` (default 1 MiB). + +**Action**: `FATAL`. A Cap'n Proto cluster config should be small (< 100 KB); a +larger file indicates corruption or a misconfiguration. The `conf_read_raw()` +function rejects files over this limit before allocating memory. + +### 10.8 Standby with corrupt saved config + +**Condition**: standby has `conf/current` pointing to a valid file, but its SHA-256 +differs from the primary's cached hash. + +**Action**: log `WARN`, discard saved config after receiving the correct one from +primary. This is normal — the saved config is stale, not necessarily bit-corrupt. +If the mismatch is detected during `fetch_config_from_primary()` (the standby's tag +≠ the primary's tag), the primary pushes the correct config and the standby overwrites: + +``` +WARN("Standby saved config tag=%016"PRIx64" differs from primary " + "tag=%016"PRIx64" — saved config may be stale, updating from primary", + saved_tag, primary_tag) +``` + +### 10.9 Trim crash safety + +**Condition**: controller crashes during `conf_trim_old_versions()`. + +**Action**: no recovery needed. Trimming is best-effort — if a crash leaves more +than `conf_keep_n` versioned files, the excess files are removed on the next +successful config update. Stale `.tmp` files from interrupted writes are also +removed during the next trim pass (which runs after every `conf_save_config()` call). +There is no boot-time cleanup of orphaned files; the trim function is the sole +cleanup mechanism. + +`conf_trim_old_versions()` always preserves the file named by the current `conf/current` +pointer: it reads the pointer file before building the deletion list and excludes that +filename regardless of mtime ordering. + +### 10.10 Duplicate OP_upgradeConfig during restart window + +**Condition**: a second `OP_upgradeConfig` arrives while the primary's `cluster_run()` +event loop is exiting (between `restart_requested = 1` and `cluster_fini()`). + +**Action**: iomgr is still accepting connections during this window, so the second +upgrade request is dispatched and may write a second config version, update +`conf/current`, and set `restart_requested = 1` (already set — a no-op). The write +procedure (§3.2) is crash-safe regardless. The first config write's versioned file +remains on disk for the trim pass to clean up later. WAL/data consistency is +preserved: both configs are valid `ClusterConfig` blobs (validated before save). +The process exits cleanly with whichever config `conf/current` names at the time +of the exec. No FATAL is needed — concurrent upgrades are benign. + +On restart via systemd, `conf/current` reflects the last saved config, which is +the one the primary will apply in Phase 2. + +### 10.11 Concurrent push and fetch on conf/current + +**Condition**: a standby receives an unsolicited push (§5.2 push direction) from the +primary while it is concurrently executing its own retry-fetch thread (§5.5). + +**Action**: both the push handler and the fetch thread call `conf_save_config()` which +ends with `rename(ptr_tmp, ptr_path)`. To avoid temp-file collisions, each call +uses a **thread-unique temp filename** based on the caller's pthread ID: + +```c +snprintf(ptr_tmp, ..., "%s/conf/current.%lu.tmp", effective_dir, (unsigned long)pthread_self()); +``` + +Each thread writes to its own `.tmp` file, fsyncs it, then renames it to `conf/current`. +POSIX `rename()` is atomic — both renames target the same destination (`conf/current`), +so whichever completes last wins, and the result is always a valid pointer file. Both +writes are for the same config version (the primary sends the same blob in both paths), +so either result is correct. Orphaned `.tmp` files from the losing thread are cleaned +by the next `conf_trim_old_versions()` pass. + +The same pattern applies to the versioned `.tmp` files in step 1 of the write procedure +(§3.2): each caller appends its thread ID to the temp filename to avoid truncating +another thread's in-progress write. + +> **Implementation note**: `conf_save_config()` must not open any temp file with +> `O_EXCL` (which would fail if another thread somehow created the same name). +> `O_CREAT|O_TRUNC` is correct for the thread-unique name. + +### 10.12 OP_upgradeConfig directed at a standby + +**Condition**: an admin tool sends `OP_upgradeConfig` to a standby controller's +endpoint. + +**Action**: the standby's `myreq_handler()` checks its HA role before dispatching: + +```c +case OPCode_upgradeConfig: + if (!ha_is_primary(&c->ha)) { + send_upgrade_error(ep, req, "Not primary — redirect to primary"); + /* Send primary address from ha_state if known */ + return -EAGAIN; + } + return handle_upgrade_config(c, ep, req); +``` + +The standby logs `WARN` and sends an error reply with the primary's address +(if known from the HA watchdog state). The admin tool is expected to retry against +the primary. + +--- + +## 11. Service Management and Deployment + +### 11.1 systemd unit configuration + +The controller is managed by a service manager (systemd or equivalent). The design +assumes the service manager handles process restart after `cluster_run()` exits +(whether from `OP_upgradeConfig` restart or crash recovery). + +**Certificate provisioning**: The SSL transport layer (§5.3) requires each controller +node to have a certificate signed by the shared CA before it can participate in config +replication or election probing. Certificate provisioning and lifecycle management +(renewal, revocation, CA rotation) are external to this design — they are assumed to +be handled by the same configuration management system that distributes `--peers` +(§11.2). At minimum, each node needs `certfile` and `keyfile` on disk before the +controller starts. A controller whose certificate has expired cannot complete the SSL +handshake and will be unable to fetch config or participate in elections — the +operational impact is equivalent to a network partition. Operators should monitor +certificate expiry and integrate renewal into their standard PKI workflows. + +**Certificate expiry warning**: At startup (Phase 1, after `create_ssl_context`), +the controller checks `X509_get0_notAfter()` on its own certificate. If the +certificate expires within `CERT_WARN_DAYS` (compile-time constant, default 30), +a WARN is logged: + +``` +WARN("Controller certificate expires in %d days (%s) — renew before expiry " + "to avoid HA election exclusion", days_remaining, asn1_time_str) +``` + +A controller whose certificate has expired cannot complete the SSL handshake +and will be unable to fetch config or participate in elections — the +operational impact is equivalent to a network partition. The startup warning +gives operators advance notice before the hard failure. + +A recommended systemd unit file: + +```ini +[Unit] +Description=XCE Controller +After=network-online.target +Wants=network-online.target +StartLimitIntervalSec=60 +StartLimitBurst=3 + +[Service] +Type=notify +NotifyAccess=all +ExecStart=/usr/local/bin/xcecontroller --peers=node0=host0:9990,node1=host1:9990,node2=host2:9990 +Restart=always +RestartSec=5 + +# Hardening +User=xce +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateDevices=yes +ReadWritePaths=/var/lib/xce +PrivateTmp=yes +ProtectKernelTunables=yes +ProtectControlGroups=yes +RestrictNamespaces=yes +SystemCallFilter=@system-service +CapabilityBoundingSet=CAP_NET_BIND_SERVICE +RestrictAddressFamilies=AF_INET AF_INET6 +ProtectHostname=yes +LockPersonality=yes + +[Install] +WantedBy=multi-user.target +``` + +Key points: +- `After=network-online.target` ensures network interfaces are up before the controller + starts, reducing the probability of connection failures during the initial config fetch. +- `RestartSec=5` with `StartLimitBurst=3` bounds restart storms. A longer + `RestartSec` (e.g., 15–30s) can be used to stagger controller restarts across + nodes in a simultaneous-boot scenario, reducing the probability that all candidates + enter election probing at the same time. +- `Type=notify` allows the controller to signal readiness to systemd after Phase 2 + completes, preventing the service manager from considering the process "started" + before it can serve requests. + +### 11.2 --peers consistency + +All controller candidates in a cluster **must** receive the same `--peers` argument. +The priority ordering (positional: candidate 0 is highest priority, candidate N-1 is +lowest) must be identical across all nodes. Inconsistent `--peers` arguments can cause +two nodes to self-declare primary (both with `my_priority == 0`) and never detect +each other. + +**Relationship to provisioning goal**: The §2 provisioning goal ("single provisioning +point: only the primary requires the config file") eliminates manual config distribution +but does not eliminate manual `--peers` distribution. Every candidate still needs the +`--peers` argument provisioned consistently — this is a prerequisite for correct HA +election behaviour. The `--peers` list is a bootstrap concern (it tells controllers +how to find each other) and cannot be bootstrapped from the config itself (circular +dependency: you need `--peers` to know who to fetch config from). Mitigations: +(1) the `--peers` list changes far less frequently than cluster config — it only +changes when nodes are added or removed from the HA cluster, which is rare compared +to queue/resource config changes that drive this design; (2) the future +peers-consistency cross-validation in `OP_heartbeat` (§11.2 below) will detect +mismatches at runtime; (3) the existing config management tooling already distributes +`--peers` for the HA foundation. + +**Recommended deployment**: distribute the `--peers` argument via a shared configuration +management system (Ansible, Puppet, Salt) or a template-generated unit file. This +ensures all nodes receive the same candidate list and ordering. The `--peers` argument +can alternatively be read from a local config file (`xce.conf` or equivalent) managed +by the same configuration system. + +**Validation**: `xce_parse_peers()` should log a WARN at startup if `my_priority` is +not in the range `[0, MAX_CANDIDATES-1]` or if any candidate entry has an empty +host or port zero — these indicate a malformed `--peers` string. + +**Future cross-validation**: A lightweight peers-consistency check in `OP_heartbeat` +(exchanging a hash of the local `--peers` string) is deferred to a follow-up release. +See `FUTURE_WORK.md`. + +--- + +## 12. Files Changed + +| File | Change | Complexity | +|---|---|---| +| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @20 :UInt64` to `ClusterConfig` (§4.4); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low | +| `src/controller/main.c` | Update `conf_file` resolution: read `conf/current` pointer file, then open named versioned file (replaces flat `conf/current_config` path); remove stack-allocated `char conf_file[]` and the `myconf.conffile = conf_file` assignment — `conffile` is now heap-allocated via `asprintf()` in `load_primary_config()` or `fetch_config_from_primary()` and freed in `myconf_fini()`; add `--conf-stale-timeout` and `--conf-keep-n` CLI flag parsing; add `--peers` validation (WARN on empty host or port 0) | Trivial | +| `src/include/myconf.h` | Add `config_version_tag` (uint64_t), `config_sha256[32]`, `conf_keep_n`, `config_generation` (uint64_t, §4.4), `conf_stale_timeout` (time_t, default 86400), and `config_stale` (bool) fields to `myconf_t` | Trivial | +| `src/include/xce/cluster_def.h` | Add `restart_requested` (atomic_int, for config-change restart), `heartbeat_listener_tid`, `heartbeat_listener_running`, and `lost_and_found_job_count` (atomic_int, incremented during `persist_recover()` when jobs are routed to the synthetic queue, §6.5) fields to `cluster_t` | Trivial | +| `src/controller/myconf.c` | Replace `get_default_conf()` with `load_primary_config()` (fail-fast + SHA-256 compute on primary path); config save/trim/validation functions moved to `config_repl.c`; add `myconf_fini()` to free `conffile` (now heap-allocated via `asprintf()` per §3.2), `cluster_config`, `config_version_tag`, and `config_sha256` | Low | +| `src/controller/cluster.c` | Split `cluster_init()` into phase 1 and phase 2; gate config load on role; move `iomgr_init` and `endpoint_init` to phase 2; add `create_lost_and_found()` call before `persist_recover`; add `OP_replConfig`, `OP_replConfigPush`, and `OP_upgradeConfig` cases to `myreq_handler()` opcode dispatch; `cluster_run()` event loop checks `c->restart_requested` and exits cleanly when set; `cluster_fini()` joins heartbeat listener thread on any exit path and frees `standby_endpoints` | Medium | +| `src/controller/myapp.c` | Restructure `myapp_run()`: call phase 1, then `am_i_primary()`, then config load/fetch, then phase 2. When `cluster_run()` returns cleanly (restart_requested=1), `myapp_run()` returns 0 to main() which exits; the systemd service manager restarts the process. Call `myconf_fini()` in the cleanup path before `cluster_fini()` | Medium | +| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0, §5.3) | Medium | +| `src/include/byteorder.h` | **New file**: `be64toh()` fallback macro for non-glibc or pre-2.25 targets (§6.4 portability note). The Meson build checks for `be64toh()` at configure time; if unavailable, this header provides a portable implementation via `__builtin_bswap64()` or `le64toh()`/`ntohll()` | Trivial | +| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources | Trivial | +| `src/controller/ha.c` | Replace `am_i_primary()` stub with real election probing; add `probe_thread()`, `wait_any_probe()`; add `start_heartbeat_listener()` / `stop_heartbeat_listener()`; add `xorshift64_next()` (or use shared `ha.h` inline); enhance `ha_check_heartbeat_nocluster()` with priority comparison for simultaneous-boot symmetry breaking (§9.0) | High | +| `src/controller/ha.h` | Add `ssl_ctx` field to `ha_state_t` (use `void*` to avoid pulling in `` — cast to `SSL_CTX*` at use sites in election probe and heartbeat listener); add `jitter_seed` (uint64_t, for xorshift64 PRNG) and `xorshift64_next()` inline function to `ha_state_t`; add `zarray_t *standby_endpoints` to `ha_state_t` for tracking connected standby controllers; add `probe_ctx_t` struct; add `election_jitter_max_ms` and `election_priority_step_ms` to `ha_config_t` with defaults `HA_DEFAULT_ELECTION_JITTER_MAX_MS 200` and `HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500` (§9.0 priority-based base delay); update "no new TLS" comment to note SSL transport is sufficient for trusted-LAN model; update `am_i_primary()` declaration; declare `ha_is_primary()` for runtime role check (§10.12) | Low | +| `src/controller/join.c` | Update `handle_controller_join()`: remove `-ENOSYS` (controller-controller sessions are valid — config exchange is dispatched from `myreq_handler()`); register connected standby endpoints in `c->ha.standby_endpoints` after successful join handshake; handle endpoint removal on disconnect | Low | +| `src/include/typemap.h` | Add `replConfigRequestMsg`, `replConfigReplyMsg`, `replConfigPushRequestMsg`, `replConfigPushReplyMsg`, `upgradeConfigRequestMsg`, `upgradeConfigReplyMsg` union arms to `xce_request_t.body` and `xce_reply_t.body` | Low | +| `src/lib/request.c` | Add encode/decode shims for `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply`; `Data` blobs managed by codecgen `encode_*_ptr()` functions | Medium | +| `src/include/persist.h` | Add `max_wal_records` field to `persist_config_t` (for WAL bound, see §13); add `wal_record_count` (atomic_uint) and `checkpoint_in_progress` (atomic_int) fields to `persist_t` for the count-based checkpoint trigger (§13.4) | Trivial | + +**New files**: `src/controller/config_repl.c`, `src/controller/config_repl.h`, and +`src/include/byteorder.h` — config save/load/trim, SHA-256 computation, upgrade handler, +standby push, and `be64toh()` portability fallback. The config replication logic lives in +its own translation unit rather than cluttering `myconf.c`. +Note: the existing `src/controller/repl.c` (WAL replication — standby slot management, +ACK tracking, catch-up) is a separate module with a similar name but different domain. +`repl.c` uses `replicator_t.standbys[]` for per-connection WAL ACK tracking; +`config_repl.c` uses `ha_state_t.standby_endpoints` for config push fan-out. +See `src/controller/repl.c` (340 lines) for the existing WAL replication module. + +**Total estimated LOC**: ~510 net new lines across all files (updated from ~480 to +account for compile-time gate, `sanitise_cn()`, `config_contains_reserved_name()`, +`byteorder.h`, and `CONF_PUSH_TIMEOUT_MS`). +Of these, ~170 are the `am_i_primary()` election probing and heartbeat listener +infrastructure. This infrastructure is **implemented by this design** (the current +stub at `ha.c:213` returns `c->ha.my_priority == 0` without probing). The parent +HA design (`persistence-and-ha-design.md` Part 5) specifies the requirements +(HA Requirements 3 and 4: election probing and heartbeat listener); this design +provides the implementation, which the parent design references. The config +replication contribution is ~310 LOC: ~250 for the pull protocol, storage layout, +and SHA-256 infrastructure; ~60 for the `OP_upgradeConfig` handler and +`push_config_to_standbys()` logic. + +--- + +## 13. WAL Record Bound + +### 13.1 Motivation + +`persist_recover()` runs in Phase 2 on the primary, before iomgr is ready to serve +`OP_replConfig`. The standby retry window is ~88 seconds (§5.5). Without a bound on +`persist_recover()` time, the standby may exhaust retries and FATAL even though the +primary is healthy and making progress. + +### 13.2 Mechanism + +Add `max_wal_records` to `persist_config_t` (default: 10,000). After every +`persist_append()` call in `persist.c`, if the active WAL segment exceeds +`max_wal_records`, trigger an asynchronous checkpoint: + +``` +persist_append(wal, entry): + append entry to WAL segment + if n_records(wal) > max_wal_records: + schedule_checkpoint() /* async — queues a vtask */ +``` + +The checkpoint flushes all in-memory state (jobs, queues, resources) to a fresh +snapshot file, truncates the WAL, and updates `snap_ctx` to reference the new +snapshot. Subsequent `persist_recover()` calls only replay records since the last +completed checkpoint. + +### 13.3 Time bound + +With `max_wal_records = 10000`, each record being a Cap'n Proto decode + in-memory +apply, `persist_recover()` replays at most 10,000 records. Benchmarked estimate: +~2-5 seconds on typical HPC controller hardware. The ~88-second retry window provides +>10× headroom. + +### 13.4 Checkpoint scheduling + +Checkpoints are scheduled via the `delaymgr` (available in Phase 2 after +`delaymgr_init`) rather than `vtaskmgr` directly. Using `delaymgr` avoids +saturating the vtaskmgr thread pool with repeated checkpoint requests during +burst-append scenarios — `delaymgr` coalesces multiple requests into a single +delayed dispatch. + +**`schedule_checkpoint()` specification**: + +```c +/* schedule_checkpoint — request an async WAL checkpoint via delaymgr. + * + * Idempotent: if a checkpoint is already scheduled or in progress, + * this is a no-op (guarded by checkpoint_in_progress atomic). + * + * @param persist The persist_t instance (has access to cluster_t via + * persist->cluster for delaymgr and vtaskmgr handles). + * + * Thread-safety: safe to call from any thread. Uses atomic CAS on + * persist->checkpoint_in_progress to ensure only one checkpoint + * is queued at a time. + */ +static void schedule_checkpoint(persist_t *persist) +{ + /* CAS: 0 → 1. If already 1 (in-progress or queued), no-op. */ + int expected = 0; + if (!atomic_compare_exchange_strong(&persist->checkpoint_in_progress, + &expected, 1)) { + return; /* already scheduled or running */ + } + + /* Schedule via delaymgr with a short delay to coalesce + * burst-append requests into a single checkpoint dispatch. + * delay_task() uses time_t (seconds) granularity — see src/lib/delay.c:187. The actual API is delay_task(delaymgr_t*, int period, int runcount, + * delay_task_entry_fn_t, void*). */ + delay_task(&persist->cluster->delaymgr, + 1 /* 1s — coalesce burst appends; delay_task() takes seconds */, + 1 /* runcount — fire once */, + checkpoint_task_fn, + persist); + atomic_store(&persist->wal_record_count, 0); /* reset counter after checkpoint */ +} + +/* checkpoint_task_fn — executed by vtaskmgr worker, dispatched by delaymgr. + * Returns int per delay_task_entry_fn_t signature. */ +static int checkpoint_task_fn(void *arg) +{ + persist_t *persist = (persist_t *)arg; + persist_checkpoint(persist); /* existing synchronous checkpoint */ + atomic_store(&persist->checkpoint_in_progress, 0); + return 0; +} +``` + +The WAL record-count trigger in `persist_wal_sync_append()` and +`persist_wal_async()` (the actual WAL write functions in `persist.h:109-111`) +calls `schedule_checkpoint()` after appending each record when the record count +exceeds `max_wal_records`: + +```c +/* After appending WAL record: */ +if (atomic_fetch_add(&persist->wal_record_count, 1) >= persist->config.max_wal_records) { + schedule_checkpoint(persist); +} +``` + +`persist->wal_record_count` is an `atomic_uint` in `persist_t`, reset to 0 +after each checkpoint completes. The primary continues serving requests while +the checkpoint progresses. Multiple `persist_append()` calls may queue redundant +checkpoints — `schedule_checkpoint()` is idempotent via the atomic CAS guard. + +Records appended to the WAL during an in-progress checkpoint may push +`wal_record_count` above `max_wal_records` again before the checkpoint +completes. This is intentional — `wal_record_count` is a soft, lossy +signal whose sole purpose is to trigger checkpoint scheduling. It is +reset to 0 after `persist_checkpoint()` returns, so any records appended +during the checkpoint will be counted in the next trigger window. This +does not affect correctness: the checkpoint always flushes all in-memory +state regardless of the counter value. + +### 13.5 Configuration + +`max_wal_records` is set in `persist_config_t` alongside existing WAL parameters +(`wal_max_size`, `snap_keep_n`). Administrators can tune it per workload profile. +Larger values mean longer `persist_recover()` but fewer checkpoints (less I/O). +Smaller values bound recovery time more tightly at the cost of more frequent +snapshot writes. The minimum allowed value is 100 (checked at startup in `persist_init()`); +values below this are rejected with a FATAL log to prevent checkpoint storms. + +`max_wal_records` is a soft trigger for checkpoint scheduling, not a hard +limit on record count. After `persist_append()` appends the new entry, if the +WAL segment now contains more than `max_wal_records` entries, +`schedule_checkpoint()` is called; subsequent appends proceed immediately +without waiting. The WAL may temporarily exceed `max_wal_records` during a +burst of appends if the asynchronous checkpoint is slow to complete. + +The parent design (`persistence-and-ha-design.md §2.3`) is the authoritative +source for WAL growth controls. This section adds only the record-count-based +trigger needed to bound `persist_recover()` time for the standby retry window +(§5.5). The parent design provides: `wal_max_size_mb` hard limit (2 GB default) +as an emergency trigger, `checkpoint_in_progress` atomic guard, and WAL growth +warning at `2 * snap_wal_size_mb` during in-progress checkpoint. These +mechanisms are complementary — the size-based limit catches large-record workloads +while the count-based limit bounds replay time regardless of record size. + +--- + +## 14. Test Plan + +### 14.1 Unit tests (no live cluster required) + +**Phase 1 tests:** + +| Test | File | Covers | +|---|---|---| +| `test_conf_sha256_change_detection` | `test_config_repl.c` | Tag match → no push; tag differ → push | +| `test_conf_sha256_full_integrity` | `test_config_repl.c` | Full SHA-256 mismatch → integrity failure → retry | +| `test_conf_save_pointer_atomic` | `test_config_repl.c` | Write versioned file + fsync + HEAD-file swap; verify `cat conf/current` names new file after update | +| `test_conf_save_crash_after_fsync` | `test_config_repl.c` | Crash after step 1 (versioned file on disk, pointer file not updated) → next boot reads old pointer file | +| `test_conf_read_current_filename` | `test_config_repl.c` | `conf_read_current_filename()` parses filename, strips newline; returns -ENOENT when file absent | +| `test_conf_zero_tag_sentinel` | `test_config_repl.c` | Standby with no saved config sends tag=0 → primary correctly pushes config (no FATAL for zero-tag sentinel; FATAL only when primary sends zero in reply per §10.5) | +| `test_conf_trim_keep_n` | `test_config_repl.c` | After 3 updates with `keep_n=2`, only 2 versioned files remain; stale `.tmp` cleaned | +| `test_am_i_primary_no_peers` | `test_ha_role.c` (extend) | Priority-0 with no peers → primary without probing | +| `test_am_i_primary_higher_alive` | `test_ha_role.c` (extend) | Mock probe ACK from priority-0 → standby | +| `test_am_i_primary_deadline_expires` | `test_ha_role.c` (extend) | All probes time out → primary | +| `test_am_i_primary_simultaneous_boot` | `test_ha_role.c` (extend) | Three priorities start simultaneously → transient split-brain resolves; priority-0 is primary after Phase 2 watchdog exchange (§9.0) | +| `test_primary_missing_config_fatal` | `test_config_repl.c` | Priority-0 + no conffile → `load_primary_config()` returns -ENOENT | +| `test_standby_no_config_no_primary_fatal` | `test_config_repl.c` | Priority-1 + fetch fails + no saved config → fatal error path | +| `test_standby_saved_config_no_primary` | `test_config_repl.c` | Priority-1 + fetch fails + saved config exists → WARN + proceed | +| `test_conf_stale_timeout` | `test_config_repl.c` | Saved config file mtime older than `conf_stale_timeout` → WARN logged, `config_stale` flag set; saved config within timeout → no WARN, flag clear | +| `test_cert_expiry_warning` | `test_ha_role.c` (extend) | Certificate expiring within `CERT_WARN_DAYS` → WARN logged; certificate far in future → no WARN (§11.1) | + +**Phase 2 tests** (require `cluster_init()` Phase 2 to be complete — iomgr, kvmgr, persist): + +| Test | File | Covers | +|---|---|---| +| `test_upgrade_config_sha256_mismatch` | `test_config_repl.c` | `OP_upgradeConfig` with wrong caller SHA-256 → rejected before disk write | +| `test_upgrade_config_invalid_capnp` | `test_config_repl.c` | `OP_upgradeConfig` with corrupt blob → Cap'n Proto decode fails → rejected | +| `test_upgrade_config_generation_incremented` | `test_config_repl.c` | `OP_upgradeConfig` with any generation value → primary ignores incoming generation, sets `config_generation = current + 1` (§4.4, §6.4 step 3) | +| `test_upgrade_config_replay_harmless` | `test_config_repl.c` | Duplicate `OP_upgradeConfig` (same bytes) → accepted again, generation incremented, restart triggered (§4.4) | +| `test_upgrade_config_wal_unknown_id` | `test_config_repl.c` | WAL replay with unknown queue ID → job routed to `lost_and_found` queue; unknown resource ID → job resource cleared, set to PEND; unknown host → routed to `lost_and_found` host | +| `test_lost_and_found_queue` | `test_config_repl.c` | `persist_recover()` with job referencing removed queue ID → job placed in `lost_and_found` queue; unknown resource ID → job resource cleared, state set to PEND | +| `test_lost_and_found_counter` | `test_config_repl.c` | `persist_recover()` routes jobs to `lost_and_found` → `lost_and_found_job_count` incremented; startup WARN summary logged if counter > 0 (§6.5) | + +### 14.2 Integration test (requires two controller processes) + +| Test | Covers | +|---|---| +| `test_standby_receives_config_on_first_boot` | Fresh standby with no conf/ → receives config → `conf/current` pointer file created, SHA-256 verified | +| `test_standby_skips_fetch_on_tag_match` | Standby reconnects with matching tag → `up_to_date=true` → no file I/O | +| `test_standby_updates_config_on_tag_mismatch` | Standby reconnects with stale tag → receives new config → pointer file updated → previous version retained | +| `test_primary_missing_config_aborts` | Start primary with no conf/ → process exits non-zero | +| `test_primary_recomputes_hash_before_send` | Primary's handler double-checks SHA-256 before sending; mismatch detected | +| `test_upgrade_config_pushes_to_standby` | Admin sends `OP_upgradeConfig` → primary ACKs → connected standby receives push → both have new tag before primary restarts | +| `test_upgrade_config_offline_standby_catches_up` | Admin upgrades config while standby is offline → standby restarts → detects tag mismatch on reconnect → pulls new config | +| `test_upgrade_config_no_standbys` | Admin upgrades config on single-node cluster (no standbys) → save succeeds → restart applies new config | + +### 14.3 Regression + +All existing HA tests (`test_ha_cluster.c`, `test_ha_role.c`, `test_ha_redirect.c`, +`test_p1_integration.c`) must continue to pass unchanged. These tests manually +zero-init `cluster_t` and call `ha_init()` directly, bypassing `cluster_init()` — the +phase split does not affect them. + +### 14.4 Future Work + +Tracked in `FUTURE_WORK.md`. Summary: + +| Test | Covers | +|---|---| +| `test_peers_cross_validation` | `OP_heartbeat` includes `--peers` hash → mismatch between two candidates → ERROR logged (§11.2 future work) | +| `test_lost_and_found_drain` | `xcecli lost-and-found drain` command → reassign or cancel sidelined jobs → `lost_and_found_job_count` reset to 0 (§6.5 observability) | +| `test_upgradeconfig_redirect` | `OP_upgradeConfig` sent to standby → error reply with primary address → admin tool retries (§10.12) |