# XCE Controller: Configuration Replication Design > **Status**: Implemented (Phase 1 active; Phase 2 gated) > **Last Updated**: 2026-05-13 > **Target**: `src/controller/` (C daemon) > **Depends on**: `persistence-and-ha-design.md` Part 5 (HA Failover) > **Branch**: `persistent_ha` > **Implementation commits**: `683c092`–`40c7a93` (20 commits) --- ## Table of Contents 1. [Problem Statement](#1-problem-statement) 2. [Goals and Non-Goals](#2-goals-and-non-goals) 3. [Config Storage Layout](#3-config-storage-layout) 4. [SHA-256 as Version and Integrity Token](#4-sha-256-as-version-and-integrity-token) 5. [OP_replConfig Protocol](#5-op_replconfig-protocol) 6. [OP_upgradeConfig Protocol](#6-op_upgradeconfig-protocol) 7. [Startup Sequence](#7-startup-sequence) 8. [cluster_init() Phase Split](#8-cluster_init-phase-split) 9. [am_i_primary() Implementation](#9-am_i_primary-implementation) 10. [Error Handling](#10-error-handling) 11. [Service Management and Deployment](#11-service-management-and-deployment) 12. [Files Changed](#12-files-changed) 13. [WAL Record Bound](#13-wal-record-bound) 14. [Test Plan](#14-test-plan) --- ## 1. Problem Statement ### 1.1 Current behaviour Every controller candidate reads its cluster configuration (`ClusterConfig` Cap'n Proto blob) from a local file at `${workdir}/${name}/conf/current_config` on startup. This means: - Every candidate node must be independently provisioned with a copy of the config file. - Any config change (new queue, resource definition update) must be applied to every candidate separately. There is no enforcement mechanism — a missed copy is silent. - A standby running a stale config promotes with wrong queue/resource definitions. The mismatch is only discovered when the promoted standby starts serving clients. ### 1.2 Proposed behaviour The **primary** (highest-priority candidate) is the sole config authority: - The primary reads `ClusterConfig` from its local `conf/current/` on startup. If the file is missing, it is a fatal error — the cluster cannot start. - A **standby** that has no local config fetches it from the primary over the existing SSL/iomgr pipeline before completing initialisation. The received config is saved locally so the standby can restart without the primary being available. - On every reconnect, the standby sends its current config's SHA-256 tag (first 8 bytes). The primary skips the push if the tag matches, avoiding unnecessary reinitialisation. --- ## 2. Goals and Non-Goals ### Goals **Provisioning** - Single provisioning point: only the primary candidate node requires the config file. - Detect and minimize config drift: standbys fetch from primary on every reconnect. When the primary is unreachable, a saved config is used with staleness warning (§10.3). Config drift is eliminated for the normal case (reconnect); the edge case where a standby promotes while the primary is unreachable is mitigated by the staleness warning and `config_stale` flag but cannot be fully prevented without a quorum gate (deferred to §14.4). **Residual risk accepted**: if both the primary and all standbys with current config are simultaneously down, a standby with only a stale saved config will promote and serve with potentially wrong queue/resource definitions — the same failure mode described in §1.1. The `config_stale` flag and log warnings make this discoverable (the current behaviour is silent), but do not prevent it. Operators who require hard prevention should implement the quorum gate (§14.4). **Integrity** - Atomic, crash-safe config update on the standby. - SHA-256 as the token for both change detection and cryptographic integrity validation. **Observability** - Human-visible config version in the filesystem (SHA-256-named files + pointer file). **Remote config push** - `OP_upgradeConfig` allows an admin tool to push a new config blob to the primary via the controller endpoint. The primary saves it, pushes it to connected standbys (best-effort, with ACK count in the reply), then restarts itself to apply the new config (~4–13s downtime; see §6.6 for timing decomposition). The admin tool that drives `OP_upgradeConfig` is out of scope for this design but is a prerequisite for Phase 2 — it must encode `ClusterConfig` to Cap'n Proto binary, compute SHA-256, and send the request via the controller's internal endpoint (see `FUTURE_WORK.md`). ### Non-Goals **Scope** - Hot-reload of config without restart. Config drives kvmgr slot counts and resource definitions that are baked in at Phase 2 init time; a restart is required to apply them. - YAML config parsing in the controller. The controller reads only the Cap'n Proto binary blob. A separate admin tool (out of scope) produces the blob. **Responsibility boundaries** - Semantic validation of `ClusterConfig` content (e.g., non-zero queue count, valid port ranges, host list completeness). The controller validates structural correctness only (valid Cap'n Proto decode, size within `CONF_MAX_SIZE`). Semantic correctness is the responsibility of the admin tool that produces the blob. **Compatibility** - Backward compatibility. This is the first version of the protocol. ### Alternatives Considered **Config-management-based distribution**: The simplest alternative is to distribute the `ClusterConfig` blob via the same config management system (Ansible, Puppet, Salt) already recommended for `--peers` distribution (§11.2). This avoids all new controller code. | Approach | Provisioning | Drift detection | Bootstrap without pre-seeding | Restart-free change | LOC | |---|---|---|---|---|---| | Config management only | Multi-node push | Manual (diff files) | No | No | 0 | | Config-in-join-reply | Primary-only | Overwrite on every join | Yes (join reply) | No (restart required) | ~50 | | This design | Primary-only | Automatic (SHA-256 tag) | Yes (pull from primary) | No (restart required) | ~310 | Config management alone works for clusters where all controller nodes are managed by the same automation system. This design addresses the gap where that assumption breaks down: a newly provisioned standby with no pre-existing config file, or a standby rejoining after extended offline where its saved config may be stale. The HEAD-file + SHA-256 approach also provides cryptographic proof that two controllers run identical configs without parsing binary blobs — useful for operator verification. **Config-in-join-reply (minimal approach)**: A ~50 LOC alternative: include the raw config bytes in the controller join reply. When a standby completes `OP_join`, the primary attaches the current `ClusterConfig` blob. The standby saves it to `conf/current_config` (overwriting any stale copy). This handles the bootstrap case and the reconnect case (join reply always carries current config). It does not provide: tag-based push skip (every join sends the full blob), cryptographic integrity validation, version history/retention, or `OP_upgradeConfig` live push. For clusters with infrequent joins and small configs (<100 KB), this is sufficient. The full design is chosen for clusters where config integrity verification, change auditability, and the push mechanism for online config updates are operational requirements. **WAL-based config replication (considered and rejected)**: XCE already has a WAL replication path (`src/controller/repl.c`, ~340 lines) that streams job/queue/resource mutations from primary to standbys. Extending it to carry serialised `ClusterConfig` blobs would reuse the existing replication infrastructure. This approach is rejected for three reasons: (1) **Semantic mismatch** — WAL replication is continuous and ordered; config changes are one-shot, admin-initiated events that should not be interleaved with job mutation records (a WAL ordering dependency between config records and operation records creates unnecessary coupling). (2) **Atomicity** — config changes require saving the full blob, computing SHA-256, updating the HEAD-file pointer, and trimming old versions atomically; this is a self-contained operation with its own integrity guarantees distinct from WAL append semantics. (3) **Bootstrap ordering** — WAL replication requires `persist_init` and a replica connection to be established (Phase 2), but the standby needs config before Phase 2 to initialise iomgr and kvmgr (§7). A separate config fetch protocol (§5) avoids this circular dependency. **Why this design**: The chosen approach layers on the full protocol not because the bootstrap case demands it, but because the same SHA-256 infrastructure underpins: (1) tag-based skip on reconnect (avoiding unnecessary deserialisation), (2) cryptographic integrity validation of received blobs, (3) human-visible version identity via `conf/current` pointer, and (4) the `OP_upgradeConfig` push mechanism for remote admin-driven config changes. Each layer individually could be deferred; together they form a coherent config management lifecycle that the simpler approaches do not provide. ### Phased Delivery The design is organised into two phases, but **both phases ship together in a single PR**. The "phase" distinction refers to activation readiness, not separate deliveries: **Phase 1 — Pull-only config replication** (active on merge; delivers Provisioning, Integrity, Observability goals): - `OP_replConfig` pull protocol (§5) with SHA-256 tag comparison - HEAD-file + versioned files storage layout (§3) - Standby fetch with retry (§5.5) - `am_i_primary()` election probing (§9, shared with parent HA design) - ~310 LOC **Phase 2 — Remote config push** (infrastructure ships now, feature activates when admin tool and `validate_host()` are complete; see `FUTURE_WORK.md`): - `OP_upgradeConfig` handler (§6.4), `OP_replConfigPush` fan-out (§6.3) - `lost_and_found` queue/host routing (§6.5) - `standby_endpoints` tracking, `restart_requested` mechanism - WAL record bound (§13) - ~170 LOC The `OP_upgradeConfig` handler is gated behind a compile-time flag `CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled — see §5.3 safety gate). The gate MUST NOT be set to 1 in any production build until BOTH prerequisites are met: 1. **`validate_host()`** is implemented with a real authorisation check (not a stub that returns success for all peers). Minimum viable check: verify the peer's certificate CN against an allowlist of authorised admin hosts. See `FUTURE_WORK.md` §6 for the implementation plan. 2. **Admin config tool** exists and has been tested end-to-end. It must encode `ClusterConfig` to Cap'n Proto binary, compute SHA-256, connect via SSL, send `OP_upgradeConfig`, display the reply (version tag, acked count, error), and poll `OP_probe` after restart. See `FUTURE_WORK.md` §1 for the tool specification. **Prerequisite delivery commitment**: Both prerequisites are tracked in `FUTURE_WORK.md` and MUST be completed before the next production release that sets `CONF_ENABLE_UPGRADE_CONFIG=1`. A release may ship with the gate at 0 (Phase 1 active, Phase 2 infrastructure compiled but inert). The gate is removed from the codebase when both prerequisites are merged and integration-tested. Phase 1 alone eliminates manual provisioning and config drift — the core problems stated in §1. --- ## 3. Config Storage Layout ### 3.1 Directory structure The parent HA design (`persistence-and-ha-design.md` Part 5) specifies a config storage layout under `conf/`. This design replaces that with the HEAD-file + versioned files approach described below. This design replaces that with a **HEAD-file + versioned files** layout that is atomically crash-safe. The HEAD-file pattern avoids symlink(2), which is not atomic on all filesystems and would require a separate atomic-rename technique anyway. **Local filesystem only — NFS is explicitly unsupported.** The controller's working directory (including `conf/`, `data/wal/`, and `data/snap/`) MUST reside on a local filesystem. `persist_init()` (persist.c:217-222) detects and rejects NFS mounts at startup with a FATAL error. Config replication assumes local `rename(2)` atomicity and `fsync(2)` durability guarantees; neither holds on NFS. No effort is made to support network or distributed filesystems for config storage. Cross-filesystem portability is a non-goal. ``` ${workdir}/${name}/ └── conf/ ├── current ← plain text pointer: "cluster.bin.a1b2c3d4e5f6\n" ├── cluster.bin.0f1e2d3c4a5b ← previous version (retained) └── cluster.bin.a1b2c3d4e5f6 ← current version ``` The filename suffix is the **first 12 hex chars of the file's SHA-256 hash**. `cat conf/current` gives the active version filename at a glance — identical to Git's `HEAD` file pattern. Two standbys with identical `conf/current` contents are provably running identical configs without parsing any binary content. > **Note**: The current code uses a flat path `conf/current_config` > (`main.c:115`). This is replaced by a two-step resolution: read `conf/current` to > get the versioned filename, then open that file. The `conffile` field in `myconf_t` > is a **heap-allocated string** set via `asprintf()` during Phase 1 by > `load_primary_config()` (primary path) or `fetch_config_from_primary()` (standby > path) after reading the `conf/current` pointer file and resolving it to the > versioned file path (e.g., `${workdir}/${name}/conf/cluster.bin.a1b2c3d4e5f6`). > It is freed in `myconf_fini()` alongside the other owned fields. The two-step > path resolution is load-bearing for the atomic `rename(2)` semantics described > in §3.2 — the pointer file swap mutates which versioned file is current, and > all code that needs config bytes resolves through `conf/current` → `conffile` > to get the correct path. ### 3.2 Write procedure (standby receiving new config) **`effective_dir`** throughout this section denotes `${workdir}/${name}` — the controller instance's base directory. All conf/ paths are relative to it. ```c // 0. Ensure conf/ directory exists with restrictive permissions snprintf(conf_dir, ..., "%s/conf", effective_dir); mkdir(conf_dir, 0700); /* same as data/wal/ and data/snap/ — trusted LAN model. * On EEXIST, verify existing permissions with fstat() and * chmod() to 0700 if needed — mkdir() silently ignores * pre-existing directories with wrong permissions. */ // 1. Write new versioned file atomically // Thread-unique .tmp suffix avoids truncation under concurrent push+fetch (§10.11) snprintf(tmp_path, ..., "%s/conf/cluster.bin.%s.%lu.tmp", effective_dir, hex_tag, (unsigned long)pthread_self()); snprintf(ver_path, ..., "%s/conf/cluster.bin.%s", effective_dir, hex_tag); fd = open(tmp_path, O_WRONLY|O_CREAT|O_TRUNC, 0600); if (fd < 0) { ERROR("open(%%s) failed: %%s", tmp_path, strerror(errno)); return -errno; } if (write(fd, bytes, len) != (ssize_t)len) { ERROR("short write on %%s: %%s", tmp_path, strerror(errno)); close(fd); unlink(tmp_path); return -EIO; } if (fsync(fd) != 0) { ERROR("fsync(%%s) failed: %%s", tmp_path, strerror(errno)); close(fd); unlink(tmp_path); return -EIO; } close(fd); rename(tmp_path, ver_path); /* atomic: versioned file now on disk */ // 2. Atomically update the HEAD-style pointer file // Thread-unique .tmp suffix avoids race under concurrent push+fetch (§10.11) snprintf(ptr_tmp, ..., "%s/conf/current.%lu.tmp", effective_dir, (unsigned long)pthread_self()); snprintf(ptr_path, ..., "%s/conf/current", effective_dir); char target[NAME_MAX]; snprintf(target, sizeof(target), "cluster.bin.%s\n", hex_tag); fd = open(ptr_tmp, O_WRONLY|O_CREAT|O_TRUNC, 0600); write(fd, target, strlen(target)); fsync(fd); close(fd); rename(ptr_tmp, ptr_path); /* atomic pointer swap — no symlink(2) required */ // 2.5. Durability: fsync the conf/ directory to persist the rename(2) fd = open(conf_dir, O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // 3. Trim old versioned files beyond keep_n (default: 2) conf_trim_old_versions(effective_dir, keep_n); ``` **Read procedure** — called at startup to resolve `conffile`: ```c static int conf_read_current_filename(const char *conf_dir, char *out, size_t outsz) { char head_path[PATH_MAX]; snprintf(head_path, sizeof(head_path), "%s/current", conf_dir); FILE *f = fopen(head_path, "r"); if (!f) return -ENOENT; /* Distinguish empty file from EOF — fgets returns NULL for both. * An empty pointer file means the controller crashed between open() * and the first write()+fsync()+close() in §3.2 step 2. */ struct stat st; if (fstat(fileno(f), &st) == 0 && st.st_size == 0) { fclose(f); ERROR("conf/current is empty — treating as missing config (§10.6a)"); return -ENODATA; } if (!fgets(out, outsz, f)) { fclose(f); return -EIO; } fclose(f); out[strcspn(out, "\n")] = '\0'; /* strip trailing newline */ /* Validate: reject path separators, require "cluster.bin." prefix, * require exactly 12 hex chars in the suffix. */ if (strchr(out, '/')) { ERROR("conf/current contains path separator — corrupted pointer file"); return -EINVAL; } if (strncmp(out, "cluster.bin.", 12) != 0) { ERROR("conf/current filename '%s' does not start with 'cluster.bin.'", out); return -EINVAL; } const char *hex = out + 12; if (strlen(hex) != 12 || strspn(hex, "0123456789abcdef") != 12) { ERROR("conf/current suffix '%s' is not 12 hex chars", hex); return -EINVAL; } return 0; } ``` After calling this, `myconf->conffile` is set via `asprintf()` to `"${conf_dir}/${out}"` — a heap-allocated copy that survives past the caller's stack frame. It is freed in `myconf_fini()` alongside the other owned fields. **Crash safety**: if the process crashes after step 1 but before step 2, `conf/current` still contains the old versioned filename — the standby boots on the previous config and re-fetches on the next connection. If it crashes mid-step-1 (before the rename of the `.tmp` file), the orphaned `.tmp` file is ignored on next boot. ### 3.3 Retention Keep the last `conf_keep_n` versioned files on disk (default: 2 — current + one previous). Trimming deletes the oldest by mtime after every successful update. Stale `.tmp` files from interrupted writes are also cleaned during the trim pass. `conf_keep_n` is a field in `myconf_t` — it is needed during config save (between Phase 1 and Phase 2), before `persist_init()` runs and `persist_config_t` is available. It is initialised from a hardcoded default (2) or the `--conf-keep-n` CLI flag during Phase 1, before any config is loaded. It is not read from the cluster config, so it is available before config fetch on standbys. --- ## 4. SHA-256 as Version and Integrity Token ### 4.1 Why SHA-256 `configSeq` and CRC32 were both considered and rejected: | Property | configSeq | CRC32 | SHA-256 | |---|---|---|---| | Change detection | Compare ints | Compare CRC32 | Compare first 8 bytes | | Integrity validation | Separate field | Same field (CRC) | Full 32-byte hash | | Admin tool required | Yes | No | No | | Push skip on match | No (always incr) | Yes (same content → same CRC) | Yes (same content → same hash) | | Collision resistance | N/A | 2^-32 (vulnerable) | 2^-256 (none in practice) | | Cryptographic strength | None | None (invertible) | Full (keyless integrity) | | On-disk dependency | — | zlib `crc32()` | OpenSSL (already in Phase 1) | **Why cryptographic hash in a trusted-LAN model?** The trusted-LAN model (§5.3) means we trust the network path between controllers, but config blobs traverse additional boundaries where trust does not apply: admin workstations push configs via `OP_upgradeConfig` (§6), config files may be hand-carried on USB drives for air-gapped provisioning, and versioned files persist on disk across reboots where bit rot or operator error (copying the wrong file) are risks. SHA-256 provides end-to-end integrity across all of these paths without adding a separate checksum mechanism — the same hash that detects version drift also validates integrity. The implementation cost is negligible: OpenSSL is already initialised in Phase 1, and SHA-256 of a <100 KB blob takes microseconds. SHA-256 is computed over the **raw bytes of the `cluster.bin.*` file** — not over any in-memory struct: - The hash is **never stored inside** the Cap'n Proto payload (avoiding circularity). - The **first 8 bytes** (UInt64, big-endian as raw SHA-256 bytes; Cap'n Proto wire format is little-endian — be64toh() converts from big-endian SHA-256 bytes to host order) serve as the fast "version tag" for change detection and the versioned filename suffix (first 12 hex chars). - The **full 32 bytes** are sent in the protocol for the standby to verify cryptographic integrity of the received blob. - Since config is always written to a **new versioned file** (never overwritten in place, see §3.2), the file content is **immutable** after creation. The primary's cached `config_sha256` is always consistent with the bytes on disk. - The primary recomputes SHA-256 over the bytes read at request time and compares to the cached hash before sending — any discrepancy is logged as an error rather than silently sending mismatched data. ### 4.2 Zero-tag sentinel A **zero `version_tag` (UInt64)** is the "no config" sentinel. A standby with no saved config sends `version_tag = 0`. The probability of a legitimate config producing a zero first 8 bytes of SHA-256 is ~2^-64 — negligible. No special-case FATAL is required. ### 4.3 SHA-256 API Available via `SHA256()` from OpenSSL (``), which is already initialised in Phase 1 (`initialize_openssl()`). No new dependency. The hash is computed once at startup and cached; the per-request recomputation is a safety double-check, not the primary path. ### 4.4 Monotonic generation counter for auditing SHA-256 alone prevents accidental mismatches but does not prevent **downgrade** — an admin could push an older valid config blob (different SHA-256) and the primary would accept it as an "upgrade." `ClusterConfig` carries a monotonic `configGeneration` field (`UInt64`, default 1 on first save) as an **audit indicator**, not an enforcement mechanism: - On each successful `OP_upgradeConfig`, the primary sets `config_generation = c->myconf->config_generation + 1` in the decoded struct, re-encodes the blob, and saves — so the generation strictly increases. - The admin tool does not need to manage the generation counter — the primary is the sole authority and always increments. The `configGeneration` field in the incoming blob is ignored; the primary overwrites it with `current + 1` before saving (§6.4 step 2b). The primary does **not** reject incoming blobs based on their embedded generation value — `configGeneration` provides an audit trail (visible in the Cap'n Proto blob for post-hoc inspection) but does not mechanically prevent an operator from pushing an older config with a different SHA-256. Operators who need downgrade prevention should compare generation values out-of-band before calling `OP_upgradeConfig`. - Crash recovery: the generation is embedded in the `cluster.bin` Cap'n Proto blob, so it survives restarts without a sidecar file. - Default value: `configGeneration` is set to 1 by `load_primary_config()` on first load. If an existing config blob does not include the field (backward compat), the primary treats it as generation 0 and re-encodes with generation 1 on the next save. Replay behaviour: a retried `OP_upgradeConfig` with identical config content is accepted again — the primary re-encodes with an incremented generation, producing a new SHA-256, and restarts. **This is not idempotent:** applying the same config bytes twice produces different on-disk state (generation counter is incremented both times) and triggers two restarts, not one. Systemd's `StartLimitBurst=3` (§11.1) bounds restart storms to 3 within 60 seconds; the admin tool's retry logic must implement deduplication (e.g., comparing the SHA-256 of the just-pushed blob against the primary's reported `versionTag` in the reply) rather than blind retry. The generation counter aids auditing of an operator accidentally pushing a stale config fetched from another controller: the generation field embedded in the blob provides a human-readable version indicator for auditing, even though the primary does not reject blobs based on their generation value. --- ## 5. OP_replConfig Protocol ### 5.1 Schema additions ```capnp # controller.capnp enum OPCode { ...existing entries... replCatchUpDone @28; replConfig @29; # ← new: standby → primary config pull replConfigPush @30; # ← new: primary → standby config push (after OP_upgradeConfig) upgradeConfig @31; # ← new: admin → primary live config upload } struct ReplConfigRequest $C.mapname("repl_config_req_t") { versionTag @0: UInt64 $C.mapname("version_tag"); # 0 = no config saved } struct ReplConfigReply $C.mapname("repl_config_reply_t") { upToDate @0: Bool $C.mapname("up_to_date"); # true = tag matched, no data versionTag @1: UInt64 $C.mapname("version_tag"); # primary's tag (0 if upToDate) sha256 @2: Data $C.mapname("sha256"); # full 32-byte SHA-256 (empty if upToDate) bytes @3: Data $C.mapname("bytes"); # raw cluster.bin bytes (empty if upToDate) } struct ReplConfigPushRequest $C.mapname("repl_config_push_req_t") { versionTag @0: UInt64 $C.mapname("version_tag"); # primary's new tag sha256 @1: Data $C.mapname("sha256"); # full 32-byte SHA-256 bytes @2: Data $C.mapname("bytes"); # raw cluster.bin bytes } struct ReplConfigPushReply $C.mapname("repl_config_push_reply_t") { accepted @0: Bool $C.mapname("accepted"); # true = saved and verified versionTag @1: UInt64 $C.mapname("version_tag"); # standby's tag after save } struct UpgradeConfigRequest $C.mapname("upgrade_config_req_t") { bytes @0: Data $C.mapname("bytes"); # raw cluster.bin blob sha256 @1: Data $C.mapname("sha256"); # caller-computed SHA-256 (primary verifies) } struct UpgradeConfigReply $C.mapname("upgrade_config_reply_t") { versionTag @0: UInt64 $C.mapname("version_tag"); # new tag if accepted; 0 on error error @1: Text $C.mapname("error"); # non-empty if rejected ackedCount @2: UInt32 $C.mapname("acked_count"); # number of standbys that ACKed push } ``` Add union arms to `Request.body` and `RequestReply.data` with explicit field numbers: `replConfigRequestMsg @22: ReplConfigRequest`, `replConfigReplyMsg @8: ReplConfigReply`, `replConfigPushRequestMsg @23: ReplConfigPushRequest`, `replConfigPushReplyMsg @9: ReplConfigPushReply`, `upgradeConfigRequestMsg @24: UpgradeConfigRequest`, `upgradeConfigReplyMsg @10: UpgradeConfigReply`. The field numbers follow the next free slots in each union (after `hostctrlmsg @21` in `Request.body`, after `clusterconf @7` in `RequestReply.data`). **Verified**: The OPCode enum currently ends at `replCatchUpDone @28`. The `Request.body` union ends at `hostctrlmsg @21`. The `RequestReply.data` union ends at `clusterconf @7`. All proposed field numbers are free as of the current `controller.capnp` schema. ### 5.2 Message flow **Pull (standby-initiated, at startup):** ``` Standby Primary │ │ │── OP_replConfig { version_tag=0xA3... } ─▶│ │ │ compare: 0xA3 != 0xF1 │◀── reply { up_to_date=false, │ │ versionTag=0xF1..., │ │ sha256=..., bytes=... } ───────│ │ │ │ validate: SHA256(bytes) == sha256 ✓ │ │ write cluster.bin.f1e2d3c4a5b │ │ update conf/current pointer file │ │── OP_replConfig { version_tag=0xF1 } ───▶│ (optional confirm) │◀── reply { up_to_date=true } ─────────────│ ``` The confirmation request is optional but provides assurance that the standby's saved copy matches. Implementations may skip it. **If tag matches on first request:** ``` │── OP_replConfig { version_tag=0xF1 } ───▶│ │◀── reply { up_to_date=true } ─────────────│ │ proceed to cluster_init() phase 2 │ ``` **Push (primary-initiated, after OP_upgradeConfig — see §6):** ``` Primary Standby │ │ │── OP_replConfigPush { versionTag=0xF2, │ │ sha256=..., bytes=... } ─────────────▶│ │ │ validate SHA-256 │ │ write cluster.bin.f2... │ │ update conf/current │◀── reply { accepted=true, │ │ versionTag=0xF2 } ─────────────│ ``` For the push direction the primary sends an `OP_replConfigPush` request (a dedicated opcode, not a reused reply) to each connected standby after saving a new config via `OP_upgradeConfig`. The standby validates, saves, and ACKs with a `ReplConfigPushReply` confirming the new tag. This preserves the existing request-reply dispatch pattern in `myreq_handler()`. ### 5.3 Authentication model `OP_replConfig` has no application-layer authentication beyond SSL transport-layer encryption. Any controller that completes an SSL handshake (using a valid cert signed by the shared CA) can request the full cluster config blob. This is consistent with the trusted-LAN model adopted by the parent HA design — controller-to-controller sessions are authenticated at the SSL layer, not at the application protocol layer. No additional auth token, HMAC, or challenge-response is added here. `OP_upgradeConfig` (§6) is more sensitive — it changes the cluster configuration and triggers a primary restart. Access control for `OP_upgradeConfig` will be enforced through the existing `validate_host()` mechanism (currently a stub) when it is enhanced — the upgrade handler calls `validate_host()` to check whether the peer is authorised before processing the request. **Safety gate until `validate_host()` is implemented**: Since `validate_host()` is a stub that returns success for all peers, `OP_upgradeConfig` is gated behind a compile-time flag `CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled). Set to 1 in the Meson build options (`-Denable-upgrade-config=true`) when the organisation has implemented `validate_host()` and is ready for the feature. This prevents a half-implemented admin endpoint from being reachable in production before its authorisation check is complete. The compile-time gate is removed once `validate_host()` has a production implementation (tracked in `FUTURE_WORK.md`). ### 5.4 Primary handler The primary handles `OP_replConfig` via a new case in `cluster.c:myreq_handler()` (the existing opcode dispatch switch). The handler is distinct from `handle_controller_join()` which handles the join handshake. **Controller-controller join handshake** (`handle_controller_join()` in `join.c`): 1. The standby opens a TCP connection to `primary_addr:primary_port` (from `--peers`) and completes an SSL handshake (TLS transport per §9.0 `ssl_ctx`). 2. The standby sends `OPCode_join` with `HostRole_controller` and its own host identity (hostname from `--peers`). This identifies the session as a controller-controller (not client) connection. 3. The primary's join handler verifies the peer is a known controller (listed in `--peers`) and returns `OPCode_join` reply with status 0. 4. After successful join, the primary appends the endpoint to `c->ha.standby_endpoints` (replacing the current `-ENOSYS` stub — see §6.3). The standby proceeds to dispatch `OP_replConfig` on the same connection to fetch its config. The primary must have completed Phase 2 (iomgr + endpoint up) before it can serve `OP_replConfig` or accept join handshakes. The standby retry policy (§5.5) handles this startup timing gap. ```c static int handle_config_request(cluster_t *c, endpoint_t *ep, xce_request_t *req) { uint64_t their_tag = req->data.replConfigRequestMsg->version_tag; uint64_t my_tag = c->myconf->config_version_tag; xce_reply_t reply; init_reply_by_req(&reply, req); if (their_tag == my_tag) { reply.status = 0; reply.data.replConfigReplyMsg->up_to_date = true; xce_reply(ep, c->myhost, &reply); return 0; } /* Read raw bytes from conffile and recompute SHA-256 for double-check */ uint8_t *bytes; size_t len; int rc = conf_read_raw(c->myconf->conffile, &bytes, &len); if (rc != 0) { reply.status = rc; xce_reply(ep, c->myhost, &reply); return rc; } /* Recompute to guard against external pointer-file or versioned-file replacement */ uint8_t recomputed[32]; SHA256(bytes, len, recomputed); if (memcmp(recomputed, c->myconf->config_sha256, 32) != 0) { ERROR("Config file changed since startup — refusing to send stale hash"); free(bytes); reply.status = -EIO; xce_reply(ep, c->myhost, &reply); return -EIO; } reply.status = 0; reply.data.replConfigReplyMsg->up_to_date = false; reply.data.replConfigReplyMsg->version_tag = my_tag; reply.data.replConfigReplyMsg->sha256 = /* recomputed, 32 */; reply.data.replConfigReplyMsg->bytes = /* alloc_and_copy(bytes, len) */; xce_reply(ep, c->myhost, &reply); free(bytes); return 0; } ``` The full 32-byte SHA-256 and raw config bytes are `Data` fields in the capnp schema. The codecgen generates `encode_ReplConfigReply_ptr()` / `decode_ReplConfigReply_ptr()` which internally allocate and manage the `Data` blob storage in the capn segment. The C-side shim in `src/lib/request.c` calls these generated functions. See §12 for all affected files. `conf_read_raw()` and `CONF_MAX_SIZE` (defined in `config_repl.c`) are the canonical entry points for reading config bytes — all callers that need raw config bytes (primary handler, `load_primary_config()`, `conf_save_config()`) go through `conf_read_raw()`, which enforces the size limit before allocating memory. `conf_read_raw()` returns `-EFBIG` when the file exceeds `CONF_MAX_SIZE` — callers in request-handling context (the primary handler) send an error reply to the peer; only `load_primary_config()` (at startup) escalates this to `FATAL`. `c->myconf->config_version_tag` and `c->myconf->config_sha256` are computed once in `load_primary_config()` and cached in `myconf_t`: ```c typedef struct { cluster_config_t* cluster_config; char* conffile; char* certfile; char* keyfile; char* workdir; char* name; xce_cluster_config_t peers_cfg; uint64_t config_version_tag; /* ← new: first 8 bytes of SHA-256 */ uint8_t config_sha256[32]; /* ← new: full SHA-256 of conffile bytes */ int conf_keep_n; /* ← new: config version retention count */ uint64_t config_generation; /* ← new: monotonic anti-downgrade guard (§4.4) */ } myconf_t; ``` ### 5.5 Standby fetch The standby makes a request using a standalone `SSL_CTX` created from the same cert/key pair during Phase 1. A dedicated pthread performs a **non-blocking `connect()`** with `poll()` to enable early exit when the election determines a role before the connect completes. This is the same mechanism used by the election probes (§9): 1. Create standalone `SSL_CTX` from `certfile`/`keyfile` (already loaded in Phase 1). 2. Non-blocking `socket()` + `connect()` to the primary candidate. 3. `poll()` with timeout `probe_connect_timeout_ms` (500ms), polling `election_done` between attempts. 4. Once connected, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` handling. 5. Send `OP_replConfig`, receive reply, join thread. No iomgr dependency — this runs after `am_i_primary()` resolves to STANDBY and before Phase 2 (which starts iomgr). The non-blocking connect + poll + SSL_connect approach is the same mechanism used by the election probes (§9) — the config fetch and election probe code paths share this pattern deliberately so both benefit from the same error-handling and timeout logic. **Retry policy**: if the primary is not yet ready (iomgr not up, still in Phase 2 initialisation including `persist_recover()`), the TCP connection is refused or the request times out. The standby retries with exponential backoff: ``` initial_delay_ms = 500 max_delay_ms = 5000 max_attempts = 20 delay = min(initial_delay_ms * 2^attempt, max_delay_ms) total window ≈ 88s (500+1k+2k+4k+16×5k = 87.5s) ``` The retry window is sized to accommodate the primary's Phase 2 startup time (§8.5), specifically `persist_recover()` bounded by `max_wal_records` (see §13). The primary must complete Phase 2 startup (iomgr, persist_recover, endpoint up) before it can serve `OP_replConfig`; standbys retry during this window. Note: `iomgr_init` and `endpoint_init()` complete early in Phase 2 (before `persist_recover()`), so client connections can be dispatched before config state is fully recovered. The `OP_replConfig` handler depends on config being loaded and cached in `myconf` (populated before Phase 2), not on `persist_recover()` completion — the handler reads from `myconf`, not from WAL-recovered state. Post-recovery hooks (`HOOK_AFTER_EVENT`) do not affect config fetch latency. Each retry iteration checks `atomic_load(&c->ha.role) == CTRL_STANDBY` before attempting the connect. If the HA watchdog has promoted this node to primary (e.g., the original primary was detected as failed during the retry window), the fetch thread logs `INFO("Promoted to primary during config fetch — " "aborting fetch")` and returns without error. The caller proceeds with `load_primary_config()` on the primary path. If all attempts fail, the standby checks whether `conf/current` points to a valid versioned file: - **Yes** (saved config exists): log WARN and proceed — see §10.3. - **No** (no saved config): treat as FATAL — see §10.2. --- ## 6. OP_upgradeConfig Protocol ### 6.1 Purpose `OP_upgradeConfig` allows an admin tool to upload a new `ClusterConfig` blob to the running primary at runtime. The primary validates (SHA-256 transit integrity, Cap'n Proto decode), increments the monotonic generation counter, saves, pushes to all connected standbys (best-effort), then restarts itself to apply the new configuration. ### 6.2 Message flow ``` Admin tool Primary Standbys (connected) │ │ │ │─ OP_upgradeConfig▶│ │ │ (bytes, sha256) │ 1. verify SHA256(bytes) │ │ │ 2. decode Cap'n Proto │ │ │ 3. set gen=current+1, re-encode│ │ │ 4. write cluster.bin. │ │ │ 5. update conf/current │ │ │ 6. update config_version_tag │ │ │ 7. push to connected standbys │ │ │── OP_replConfigPush ──────────▶│ │ │ (versionTag, sha256, bytes) │ save to disk │ │◀─ reply { accepted, tag } ─────│ │◀─ ACK (new tag, acked_count) ──│ │ │ │ │ │ │ │ │ │ 8. log "config upgraded, │ │ │ requesting restart" │ │ │ 9. set restart_requested=1 │ │ │ cluster_run() exits loop │ │ │ myapp_run() returns 0 │ │ ↓ │ │ [process exits; systemd/service manager │ │ restarts with same args and new config]│ ``` ### 6.3 Push semantics (hash-first, best-effort, parallel) The primary pushes the new config's hash and tag to all connected standbys **before** ACKing the admin tool. This reduces the silent-rollback window: the admin receives the ACK only after at least attempting to notify all connected standbys. The push is best-effort and **parallel**: all standbys are sent the config simultaneously, and the primary waits at most `CONF_PUSH_TIMEOUT_MS` (default 500 ms, compile-time constant in `config_repl.c`) total for all ACKs. This bounds the pre-restart delay to a single timeout period regardless of standby count. The 500ms default is derived from the expected push workload: a single SSL write (config blob < 1 MiB) + standby SHA-256 verify + disk fsync + SSL ACK reply, bounded by two intra-LAN RTTs (~1ms each) + one local fsync (~50ms). The per-standby worst case is ~55ms; 500ms accommodates up to 9 concurrent standbys or a single degraded standby. **Push-then-ACK ordering**: The primary saves the new config, pushes to standbys, then ACKs the admin tool. The ACK includes the number of standbys that confirmed the push (`acked`), so the admin tool can detect when zero standbys received the update. The ACK reply is extended with: ```capnp struct UpgradeConfigReply $C.mapname("upgrade_config_reply_t") { versionTag @0: UInt64 $C.mapname("version_tag"); # new tag if accepted; 0 on error error @1: Text $C.mapname("error"); # non-empty if rejected ackedCount @2: UInt32 $C.mapname("acked_count"); # number of standbys that ACKed push } ``` If `ackedCount == 0` and standbys are expected, the admin tool logs a WARNING that the config may not have reached any standby. ```c /* Fan-out: launch one pthread per connected standby */ for (int i = 0; i < n_standbys; i++) pthread_create(&push_threads[i], NULL, push_config_thread, &push_ctx[i]); /* Single aggregate wait — 500ms total, not 500ms per standby. */ atomic_int push_done; atomic_init(&push_done, 0); int acked = wait_all_with_timeout(threads, ctxs, n, CONF_PUSH_TIMEOUT_MS, &push_done); LOG("Config pushed to %d/%d standbys; restarting", acked, n_standbys); ``` Implemented signature (in `config_repl.c`): ```c static int wait_all_with_timeout(pthread_t *threads, push_ctx_t *ctxs, int n, uint32_t timeout_ms, atomic_int *done); ``` Uses a 5ms poll loop against `ctx->done` flags until the CLOCK_MONOTONIC deadline, then sets `*done = 1` and joins all started threads. The `done` flag allows thread implementations that check it to exit early; `push_thread` sets `ctx->done = 1` on natural completion, so joins return immediately for threads that finished within the timeout. Offline standbys recover on their next reconnect via the normal pull mechanism (§5.2) — their `conf/current` will have the old tag, the new primary will detect the mismatch and push via `OP_replConfig`. **Failure mode — push not received + primary fails to restart**: If push to all standbys times out (e.g., all standbys offline) AND the primary fails to come back after restart (e.g., the new config triggers a Phase 2 FATAL), the HA watchdog on the standbys promotes one of them. The promoted standby boots with its **saved config** — which is the old version, since it never received the push. The cluster silently rolls back to the previous config. The admin tool now receives `ackedCount == 0` in the ACK reply, providing a clear signal that no standby received the update. If `ackedCount > 0`, at least one standby has the new config and will have it available when it promotes. The silent rollback scenario is limited to `ackedCount == 0` cases where all standbys were offline during the push. Future work may add a quorum-ACK gate that refuses to restart unless a minimum number of standbys confirm (§14.4). **Standby endpoint tracking**: The primary maintains a list of connected standby controller endpoints in `ha_state_t` as a `zarray_t *standby_endpoints` (using the existing `zarray_t` infrastructure, which already manages `c->endpoints` for client connections). This is Phase 2 infrastructure — needed only for `OP_replConfigPush` fan-out during live config upgrade. This list is populated and pruned by the join/disconnect handlers: - **Join**: `handle_controller_join()` (join.c) appends the new controller endpoint to `c->ha.standby_endpoints` after completing the join handshake. This replaces the current `-ENOSYS` stub. - **Disconnect**: `handle_endpoint_error()` removes the endpoint from `standby_endpoints` when a controller-controller connection drops. The existing `HostRole_controller` case in `handle_endpoint_error()` (cluster.c) must be updated to scan `c->ha.standby_endpoints` for the matching endpoint and call `zarray_del()` before the `break`. - **Usage**: `push_config_to_standbys()` iterates `standby_endpoints` to obtain `n_standbys` and the per-standby context for the fan-out push threads. **Thread safety**: `push_config_to_standbys()` takes a **snapshot** of `standby_endpoints` under `c->ha.lock` before spawning push threads. This prevents a concurrent join or disconnect handler from modifying the array while threads are iterating it. The snapshot is a shallow copy of endpoint pointers into a stack-allocated `zarray_t`; endpoints pinned by active push threads are protected from `endpoint_free()` by the endpoint's internal refcount. After all push threads complete, the snapshot array is freed. Only endpoints that completed the SSL handshake and join handshake are in this list — the SSL layer provides authentication (per §5.3 trusted-LAN model). ### 6.4 Handler (primary side) ```c static int handle_upgrade_config(cluster_t *c, endpoint_t *ep, xce_request_t *req) { const uint8_t *bytes = req->data.upgradeConfigRequestMsg->bytes.ptr; size_t len = req->data.upgradeConfigRequestMsg->bytes.len; const uint8_t *sha256 = req->data.upgradeConfigRequestMsg->sha256.ptr; /* 0. Reject oversized blobs before hashing */ if (len > CONF_MAX_SIZE) { send_upgrade_error(ep, req, "Config blob exceeds CONF_MAX_SIZE"); return -EINVAL; } /* 0b. Validate input field lengths before accessing pointers. * Cap'n Proto Data fields with default/empty value have ptr=NULL, * len=0. Gated behind CONF_ENABLE_UPGRADE_CONFIG (default 0, §5.3). */ if (len == 0 || req->data.upgradeConfigRequestMsg->sha256.len != 32) { send_upgrade_error(ep, req, "Invalid request: bytes or sha256 field missing"); return -EINVAL; } /* 0a. Authorisation check — gated behind CONF_ENABLE_UPGRADE_CONFIG * (default 0, §5.3). validate_host() is a stub that returns * success for all peers until enhanced with a real check. * When the compile-time gate is removed, this call is mandatory. */ if (!validate_host(ep, c)) { send_upgrade_error(ep, req, "Not authorised — peer not in allowed hosts"); return -EPERM; } /* 1. Verify caller-supplied SHA-256 */ uint8_t computed[32]; SHA256(bytes, len, computed); if (memcmp(computed, sha256, 32) != 0) { send_upgrade_error(ep, req, "SHA-256 mismatch — blob corrupted in transit"); return -EINVAL; } /* 2. Basic Cap'n Proto decode to catch malformed blobs */ cluster_config_t *new_cfg = decode_cluster_config(bytes, len); if (!new_cfg) { send_upgrade_error(ep, req, "Failed to decode ClusterConfig blob"); return -EINVAL; } /* 2.5 Minimum viability check — rejects trivially invalid configs * that would cause Phase 2 FATAL on restart (§10.7 semantic validation * is a non-goal; this only catches the most basic misconfiguration). */ if (new_cfg->queueCount == 0) { free_cluster_config(new_cfg); send_upgrade_error(ep, req, "Config requires at least one queue"); return -EINVAL; } if (new_cfg->port < 1024 || new_cfg->port > 65535) { free_cluster_config(new_cfg); send_upgrade_error(ep, req, "Config port out of valid range (1024–65535)"); return -EINVAL; } /* 3. Re-encode with incremented generation and recompute SHA-256. * The primary is the sole authority for generation numbers — the admin * tool's configGeneration field is ignored; we always use current+1. */ uint64_t new_generation = c->myconf->config_generation + 1; new_cfg->config_generation = new_generation; uint8_t *new_bytes = NULL; size_t new_len = 0; int rc = encode_cluster_config(new_cfg, &new_bytes, &new_len); free_cluster_config(new_cfg); if (rc != 0) { send_upgrade_error(ep, req, "Failed to re-encode config"); return rc; } SHA256(new_bytes, new_len, computed); /* Use computed (fresh SHA-256) for tag and save; caller-supplied sha256 * was validated for transit integrity above and is no longer needed. */ /* 4–6. Save and update pointer */ char hex_tag[13]; bytes_to_hex(computed, 6, hex_tag); /* 6 source bytes → 12 hex chars + NUL */ rc = conf_save_config(c->myconf, new_bytes, new_len, computed); if (rc != 0) { free(new_bytes); send_upgrade_error(ep, req, "Failed to save new config"); return rc; } /* Update conffile to point to the newly saved versioned file. * Without this, handle_config_request() (§5.4) would read the old * versioned file and its recomputed SHA-256 would mismatch * c->myconf->config_sha256 (already updated below), causing * spurious -EIO replies to standbys during the restart window. */ free(c->myconf->conffile); asprintf(&c->myconf->conffile, "%s/conf/cluster.bin.%s", c->myconf->workdir, hex_tag); uint64_t new_tag; memcpy(&new_tag, computed, 8); /* stored big-endian; swap to host order */ new_tag = be64toh(new_tag); c->myconf->config_version_tag = new_tag; memcpy(c->myconf->config_sha256, computed, 32); c->myconf->config_generation = new_generation; /* 7. Best-effort parallel push to connected standbys. * Push runs BEFORE ACK so the admin tool knows how many standbys * received the update. The reply includes acked_count. If the * primary crashes before push, standbys catch up on their * next reconnect via the normal pull path (§5.2). */ int acked = push_config_to_standbys(c, new_bytes, new_len, computed, new_tag); free(new_bytes); /* 8. ACK the admin — includes acked_count so the admin tool knows * how many standbys confirmed the push. A zero acked_count with * standbys expected is a WARNING condition (see §6.3). */ send_upgrade_reply(ep, req, new_tag, acked); /* Flush the reply to the kernel TCP buffer before signaling restart. * Without this, the iomgr shutdown path may close the endpoint before * the ACK bytes are flushed, causing a connection reset. The admin * tool would be unable to distinguish success from failure. */ endpoint_flush(ep); /* block until kernel send buffer is drained */ /* 9. Request restart — controller is managed by systemd or equivalent * service manager. cluster_run() exits its event loop when this flag is * set, myapp_run() returns cleanly, and the service manager restarts * the process with the same arguments. On restart, conf/current points * to the new config versioned file. No execve() or saved-argv needed. * * IMPORTANT: The main event loop blocks on cbuf_get(&c->wbuf, &stop) * (mylock.c pthread_cond_wait). Setting restart_requested alone cannot * unblock it. We push a stop signal via cbuf_put — the same pattern * used by the existing OPCode_shutdown handler (cluster.c:152-157). * cluster_run() checks restart_requested after the loop exits to * distinguish a clean config-upgrade restart from a shutdown. */ /* peer_cn is the X.509 certificate Common Name — sanitise before logging * to prevent log injection via crafted cert CN containing newlines or * control characters. Limit to 64 printable ASCII chars. */ LOG("Config upgraded to tag=%016"PRIx64" by CN=%s from %s:%u — requesting restart", new_tag, sanitise_cn(peer_cn, 64), peer_ip, peer_port); c->restart_requested = 1; int stop = 1; cbuf_put(&c->wbuf, &stop); /* unblock cluster_run() main loop */ return 0; } ``` ### 6.5 WAL compatibility after config change `persist_recover()` replays WAL records against the in-memory state built from the new config. If the new config **removes** a queue or resource whose ID appears in existing WAL records, replay encounters an unknown ID. **Distinction from existing skip-on-unknown-type behavior** The existing `persist_recover()` already skips WAL records whose *type* is unknown (a forward-compatibility provision for format extensions). This design adds a separate handling strategy for *ID* references within known record types: | Condition | Existing behavior | This change | |---|---|---| | Unknown WAL record type | WARN + skip (safe — future format) | Unchanged | | Known record type, unknown queue ID | Not checked today | **Route to `lost_and_found` queue** | | Known record type, unknown resource/host ID | Not checked today | **Route to `lost_and_found` host or PEND** | Unknown-type records are safe to skip because the type may be from a newer format. Unknown-ID references were previously treated as FATAL, but this made config upgrade operationally fragile — a single zombie job referencing a removed resource would brick the controller permanently (it cannot start because of FATAL, and it cannot upgrade config because that would also FATAL). **Routing strategy for unknown IDs** Instead of FATAL, `apply_wal_entry()` routes unknown-ID references as follows: - **Unknown `queue_id`**: The job is placed in a built-in `lost_and_found` queue. The `lost_and_found` queue is always present in the queue table regardless of config content. Operators can inspect it after recovery and reassign or cancel jobs at their discretion. - **Unknown `host` (node name)**: The job's host is set to `lost_and_found`, a synthetic host entry that accepts jobs but marks them as requiring operator attention. - **Unknown `resource_id`**: The job's resource assignment is cleared and the job is set to `PEND` state. The job will pend until an operator re-creates the resource in the config or cancels the job. A WARN log identifies the orphaned resource reference: ``` WARN("WAL record references unknown queue id=%u at offset %zu in %s — " "routing job to lost_and_found queue", queue_id, record_offset, wal_path) WARN("WAL record references unknown resource id=%u at offset %zu in %s — " "clearing resource assignment, job set to PEND", resource_id, record_offset, wal_path) WARN("WAL record references unknown host '%s' at offset %zu in %s — " "routing to lost_and_found host", host_name, record_offset, wal_path) ``` **`lost_and_found` queue and host** These are synthetic entries created by `cluster_init()` Phase 2 after `kvmgr makeInstance` and **before `persist_recover()`** (see startup diagram in §7). This ordering is load-bearing: unknown queue/resource/host IDs encountered during `persist_recover()` must have valid targets to route to. Consider adding a runtime assertion in `create_lost_and_found()` that validates `kvmgr` is initialized and `persist_recover()` has not yet been called. They are not part of the admin-supplied config blob — they are added by the controller itself. This ensures they always exist, even after a config change that removes all user-defined queues or hosts. Their existence is not persisted across restarts; they are re-created on each boot. **Implementation**: `create_lost_and_found()` adds entries to kvmgr's queue table and host table via the existing `queue_insert()` and `host_insert()` APIs, using reserved names `"lost_and_found"`. These are normal kvmgr entries — no special-casing in the queue/host lookup paths is required. **Name reservation**: the name `"lost_and_found"` is permanently reserved. The `OP_upgradeConfig` handler rejects any `ClusterConfig` blob containing a queue or host named `"lost_and_found"` — the admin tool receives an error: `"Reserved name: 'lost_and_found' queue/host names are reserved by the controller. Rename the user-defined entry and retry."` This prevents the ambiguity between user-defined and synthetic entries at config ingestion time rather than at WAL replay time. `create_lost_and_found()` always creates the synthetic queue and host; if the name somehow exists (e.g., loaded from a pre-upgrade saved config), it skips creation and logs ERROR. After creation, standard lookup functions (`get_queue_by_name()`, `get_host_by_name()`) resolve them. **Enforcement at all entry points**: Both `load_primary_config()` (primary at startup, §7.1) and `handle_upgrade_config()` (live upgrade, §6.4) perform the same `config_contains_reserved_name()` check. This ensures the synthetic `lost_and_found` queue and host cannot be aliased with a user-defined entry regardless of how config enters the system — cold boot or live push. The check is implemented once in `config_repl.c` and called from both paths. **Routing helper**: A `resolve_queue_or_lost_and_found(cluster_t *c, uint32_t queue_id)` helper in `config_repl.c` is called inside `apply_wal_entry()` (persist.c) and `restore_job()` (job.c) when a WAL record references an unknown queue ID. The helper: 1. Looks up `queue_id` in the queue table. 2. If found, returns the queue pointer. 3. If not found, logs a WARN (with record offset and WAL path, per §6.5 examples) and returns the `lost_and_found` queue pointer. Analogous helpers for host resolution (`resolve_host_or_lost_and_found()`) and resource resolution (`clear_resource_and_pend()`) follow the same pattern. All three are called from the WAL apply path in `persist_recover()` → `apply_wal_entry()`. **Observability**: The `lost_and_found` queue and host are discoverable via `xcecli queue list` and `xcecli host list` — they appear as normal entries with the reserved name `"lost_and_found"`. A metric counter `lost_and_found_job_count` is incremented each time a job is routed to this queue during `persist_recover()`. If `lost_and_found_job_count > 0` at startup, a WARN summary is logged. The counter is stored in `cluster_t` as an `atomic_int`, initialised to 0 at startup and incremented during `persist_recover()`. It is not persisted across restarts. The counter is reset to 0 after a successful drain operation (e.g., the future `xcecli lost-and-found drain` command reassigns or cancels all sidelined jobs). Jobs accumulate across config upgrades (each upgrade writes new versioned files but the WAL is truncated after checkpoint — see §13.4), so repeated upgrades do not cause unbounded growth. Operators should inspect and drain `lost_and_found` after a config upgrade; a future `xcecli lost-and-found drain` command can automate reassignment or cancellation (§14.4 future work). **Job submission gate**: The API server (`src/frontend/apiserver`) MUST reject job submissions targeting the `lost_and_found` queue with a clear error: `"Cannot submit jobs to 'lost_and_found' — this is an internal recovery queue."` The queue appears in `xcecli queue list` for operator visibility but does not accept external job submissions. This prevents accidental routing of production jobs to the synthetic recovery queue and keeps `lost_and_found_job_count` meaningful (only incremented for genuine WAL-recovered orphans, not operator mistakes). The check is enforced at the API layer (apiserver), not in the controller, so standby-only deployments without the API server still have the controller-side protection of §6.5's name reservation. ### 6.6 Downtime estimate for config upgrade The primary is unavailable from when it sets `restart_requested` until Phase 2 completes on restart: **~4–7 seconds nominal, ~8–13 seconds with 1000 jobs** (per §8.5 timing table). This is the same duration as a cold start — the process exits cleanly and systemd restarts it with the same arguments. Standbys do **not** restart during config upgrade — they receive the new config via `OP_replConfigPush` and save it to disk, but continue running with their current in-memory state. If the primary fails to come back (e.g., the new config is corrupt), the HA watchdog on the standbys will detect the primary's absence and trigger an election. The new primary then boots with the already-saved config. The admin tool receives the ACK before the restart begins (§6.4 step 6), so the caller knows the config was accepted within `probe_connect_timeout_ms` (500ms for push fan-out + network round-trip). The restart itself is asynchronous — the admin tool should poll `OP_probe` or check the primary's log to confirm it's back online. ### 6.7 In-flight WAL checkpoint during config-upgrade restart If an async WAL checkpoint (§13.4) is in progress when `handle_upgrade_config()` triggers restart, the restart must not interrupt the checkpoint mid-write (which could leave a partial snapshot file on disk). **Handling**: before setting `restart_requested`, the handler waits for any in-flight checkpoint to complete: ```c /* Wait for in-flight checkpoint before restarting. * schedule_checkpoint() is async via vtaskmgr; the checkpoint task sets * persist->checkpoint_done when finished (same atomic used by the * idempotency guard in schedule_checkpoint()). */ struct timespec ckpt_wait_start, ckpt_wait_now; clock_gettime(CLOCK_MONOTONIC, &ckpt_wait_start); while (atomic_load(&c->persist->checkpoint_in_progress)) { usleep(10000); /* 10ms spin — checkpoint is I/O-bound, completes quickly */ clock_gettime(CLOCK_MONOTONIC, &ckpt_wait_now); if ((ckpt_wait_now.tv_sec - ckpt_wait_start.tv_sec) > 30) { ERROR("Checkpoint did not complete within 30s — restarting anyway"); break; } } ``` This is safe because: - The checkpoint is bounded: it flushes in-memory state and truncates the WAL, completing in well under 1 second on typical hardware. - The spin-wait does not hold any locks — `handle_upgrade_config()` runs on the admin connection's thread, not the main event loop. - If the checkpoint hangs (bug), the 30s timeout unblocks the restart; the controller is not permanently stuck. The admin tool's connection timeout provides a second bound. - After the checkpoint completes, the WAL is empty (truncated at checkpoint), so the next `persist_recover()` on restart replays zero records — the fresh snapshot already contains all state. --- ## 7. Startup Sequence The startup sequence for both roles is updated from the current monolithic `cluster_init()` to a two-phase model. ``` Controller starts │ ▼ Parse --peers → my_priority, my_port │ ▼ Phase 1 — minimal init (no config, no iomgr) ┌─────────────────────────────────────────┐ │ xce_hook_init + xce_install_hooks │ │ initialize_openssl │ │ ssl_ctx = create_ssl_context(cert, key) │ │ ha_init(&c->ha, &ha_cfg, my_priority) │ │ c->ha.ssl_ctx = ssl_ctx /* stash */ │ │ (c->ha.ssl_ctx and c->iomgr.ssl_ctx │ │ are the same pointer; Phase 2 calls │ │ iomgr_set_ssl_ctx(c->iomgr, ssl_ctx) │ │ for the existing iomgr path. The │ │ ha.h "no new TLS" comment is removed; │ │ see §11 ha.h) │ │ ha_install_sigusr1 │ └─────────────────────────────────────────┘ │ ▼ am_i_primary()? ← real election probing (§9) │ ┌────┴────────────────────────────────────┐ │ │ PRIMARY STANDBY │ │ ▼ ▼ load_primary_config() INFO("Starting as standby ← reads conf/current (priority %d); requesting ← computes SHA-256 config from primary %s:%u", ← FATAL if missing my_priority, host, port) │ │ │ fetch_config_from_primary() │ ← OP_replConfig exchange │ ← save to conf/ (§3.2) │ ← retry with backoff (§5.5) │ ← FATAL if no config at all │ │ └────────────────┬────────────────────────┘ │ ▼ Phase 2 — full init (config available) ┌─────────────────────────────────────┐ │ iomgr_init │ │ endpoint_init(port from --peers) │ │ HOOK_BEFORE_EVENT → load resources │ │ (must check cluster_config!=NULL │ │ and early-return — config already │ │ loaded before Phase 2) │ │ kvmgr makeInstance (sizes from cfg) │ │ xstring_table_init, job_obj_init │ │ create_lost_and_found() │ │ ← lost_and_found queue + host │ │ ← always present regardless of │ │ config content (§6.5) │ │ vtaskmgr_init, delaymgr_init │ │ persist_init │ │ ha_promote (primary) or │ │ CTRL_STANDBY (standby) │ │ HOOK_AFTER_EVENT → build queues, │ │ groups, load jobs │ │ persist_recover │ └─────────────────────────────────────┘ │ ▼ cluster_start() ``` ### 7.1 Primary config load — fail-fast behaviour `load_primary_config()` replaces `get_default_conf()` on the primary path: ```c static int load_primary_config(cluster_t *c) { if (access(c->myconf->conffile, F_OK) != 0) { FATAL("Primary candidate (priority 0) has no config file '%s' — " "provision the config before starting the primary controller", c->myconf->conffile); return -ENOENT; } /* read, decode Cap'n Proto, compute SHA-256, cache version_tag + full hash in myconf */ ... } ``` /* After decoding, check for reserved "lost_and_found" name — same check * as OP_upgradeConfig handler (§6.5). Reject at boot rather than * silently aliasing the synthetic queue/host with user-defined entries. */ if (config_contains_reserved_name(new_cfg)) { FATAL("Config file '%s' contains reserved name 'lost_and_found' — " "rename the user-defined queue or host and retry", c->myconf->conffile); return -EINVAL; } A priority-0 node with a missing config file must not start — it would boot with empty queues and resources, silently losing all cluster configuration. ### 7.2 Standby INFO log on startup When a standby starts and its election probe detects a live higher-priority candidate: ``` INFO("HA: higher-priority candidate %s:%u is alive — starting as standby " "(my_priority=%d)", primary_host, primary_port, my_priority) INFO("HA: requesting cluster config from primary %s:%u (my_tag=%016"PRIx64")", primary_host, primary_port, saved_tag) ``` After successful config fetch: ``` INFO("HA: config received and saved (tag=%016"PRIx64") — proceeding with full init", new_tag) ``` --- ## 8. cluster_init() Phase Split ### 8.1 What moves to phase 1 (before config) From the current `cluster_init()` (`cluster.c`): | Line(s) | Code | Phase | |---|---|---| | 544–546 | `xce_hook_init`, `xce_install_hooks` | 1 | | 557–561 | `initialize_openssl`, `create_ssl_context` | 1 | | 665–671 | `ha_init`, `ha_install_sigusr1` | 1 | | new | `zarray_create(&c->ha.standby_endpoints, 4, NULL)` — initialise the standby endpoint tracking array in `ha_state_t` before any join handler can append to it (§6.3). Uses NULL killer (endpoints are freed by `endpoint_free()` called by the iomgr disconnect path, not by zarray) | 1 | ### 8.2 What stays in phase 2 (after config) | Line(s) | Code | Phase | |---|---|---| | new (before iomgr) | Stop heartbeat listener: set `atomic_store(&c->heartbeat_listener_running, 0)` and `pthread_join(c->heartbeat_listener_tid, NULL)` — the join must complete before `iomgr_init` to free the listening port and prevent `EADDRINUSE` | 2 | | 580 | `iomgr_init` | 2 | | 589–594 | `endpoint_init` (port from `--peers`) | 2 | | 548–552 | `xce_run_hooks HOOK_BEFORE_EVENT` → in `xce_config_init()` (cluster.c:411-417), add `if (c->myconf->cluster_config != NULL) return 0;` guard before the existing `get_default_conf(c)` call. Config is already loaded by `load_primary_config()` or `fetch_config_from_primary()` before Phase 2. The HOOK_AFTER_EVENT case (cluster.c:419-446: `cluster_build_config`/resources/queues/groups/`load_jobs`) is unchanged. The static global `cf` in myconf.c and its associated HOOK_AFTER_EVENT SHUTDOWN cleanup become dead code — `myconf_fini()` takes over freeing `cluster_config`. | 2 | | 563–579 | `kvmgr makeInstance` (sizes from config) | 2 | | 584–585 | `xstring_table_init`, `job_obj_init` | 2 | | 602–608 | `vtaskmgr_init`, `delaymgr_init` | 2 | | 614–629 | `persist_init` | 2 | | 674–679 | `ha_promote_to_primary` or `CTRL_STANDBY` | 2 | | 681–683 | `xce_run_hooks HOOK_AFTER_EVENT` | 2 | | new | `create_lost_and_found()` — synthetic queue + host (§6.5) — **must complete before `persist_recover()` begins; see §6.5 ordering note** | 2 | | 686–691 | `persist_recover` | 2 | ### 8.3 kvmgr sizing with default values The kvmgr hash slot counts (`resmgr->nhash_slots`, `jobmgr->nhash_slots`, etc.) come from `ClusterConfig`. The slot count only affects hash collision rate — it has no effect on correctness. A standby that has never received config can safely use the same defaults as `fill_default_cluster_config()` while waiting, then re-initialize with the correct sizes once config arrives. In practice, the standby receives config before phase 2 begins, so it always uses the correct sizes. The default fallback is belt-and-suspenders only. ### 8.4 Listening port `endpoint_init()` currently reads the port from `cluster_config->port`. After the split, phase 2 derives the port from the `--peers` candidate list (available from CLI parsing before any phase): ```c /* Find own entry in peers list by name match */ uint32_t my_port = 0; for (int i = 0; i < pcfg->candidates_count; i++) { if (strncmp(pcfg->candidates[i].name, c->myconf->name, XCE_MAXNAME_LEN - 1) == 0) { my_port = pcfg->candidates[i].port; break; } } if (my_port == 0) my_port = 9990; /* design default */ ``` This is already available in `myconf.peers_cfg` before any config is loaded. `xce_candidate_t` (defined in `src/include/xce/cluster.h`) carries a `port` field populated by `xce_parse_peers()` from the `name=host:port` CLI syntax — confirmed in the struct definition at line 21 of that header. No new parsing is needed. ### 8.5 Phase 2 timing decomposition The standby retry window must be sized to survive Phase 2 startup on the primary. Each Phase 2 component contributes: | Component | Nominal | With 1000 jobs | |---|---|---| | `iomgr_init` + `endpoint_init` | <50ms | <50ms | | `kvmgr makeInstance` ×5 | <100ms | <100ms | | `xstring_table_init` + `job_obj_init` | <50ms | <50ms | | `HOOK_BEFORE_EVENT` (load resources) | <500ms | <2s | | `vtaskmgr_init` + `delaymgr_init` + `apiserver_init` | <200ms | <500ms | | `persist_init` | <100ms | <200ms | | `persist_recover` (bounded by §13, ≤10000 records) | 2–5s | 2–5s | | `HOOK_AFTER_EVENT` (queues, groups, jobs) | <1s | <5s | | **Total** | **~4–7s** | **~8–13s** | The ~88-second standby retry window provides 7×–11× headroom over the worst case. Operators of very large clusters (10000+ jobs) should verify Phase 2 duration from the primary's startup log and tune `max_attempts` × `max_delay_ms` accordingly. --- ## 9. am_i_primary() Implementation The current `am_i_primary()` stub (`ha.c:213`) returns `c->ha.my_priority == 0`, which is never correct for a multi-node cluster — it never probes whether a higher-priority candidate is actually alive. ### 9.0 Simultaneous-boot symmetry breaking When all candidates start simultaneously, nobody is primary yet and nobody responds to election probes. Without symmetry breaking, every candidate would expire its deadline and self-declare primary — a multi-way split-brain. The mechanism has two parts: **1. Dedicated heartbeat listener (all self-declared primaries)** After Phase 1, **every candidate that `am_i_primary()` returns true for** launches a minimal heartbeat listener thread before entering Phase 2. This is not limited to priority-0 — if a lower-priority candidate wins the jitter race and self-declares primary, its heartbeat listener is essential for the higher-priority true primary to detect the split-brain during its own election probe. The listener is started in a dedicated pthread immediately after `am_i_primary()` returns: ```c if (am_i_primary()) { /* Start a minimal SSL listener that responds to OP_heartbeat probes. * This lets lower-priority candidates detect us while Phase 2 initializes. */ start_heartbeat_listener(c); /* dedicated pthread — non-blocking poll loop */ load_primary_config(c); /* FATAL if missing */ ... } ``` The heartbeat listener is fully specified in the parent HA design (`persistence-and-ha-design.md` Part 5). This section only describes the integration point relevant to config replication: the listener must be active before the standby's `fetch_config_from_primary()` connects, so the standby can reach a primary that is still in Phase 2 initialisation. Summary of the listener's properties (see parent design for full specification): - Binds the port from `--peers` using `c->ha.ssl_ctx` - Non-blocking poll loop; accepts `OP_heartbeat` only; rejects other opcodes - Does NOT depend on iomgr, vtaskmgr, persist, or any Phase 2 component - Exits when the main thread clears `atomic_int heartbeat_listener_running` (set to 0 before `iomgr_init` in Phase 2). The main thread calls `pthread_join(heartbeat_listener_tid, NULL)` before `endpoint_init()` to prevent EADDRINUSE. - Registered in `cluster_t` (`c->heartbeat_listener_tid`, `c->heartbeat_listener_running`) so `cluster_fini()` can join the thread on any FATAL exit path before Phase 2 **2. Priority-based base delay + randomized jitter (all candidates)** Each candidate's election timeout has two components: a deterministic **priority-based base delay** (higher priority = shorter delay) plus a small random jitter for tie-breaking: ```c /* Priority-based base delay: priority-0 waits 0ms, priority-1 waits * election_priority_step_ms, priority-2 waits 2× that, etc. * This ensures the highest-priority node nearly always wins without probing. */ long to_ms = c->ha.config.election_timeout_ms; to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms; /* Small random jitter for tie-breaking between same-step candidates */ long jitter_ms = (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms); to_ms += jitter_ms; ``` With `election_timeout_ms = 2000`, `election_priority_step_ms = 500`, and `election_jitter_max_ms = 200`, the effective deadlines are: - Priority-0: 2000ms–2199ms (self-declares primary first) - Priority-1: 2500ms–2699ms (probes priority-0 first; if alive, becomes standby) - Priority-2: 3000ms–3199ms (probes priority-0 and priority-1 first) The priority-based base delay all but eliminates simultaneous split-brain during normal boots: priority-0 declares primary and starts its heartbeat listener ~500ms before priority-1's deadline, so priority-1's probe finds a live primary. The priority-based step and jitter together make simultaneous split-brain vanishingly unlikely for N=3 in systemd-managed deployments: - Priority-0 fires at t=0–199ms (election_timeout_ms + jitter only) - Priority-1 fires at t=500–699ms (step + jitter, probes priority-0 first) - Priority-2 fires at t=1000–1199ms (2×step + jitter) Adjacent-priority candidates collide only if the lower-priority node starts so late that its jitter-extended deadline precedes the higher-priority node's first heartbeat response, which requires the 500ms step to be less than 2×J_max (400ms) — false for the configured values. For three nodes to all self-declare primary, all three must start within the same ~200ms jitter window, which is bounded by OS scheduler precision and systemd startup ordering. The `election_priority_step_ms` is added to `ha_config_t` alongside the existing `election_timeout_ms` and `election_jitter_max_ms`. The jitter-only fallback (priority-0 with no step) retains the original behaviour when `election_priority_step_ms = 0`. The window is bounded: in the worst case (all jitter values 0ms apart), all candidates declare primary. Once both reach Phase 2, the HA watchdog resolves the split via priority comparison in heartbeat exchange (see parent HA design). This is bounded by the time for the slower candidate to complete Phase 2 (~4–13s per §8.5) plus one heartbeat interval (~500ms). **Split-brain window during simultaneous boot**: The heartbeat listener starts immediately after `am_i_primary()` returns, but it is a minimal pre-iomgr listener that only handles `OP_heartbeat`. **All candidates that self-declare primary launch a heartbeat listener**, not just priority-0 — this is critical: if a lower-priority candidate wins the jitter race and self-declares primary, its heartbeat listener provides the response that resolves the split-brain when the higher-priority (true) primary probes it. During simultaneous boot, all candidates start with `ha.term = 0` (`atomic_store_explicit(&ha->term, 0ULL, ...)` in `ha_init()`, ha.c:34). Term-based priority comparison cannot break symmetry at this stage — symmetry breaking during simultaneous boot relies entirely on the jitter mechanism (part 2 below). Term-based step-down becomes effective only after the first successful election, when the winning primary increments its term. The jitter window is therefore the primary defence against multi-way split-brain during cold start; the heartbeat listener provides the detection channel but term comparison is not yet discriminating. If a heartbeat listener fails to bind its port (`EADDRINUSE`), the candidate logs a WARN indicating a likely split-brain condition (another candidate has already bound the port) and continues initialization. This is expected during simultaneous boot with overlapping jitter windows — the candidate that bound the port first keeps it; the other continues to Phase 2 where the HA watchdog's heartbeat exchange will resolve the split-brain. If a lower-priority candidate also self-declares primary and enters Phase 2, the HA watchdog heartbeat exchange will resolve the split-brain once both candidates reach Phase 2 — but this requires the HA watchdog to compare candidate priority (from `--peers` ordering) in addition to term, since both nodes have term=0 during simultaneous boot. **This design implements** `ha_check_heartbeat_nocluster()` (ha.c:101-126) with a priority comparison at term=0 so that a lower-priority self-declared primary steps down when it receives a heartbeat from a higher-priority node with the same term. Without this enhancement, the current elapsed-time-only check cannot distinguish a legitimate primary from a split-brain peer during cold start. **CRITICAL DEPENDENCY — implemented by this design**: The `ha_check_heartbeat_nocluster()` enhancement (priority comparison at term=0) is a **hard prerequisite** for correct simultaneous-boot behavior. Without it, a lower-priority self-declared primary will not step down when it receives a heartbeat from a higher-priority peer with the same term — the split-brain is unbounded, not 4–13s. This enhancement is implemented in this design (`ha.c` — see §12 files changed) alongside the election probing. The parent HA design (`persistence-and-ha-design.md` Part 5) specifies the heartbeat exchange requirements; this design provides the priority-comparison implementation which the parent design references. Test coverage: `test_am_i_primary_simultaneous_boot` in §14.1 validates priority-based step-down at term=0 during simultaneous boot. **Implementation scope**: This design adds a `sender_priority` field to the `OP_heartbeat` request in `controller.capnp`. The heartbeat listener (§9.0) populates this field from `c->ha.my_priority`. `ha_check_heartbeat_nocluster()` (ha.c) compares `sender_priority` against `ha->my_priority` when `term == 0`: if the sender has higher priority (lower index in `--peers`), the local node steps down to standby. The data source is the heartbeat message, not `ha_state_t`, so both the schema change and the handler change are part of this design. The effective split-brain window is therefore the time for the secondary-primary to complete Phase 2 and start processing heartbeats: ~4–13s per §8.5. During this window both self-declared primaries have `iomgr_init` and `endpoint_init` complete (per the Phase 2 ordering in §7), so both are accepting client connections before `ha_promote` resolves the split-brain. Conflicting client operations (job submissions, queue modifications) submitted to both primaries during this window may diverge. This falls within the parent design's acknowledged `split_brain_risk_accepted` model, with the caveat that the window is longer than a single heartbeat interval during cold start. **Data divergence during split-brain**: The losing primary's WAL is discarded when it steps down to standby — any client operations it accepted during the split-brain window are lost. This design does not add a reconciliation protocol; the scope is bounded to the existing HA model. Operators who cannot tolerate any data loss during cold start should not submit client operations until the primary's `INFO("Promoted to primary")` log message confirms Phase 2 is complete. The typical cold-start window occurs only during initial cluster boot or full-cluster restart — not during normal single-node failover where an existing primary with a term > 0 prevents the split-brain scenario from occurring. **PRNG**: The jitter calculation uses an xorshift64 PRNG seeded from `/dev/urandom`. The PRNG state is stored in `ha_state_t.jitter_seed` (one per node, initialised during Phase 1). The xorshift64 implementation should be a shared inline function in `ha.h` (or a small `prng.h` utility header) so both the election probe (§9.1) and the heartbeat listener (parent HA design) use the same generator without code duplication. The seed is initialised from `/dev/urandom` (read 8 bytes) during Phase 1. If `/dev/urandom` is unavailable, fall back to `clock_gettime(CLOCK_MONOTONIC, &ts)` nanoseconds XOR'd with `((uint64_t)priority << 56) ^ ((uint64_t)getpid() << 16)` with a `WARN` log. The shift mixing avoids the correlation problem where two systemd-started processes in the same nanosecond produce jitter values differing only by their priority index (a small integer). The `time(NULL) ^ getpid()` fallback is rejected because it produces nearly identical values for processes started simultaneously by systemd (same second, sequential PIDs) — exactly the scenario where symmetry breaking matters most. ### 9.1 Algorithm (design doc §5.4) ```c int am_i_primary(cluster_t *c) { /* Priority 0: unconditionally primary — highest-priority node does not probe. * Safe because all controllers receive the same --peers list, so ordering is * deterministic and priority-0 is unique across the cluster. */ if (c->ha.my_priority == 0) return 1; /* Launch one pthread per candidate with priority < my_priority. * Each thread makes a non-blocking connect + poll loop to the candidate, * sends OP_heartbeat. If ANY thread gets a live ACK before the shared * election_timeout_ms deadline → return 0 (standby). * If the deadline expires with no ACK → return 1 (primary). */ atomic_int election_done = 0; probe_ctx_t probes[MAX_HA_CANDIDATES]; int n_probes = 0; for (int i = 0; i < c->ha.my_priority; i++) { probes[n_probes++] = (probe_ctx_t){ .candidate = &c->myconf->peers_cfg.candidates[i], .ssl_ctx = c->ha.ssl_ctx, /* stashed in Phase 1 */ .timeout_ms = c->ha.config.probe_connect_timeout_ms, .election_done = &election_done, .got_ack = 0, /* set to 1 by probe_thread on success */ }; pthread_create(&probes[n_probes-1].thread, NULL, probe_thread, &probes[n_probes-1]); } /* Wait for first ACK or deadline (with jitter for simultaneous-boot, see §9.0) */ long to_ms = c->ha.config.election_timeout_ms; to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms; to_ms += (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms); struct timespec deadline; clock_gettime(CLOCK_MONOTONIC, &deadline); deadline.tv_sec += to_ms / 1000; deadline.tv_nsec += (to_ms % 1000) * 1000000L; if (deadline.tv_nsec >= 1000000000L) { deadline.tv_sec++; deadline.tv_nsec -= 1000000000L; } int got_ack = wait_any_probe(probes, n_probes, &deadline); /* Signal all threads to stop (no pthread_cancel — unsafe through OpenSSL) */ atomic_store(&election_done, 1); for (int i = 0; i < n_probes; i++) pthread_join(probes[i].thread, NULL); return got_ack ? 0 : 1; } ``` ### 9.2 Thread coordination `pthread_cancel()` is **NOT** used (cancellation through OpenSSL shutdown handlers can deadlock). Each probe thread uses **non-blocking `connect()` + `poll()`**: 1. `socket()` + `fcntl(O_NONBLOCK)` + `connect()` → expect `EINPROGRESS` 2. `poll()` with timeout `probe_connect_timeout_ms` (default 500ms) 3. Between poll attempts, check `atomic_load(&election_done)` — if set, exit cleanly 4. If `poll()` returns -1 and `errno != EINTR`, treat as probe failure — log at TRACE level and exit thread with `*got_ack = 0` 5. After connect, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` polling; on hard SSL failures (certificate expired, protocol mismatch, peer disconnect), treat as probe failure — log at TRACE level and exit thread 6. If `send()` or `recv()` fails on the SSL socket, treat as probe failure 7. Send `OP_heartbeat`, receive ACK, set `*got_ack` via shared pointer The per-attempt poll timeout bounds the worst-case exit delay to one poll interval per thread (~500ms). TCP connect timeout (kernel SYN retry) is not reached because the thread checks `election_done` between each poll iteration and exits early. Probe failures are expected when peers are down — only TRACE-level logging is appropriate; ERROR logs would generate noise during normal operation. ### 9.3 Election outcome logging ``` /* Higher-priority candidate alive: */ INFO("HA election: candidate %s:%u responded — starting as standby (my_priority=%d)", winner_host, winner_port, my_priority) /* No higher-priority candidate alive: */ INFO("HA election: no higher-priority candidate responded within %ums — " "starting as primary (my_priority=%d)", election_timeout_ms, my_priority) ``` --- ## 10. Error Handling ### 10.1 Primary with missing config **Condition**: `am_i_primary()` returns 1 (this node is primary) AND `conf/current` does not exist or does not point to a readable versioned file. **Action**: `FATAL` log + return error from `load_primary_config()`, which propagates to `myapp_run()` → `cluster_fini()` → exit. ``` FATAL("Primary candidate (priority 0) has no config file '%s' — " "provision the config before starting the primary controller. " "Run: xce-config --workdir %s --name %s --config cluster.yaml", conffile, workdir, name) ``` ### 10.2 Standby with no primary and no saved config **Condition**: `am_i_primary()` returns 0 (standby) AND all config fetch retries fail AND `conf/current` does not exist. **Action**: `FATAL` log + exit. A standby cannot operate without any config — it cannot initialise kvmgrs, load queues, or accept connections. ``` FATAL("Standby (priority %d) could not fetch config from primary and has no " "saved config — start the primary controller first, or provision " "the config file manually at '%s'", my_priority, conffile) ``` ### 10.3 Standby with no primary but saved config **Condition**: `am_i_primary()` returns 0 AND all config fetch retries fail AND `conf/current` exists and is valid. **Action**: `WARN` log, proceed with saved config. This allows a standby to restart during a primary outage without being permanently blocked. A `conf_stale_timeout` (default: 24h, configurable via `--conf-stale-timeout` CLI flag) soft-limits staleness: configs older than the threshold are flagged but not rejected (see below). ``` WARN("Standby (priority %d) could not reach primary for config fetch — " "using saved config (tag=%016"PRIx64"). Config may be stale.", my_priority, saved_tag) ``` A `conf_stale_timeout` field (default: 24h, configurable via `--conf-stale-timeout` CLI flag) is stored in `myconf_t` alongside `conf_keep_n`. The 24h default is chosen to match the typical maintenance window: if a primary is down for less than 24h, the config on any standby that was connected before the outage is at most 24h old, which is within the expected change cadence for HPC cluster configuration. Operators may increase this for clusters with infrequent config changes, or set it to 0 to disable staleness checking entirely. When a standby boots with a saved config older than `conf_stale_timeout` seconds (determined by comparing the versioned file's mtime against `CLOCK_REALTIME`), the controller logs a WARN and proceeds, but sets a `config_stale` flag in `myconf_t`. The primary checks this flag on the first `OP_replConfig` exchange after the standby reconnects: if the stale config's tag does not match the primary's current tag, the primary pushes the new config as normal. If a standby promotes while its config is stale (older than `conf_stale_timeout`), the promotion proceeds — the stale timeout is a soft indication, not a hard block, because the config may still be functionally valid. Operators can set `conf_stale_timeout` to 0 to disable staleness checking. The `config_stale` flag is recomputed each time the controller boots (comparing the versioned file's mtime to the current time). It is not persisted across restarts. If a standby reconnects to the primary and successfully fetches a new config matching the primary's current tag, the flag is cleared (the saved config is now fresh). If the standby promotes while `config_stale` is set, it proceeds to Phase 2 as primary with the acknowledgment that the config may be stale — operators should monitor logs and trigger a config upgrade via `OP_upgradeConfig` if necessary to synchronise the cluster. ### 10.4 SHA-256 mismatch on receipt **Condition**: standby recomputes SHA-256 of received bytes ≠ `reply.sha256`. **Action**: discard the received bytes, log `ERROR`, retry the fetch (up to `max_attempts`). If all retries fail with SHA-256 mismatches, treat as §10.2. ``` ERROR("Config fetch SHA-256 mismatch from primary — expected %s, got %s. " "Retrying (%d/%d)", expected_hex, actual_hex, attempt, max_attempts) ``` ### 10.5 Zero-tag in received config (redundant check) **Condition**: primary sends `reply.version_tag = 0`. **Action**: `FATAL` on the standby. With SHA-256 the probability of a legitimate config producing `version_tag = 0` is ~2^-64 — negligible. A zero tag in the reply indicates a primary-side bug (not a hash collision). The error message distinguishes this: `"BUG: primary sent zero version_tag for config. Primary conffile " "may be empty or corrupted: '%s'"`. ### 10.6 Empty config file **Condition**: config file exists but is 0 bytes. **Action**: `FATAL`. `SHA256("")` produces a non-zero hash, but an empty file has no valid Cap'n Proto content. The error message reads: `"Config file '%s' is empty — not a " "valid ClusterConfig blob"`. ### 10.6a Empty pointer file **Condition**: `conf/current` exists but is 0 bytes (zero-length pointer file). **Action**: `WARN` on primary, treat as missing config (escalate to §10.1). Standby treats as trigger for full config fetch from primary (the saved config is unreachable because the pointer file was not fully written — typically a crash between `open()` and the first `write()`+`fsync()`+`close()` in §3.2 step 2). The error message reads: `"Config pointer file 'conf/current' exists but is empty — treating as missing config. " "This can occur if the controller crashed during a config write."` A zero-length `conf/current` is distinct from §10.6 (empty versioned file) — `fgets()` returns `NULL` on an empty file without setting `errno`, and the caller receives `-EIO` from `conf_read_current_filename()`. Adding a dedicated diagnostic before the `fgets()` call (checking file size via `fstat()` or reading one byte with `fread()`) makes the failure cause explicit rather than opaque. ### 10.7 Config file size limit **Condition**: config file exceeds `CONF_MAX_SIZE` (default 1 MiB). **Action**: `FATAL`. A Cap'n Proto cluster config should be small (< 100 KB); a larger file indicates corruption or a misconfiguration. The `conf_read_raw()` function rejects files over this limit before allocating memory. ### 10.8 Standby with corrupt saved config **Condition**: standby has `conf/current` pointing to a valid file, but its SHA-256 differs from the primary's cached hash. **Action**: log `WARN`, discard saved config after receiving the correct one from primary. This is normal — the saved config is stale, not necessarily bit-corrupt. If the mismatch is detected during `fetch_config_from_primary()` (the standby's tag ≠ the primary's tag), the primary pushes the correct config and the standby overwrites: ``` WARN("Standby saved config tag=%016"PRIx64" differs from primary " "tag=%016"PRIx64" — saved config may be stale, updating from primary", saved_tag, primary_tag) ``` ### 10.9 Trim crash safety **Condition**: controller crashes during `conf_trim_old_versions()`. **Action**: no recovery needed. Trimming is best-effort — if a crash leaves more than `conf_keep_n` versioned files, the excess files are removed on the next successful config update. Stale `.tmp` files from interrupted writes are also removed during the next trim pass (which runs after every `conf_save_config()` call). There is no boot-time cleanup of orphaned files; the trim function is the sole cleanup mechanism. `conf_trim_old_versions()` always preserves the file named by the current `conf/current` pointer: it reads the pointer file before building the deletion list and excludes that filename regardless of mtime ordering. ### 10.10 Duplicate OP_upgradeConfig during restart window **Condition**: a second `OP_upgradeConfig` arrives while the primary's `cluster_run()` event loop is exiting (between `restart_requested = 1` and `cluster_fini()`). **Action**: iomgr is still accepting connections during this window, so the second upgrade request is dispatched and may write a second config version, update `conf/current`, and set `restart_requested = 1` (already set — a no-op). The write procedure (§3.2) is crash-safe regardless. The first config write's versioned file remains on disk for the trim pass to clean up later. WAL/data consistency is preserved: both configs are valid `ClusterConfig` blobs (validated before save). The process exits cleanly with whichever config `conf/current` names at the time of the exec. No FATAL is needed — concurrent upgrades are benign. On restart via systemd, `conf/current` reflects the last saved config, which is the one the primary will apply in Phase 2. ### 10.11 Concurrent push and fetch on conf/current **Condition**: a standby receives an unsolicited push (§5.2 push direction) from the primary while it is concurrently executing its own retry-fetch thread (§5.5). **Action**: both the push handler and the fetch thread call `conf_save_config()` which ends with `rename(ptr_tmp, ptr_path)`. To avoid temp-file collisions, each call uses a **thread-unique temp filename** based on the caller's pthread ID: ```c snprintf(ptr_tmp, ..., "%s/conf/current.%lu.tmp", effective_dir, (unsigned long)pthread_self()); ``` Each thread writes to its own `.tmp` file, fsyncs it, then renames it to `conf/current`. POSIX `rename()` is atomic — both renames target the same destination (`conf/current`), so whichever completes last wins, and the result is always a valid pointer file. Both writes are for the same config version (the primary sends the same blob in both paths), so either result is correct. Orphaned `.tmp` files from the losing thread are cleaned by the next `conf_trim_old_versions()` pass. The same pattern applies to the versioned `.tmp` files in step 1 of the write procedure (§3.2): each caller appends its thread ID to the temp filename to avoid truncating another thread's in-progress write. > **Implementation note**: `conf_save_config()` must not open any temp file with > `O_EXCL` (which would fail if another thread somehow created the same name). > `O_CREAT|O_TRUNC` is correct for the thread-unique name. ### 10.12 OP_upgradeConfig directed at a standby **Condition**: an admin tool sends `OP_upgradeConfig` to a standby controller's endpoint. **Action**: the standby's `myreq_handler()` checks its HA role before dispatching: ```c case OPCode_upgradeConfig: if (!ha_is_primary(&c->ha)) { send_upgrade_error(ep, req, "Not primary — redirect to primary"); /* Send primary address from ha_state if known */ return -EAGAIN; } return handle_upgrade_config(c, ep, req); ``` The standby logs `WARN` and sends an error reply with the primary's address (if known from the HA watchdog state). The admin tool is expected to retry against the primary. --- ## 11. Service Management and Deployment ### 11.1 systemd unit configuration The controller is managed by a service manager (systemd or equivalent). The design assumes the service manager handles process restart after `cluster_run()` exits (whether from `OP_upgradeConfig` restart or crash recovery). **Certificate provisioning**: The SSL transport layer (§5.3) requires each controller node to have a certificate signed by the shared CA before it can participate in config replication or election probing. Certificate provisioning and lifecycle management (renewal, revocation, CA rotation) are external to this design — they are assumed to be handled by the same configuration management system that distributes `--peers` (§11.2). At minimum, each node needs `certfile` and `keyfile` on disk before the controller starts. A controller whose certificate has expired cannot complete the SSL handshake and will be unable to fetch config or participate in elections — the operational impact is equivalent to a network partition. Operators should monitor certificate expiry and integrate renewal into their standard PKI workflows. **Certificate expiry warning**: At startup (Phase 1, after `create_ssl_context`), the controller checks `X509_get0_notAfter()` on its own certificate. If the certificate expires within `CERT_WARN_DAYS` (compile-time constant, default 30), a WARN is logged: ``` WARN("Controller certificate expires in %d days (%s) — renew before expiry " "to avoid HA election exclusion", days_remaining, asn1_time_str) ``` A controller whose certificate has expired cannot complete the SSL handshake and will be unable to fetch config or participate in elections — the operational impact is equivalent to a network partition. The startup warning gives operators advance notice before the hard failure. A recommended systemd unit file: ```ini [Unit] Description=XCE Controller After=network-online.target Wants=network-online.target StartLimitIntervalSec=60 StartLimitBurst=3 [Service] Type=notify NotifyAccess=all ExecStart=/usr/local/bin/xcecontroller --peers=node0=host0:9990,node1=host1:9990,node2=host2:9990 Restart=always RestartSec=5 # Hardening User=xce NoNewPrivileges=yes ProtectSystem=strict ProtectHome=yes PrivateDevices=yes ReadWritePaths=/var/lib/xce PrivateTmp=yes ProtectKernelTunables=yes ProtectControlGroups=yes RestrictNamespaces=yes SystemCallFilter=@system-service CapabilityBoundingSet=CAP_NET_BIND_SERVICE RestrictAddressFamilies=AF_INET AF_INET6 ProtectHostname=yes LockPersonality=yes [Install] WantedBy=multi-user.target ``` Key points: - `After=network-online.target` ensures network interfaces are up before the controller starts, reducing the probability of connection failures during the initial config fetch. - `RestartSec=5` with `StartLimitBurst=3` bounds restart storms. A longer `RestartSec` (e.g., 15–30s) can be used to stagger controller restarts across nodes in a simultaneous-boot scenario, reducing the probability that all candidates enter election probing at the same time. - `Type=notify` allows the controller to signal readiness to systemd after Phase 2 completes, preventing the service manager from considering the process "started" before it can serve requests. ### 11.2 --peers consistency All controller candidates in a cluster **must** receive the same `--peers` argument. The priority ordering (positional: candidate 0 is highest priority, candidate N-1 is lowest) must be identical across all nodes. Inconsistent `--peers` arguments can cause two nodes to self-declare primary (both with `my_priority == 0`) and never detect each other. **Relationship to provisioning goal**: The §2 provisioning goal ("single provisioning point: only the primary requires the config file") eliminates manual config distribution but does not eliminate manual `--peers` distribution. Every candidate still needs the `--peers` argument provisioned consistently — this is a prerequisite for correct HA election behaviour. The `--peers` list is a bootstrap concern (it tells controllers how to find each other) and cannot be bootstrapped from the config itself (circular dependency: you need `--peers` to know who to fetch config from). Mitigations: (1) the `--peers` list changes far less frequently than cluster config — it only changes when nodes are added or removed from the HA cluster, which is rare compared to queue/resource config changes that drive this design; (2) the future peers-consistency cross-validation in `OP_heartbeat` (§11.2 below) will detect mismatches at runtime; (3) the existing config management tooling already distributes `--peers` for the HA foundation. **Recommended deployment**: distribute the `--peers` argument via a shared configuration management system (Ansible, Puppet, Salt) or a template-generated unit file. This ensures all nodes receive the same candidate list and ordering. The `--peers` argument can alternatively be read from a local config file (`xce.conf` or equivalent) managed by the same configuration system. **Validation**: `xce_parse_peers()` should log a WARN at startup if `my_priority` is not in the range `[0, MAX_CANDIDATES-1]` or if any candidate entry has an empty host or port zero — these indicate a malformed `--peers` string. **Future cross-validation**: A lightweight peers-consistency check in `OP_heartbeat` (exchanging a hash of the local `--peers` string) is deferred to a follow-up release. See `FUTURE_WORK.md`. --- ## 12. Files Changed | File | Change | Complexity | |---|---|---| | `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @20 :UInt64` to `ClusterConfig` (§4.4); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low | | `src/controller/main.c` | Update `conf_file` resolution: read `conf/current` pointer file, then open named versioned file (replaces flat `conf/current_config` path); remove stack-allocated `char conf_file[]` and the `myconf.conffile = conf_file` assignment — `conffile` is now heap-allocated via `asprintf()` in `load_primary_config()` or `fetch_config_from_primary()` and freed in `myconf_fini()`; add `--conf-stale-timeout` and `--conf-keep-n` CLI flag parsing; add `--peers` validation (WARN on empty host or port 0) | Trivial | | `src/include/myconf.h` | Add `config_version_tag` (uint64_t), `config_sha256[32]`, `conf_keep_n`, `config_generation` (uint64_t, §4.4), `conf_stale_timeout` (time_t, default 86400), and `config_stale` (bool) fields to `myconf_t` | Trivial | | `src/include/xce/cluster_def.h` | Add `restart_requested` (atomic_int, for config-change restart), `heartbeat_listener_tid`, `heartbeat_listener_running`, and `lost_and_found_job_count` (atomic_int, incremented during `persist_recover()` when jobs are routed to the synthetic queue, §6.5) fields to `cluster_t` | Trivial | | `src/controller/myconf.c` | Replace `get_default_conf()` with `load_primary_config()` (fail-fast + SHA-256 compute on primary path); config save/trim/validation functions moved to `config_repl.c`; add `myconf_fini()` to free `conffile` (now heap-allocated via `asprintf()` per §3.2), `cluster_config`, `config_version_tag`, and `config_sha256` | Low | | `src/controller/cluster.c` | Split `cluster_init()` into phase 1 and phase 2; gate config load on role; move `iomgr_init` and `endpoint_init` to phase 2; add `create_lost_and_found()` call before `persist_recover`; add `OP_replConfig`, `OP_replConfigPush`, and `OP_upgradeConfig` cases to `myreq_handler()` opcode dispatch; `cluster_run()` event loop checks `c->restart_requested` and exits cleanly when set; `cluster_fini()` joins heartbeat listener thread on any exit path and frees `standby_endpoints` | Medium | | `src/controller/myapp.c` | Restructure `myapp_run()`: call phase 1, then `am_i_primary()`, then config load/fetch, then phase 2. When `cluster_run()` returns cleanly (restart_requested=1), `myapp_run()` returns 0 to main() which exits; the systemd service manager restarts the process. Call `myconf_fini()` in the cleanup path before `cluster_fini()` | Medium | | `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0, §5.3) | Medium | | `src/include/byteorder.h` | **New file**: `be64toh()` fallback macro for non-glibc or pre-2.25 targets (§6.4 portability note). The Meson build checks for `be64toh()` at configure time; if unavailable, this header provides a portable implementation via `__builtin_bswap64()` or `le64toh()`/`ntohll()` | Trivial | | `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources | Trivial | | `src/controller/ha.c` | Replace `am_i_primary()` stub with real election probing; add `probe_thread()`, `wait_any_probe()`; add `start_heartbeat_listener()` / `stop_heartbeat_listener()`; add `xorshift64_next()` (or use shared `ha.h` inline); enhance `ha_check_heartbeat_nocluster()` with priority comparison for simultaneous-boot symmetry breaking (§9.0) | High | | `src/controller/ha.h` | Add `ssl_ctx` field to `ha_state_t` (use `void*` to avoid pulling in `` — cast to `SSL_CTX*` at use sites in election probe and heartbeat listener); add `jitter_seed` (uint64_t, for xorshift64 PRNG) and `xorshift64_next()` inline function to `ha_state_t`; add `zarray_t *standby_endpoints` to `ha_state_t` for tracking connected standby controllers; add `probe_ctx_t` struct; add `election_jitter_max_ms` and `election_priority_step_ms` to `ha_config_t` with defaults `HA_DEFAULT_ELECTION_JITTER_MAX_MS 200` and `HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500` (§9.0 priority-based base delay); update "no new TLS" comment to note SSL transport is sufficient for trusted-LAN model; update `am_i_primary()` declaration; declare `ha_is_primary()` for runtime role check (§10.12) | Low | | `src/controller/join.c` | Update `handle_controller_join()`: remove `-ENOSYS` (controller-controller sessions are valid — config exchange is dispatched from `myreq_handler()`); register connected standby endpoints in `c->ha.standby_endpoints` after successful join handshake; handle endpoint removal on disconnect | Low | | `src/include/typemap.h` | Add `replConfigRequestMsg`, `replConfigReplyMsg`, `replConfigPushRequestMsg`, `replConfigPushReplyMsg`, `upgradeConfigRequestMsg`, `upgradeConfigReplyMsg` union arms to `xce_request_t.body` and `xce_reply_t.body` | Low | | `src/lib/request.c` | Add encode/decode shims for `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply`; `Data` blobs managed by codecgen `encode_*_ptr()` functions | Medium | | `src/include/persist.h` | Add `max_wal_records` field to `persist_config_t` (for WAL bound, see §13); add `wal_record_count` (atomic_uint) and `checkpoint_in_progress` (atomic_int) fields to `persist_t` for the count-based checkpoint trigger (§13.4) | Trivial | **New files**: `src/controller/config_repl.c`, `src/controller/config_repl.h`, and `src/include/byteorder.h` — config save/load/trim, SHA-256 computation, upgrade handler, standby push, and `be64toh()` portability fallback. The config replication logic lives in its own translation unit rather than cluttering `myconf.c`. Note: the existing `src/controller/repl.c` (WAL replication — standby slot management, ACK tracking, catch-up) is a separate module with a similar name but different domain. `repl.c` uses `replicator_t.standbys[]` for per-connection WAL ACK tracking; `config_repl.c` uses `ha_state_t.standby_endpoints` for config push fan-out. See `src/controller/repl.c` (340 lines) for the existing WAL replication module. **Total estimated LOC**: ~510 net new lines across all files (updated from ~480 to account for compile-time gate, `sanitise_cn()`, `config_contains_reserved_name()`, `byteorder.h`, and `CONF_PUSH_TIMEOUT_MS`). Of these, ~170 are the `am_i_primary()` election probing and heartbeat listener infrastructure. This infrastructure is **implemented by this design** (the current stub at `ha.c:213` returns `c->ha.my_priority == 0` without probing). The parent HA design (`persistence-and-ha-design.md` Part 5) specifies the requirements (HA Requirements 3 and 4: election probing and heartbeat listener); this design provides the implementation, which the parent design references. The config replication contribution is ~310 LOC: ~250 for the pull protocol, storage layout, and SHA-256 infrastructure; ~60 for the `OP_upgradeConfig` handler and `push_config_to_standbys()` logic. --- ## 13. WAL Record Bound ### 13.1 Motivation `persist_recover()` runs in Phase 2 on the primary, before iomgr is ready to serve `OP_replConfig`. The standby retry window is ~88 seconds (§5.5). Without a bound on `persist_recover()` time, the standby may exhaust retries and FATAL even though the primary is healthy and making progress. ### 13.2 Mechanism Add `max_wal_records` to `persist_config_t` (default: 10,000). After every `persist_append()` call in `persist.c`, if the active WAL segment exceeds `max_wal_records`, trigger an asynchronous checkpoint: ``` persist_append(wal, entry): append entry to WAL segment if n_records(wal) > max_wal_records: schedule_checkpoint() /* async — queues a vtask */ ``` The checkpoint flushes all in-memory state (jobs, queues, resources) to a fresh snapshot file, truncates the WAL, and updates `snap_ctx` to reference the new snapshot. Subsequent `persist_recover()` calls only replay records since the last completed checkpoint. ### 13.3 Time bound With `max_wal_records = 10000`, each record being a Cap'n Proto decode + in-memory apply, `persist_recover()` replays at most 10,000 records. Benchmarked estimate: ~2-5 seconds on typical HPC controller hardware. The ~88-second retry window provides >10× headroom. ### 13.4 Checkpoint scheduling Checkpoints are scheduled via the `delaymgr` (available in Phase 2 after `delaymgr_init`) rather than `vtaskmgr` directly. Using `delaymgr` avoids saturating the vtaskmgr thread pool with repeated checkpoint requests during burst-append scenarios — `delaymgr` coalesces multiple requests into a single delayed dispatch. **`schedule_checkpoint()` specification**: ```c /* schedule_checkpoint — request an async WAL checkpoint via delaymgr. * * Idempotent: if a checkpoint is already scheduled or in progress, * this is a no-op (guarded by checkpoint_in_progress atomic). * * @param persist The persist_t instance (has access to cluster_t via * persist->cluster for delaymgr and vtaskmgr handles). * * Thread-safety: safe to call from any thread. Uses atomic CAS on * persist->checkpoint_in_progress to ensure only one checkpoint * is queued at a time. */ static void schedule_checkpoint(persist_t *persist) { /* CAS: 0 → 1. If already 1 (in-progress or queued), no-op. */ int expected = 0; if (!atomic_compare_exchange_strong(&persist->checkpoint_in_progress, &expected, 1)) { return; /* already scheduled or running */ } /* Schedule via delaymgr with a short delay to coalesce * burst-append requests into a single checkpoint dispatch. * delay_task() uses time_t (seconds) granularity — see src/lib/delay.c:187. The actual API is delay_task(delaymgr_t*, int period, int runcount, * delay_task_entry_fn_t, void*). */ delay_task(&persist->cluster->delaymgr, 1 /* 1s — coalesce burst appends; delay_task() takes seconds */, 1 /* runcount — fire once */, checkpoint_task_fn, persist); atomic_store(&persist->wal_record_count, 0); /* reset counter after checkpoint */ } /* checkpoint_task_fn — executed by vtaskmgr worker, dispatched by delaymgr. * Returns int per delay_task_entry_fn_t signature. */ static int checkpoint_task_fn(void *arg) { persist_t *persist = (persist_t *)arg; persist_checkpoint(persist); /* existing synchronous checkpoint */ atomic_store(&persist->checkpoint_in_progress, 0); return 0; } ``` The WAL record-count trigger in `persist_wal_sync_append()` and `persist_wal_async()` (the actual WAL write functions in `persist.h:109-111`) calls `schedule_checkpoint()` after appending each record when the record count exceeds `max_wal_records`: ```c /* After appending WAL record: */ if (atomic_fetch_add(&persist->wal_record_count, 1) >= persist->config.max_wal_records) { schedule_checkpoint(persist); } ``` `persist->wal_record_count` is an `atomic_uint` in `persist_t`, reset to 0 after each checkpoint completes. The primary continues serving requests while the checkpoint progresses. Multiple `persist_append()` calls may queue redundant checkpoints — `schedule_checkpoint()` is idempotent via the atomic CAS guard. Records appended to the WAL during an in-progress checkpoint may push `wal_record_count` above `max_wal_records` again before the checkpoint completes. This is intentional — `wal_record_count` is a soft, lossy signal whose sole purpose is to trigger checkpoint scheduling. It is reset to 0 after `persist_checkpoint()` returns, so any records appended during the checkpoint will be counted in the next trigger window. This does not affect correctness: the checkpoint always flushes all in-memory state regardless of the counter value. ### 13.5 Configuration `max_wal_records` is set in `persist_config_t` alongside existing WAL parameters (`wal_max_size`, `snap_keep_n`). Administrators can tune it per workload profile. Larger values mean longer `persist_recover()` but fewer checkpoints (less I/O). Smaller values bound recovery time more tightly at the cost of more frequent snapshot writes. The minimum allowed value is 100 (checked at startup in `persist_init()`); values below this are rejected with a FATAL log to prevent checkpoint storms. `max_wal_records` is a soft trigger for checkpoint scheduling, not a hard limit on record count. After `persist_append()` appends the new entry, if the WAL segment now contains more than `max_wal_records` entries, `schedule_checkpoint()` is called; subsequent appends proceed immediately without waiting. The WAL may temporarily exceed `max_wal_records` during a burst of appends if the asynchronous checkpoint is slow to complete. The parent design (`persistence-and-ha-design.md §2.3`) is the authoritative source for WAL growth controls. This section adds only the record-count-based trigger needed to bound `persist_recover()` time for the standby retry window (§5.5). The parent design provides: `wal_max_size_mb` hard limit (2 GB default) as an emergency trigger, `checkpoint_in_progress` atomic guard, and WAL growth warning at `2 * snap_wal_size_mb` during in-progress checkpoint. These mechanisms are complementary — the size-based limit catches large-record workloads while the count-based limit bounds replay time regardless of record size. --- ## 14. Test Plan ### 14.1 Unit tests (no live cluster required) **Phase 1 tests:** | Test | File | Covers | |---|---|---| | `test_conf_sha256_change_detection` | `test_config_repl.c` | Tag match → no push; tag differ → push | | `test_conf_sha256_full_integrity` | `test_config_repl.c` | Full SHA-256 mismatch → integrity failure → retry | | `test_conf_save_pointer_atomic` | `test_config_repl.c` | Write versioned file + fsync + HEAD-file swap; verify `cat conf/current` names new file after update | | `test_conf_save_crash_after_fsync` | `test_config_repl.c` | Crash after step 1 (versioned file on disk, pointer file not updated) → next boot reads old pointer file | | `test_conf_read_current_filename` | `test_config_repl.c` | `conf_read_current_filename()` parses filename, strips newline; returns -ENOENT when file absent | | `test_conf_zero_tag_sentinel` | `test_config_repl.c` | Standby with no saved config sends tag=0 → primary correctly pushes config (no FATAL for zero-tag sentinel; FATAL only when primary sends zero in reply per §10.5) | | `test_conf_trim_keep_n` | `test_config_repl.c` | After 3 updates with `keep_n=2`, only 2 versioned files remain; stale `.tmp` cleaned | | `test_am_i_primary_no_peers` | `test_ha_role.c` (extend) | Priority-0 with no peers → primary without probing | | `test_am_i_primary_higher_alive` | `test_ha_role.c` (extend) | Mock probe ACK from priority-0 → standby | | `test_am_i_primary_deadline_expires` | `test_ha_role.c` (extend) | All probes time out → primary | | `test_am_i_primary_simultaneous_boot` | `test_ha_role.c` (extend) | Three priorities start simultaneously → transient split-brain resolves; priority-0 is primary after Phase 2 watchdog exchange (§9.0) | | `test_primary_missing_config_fatal` | `test_config_repl.c` | Priority-0 + no conffile → `load_primary_config()` returns -ENOENT | | `test_standby_no_config_no_primary_fatal` | `test_config_repl.c` | Priority-1 + fetch fails + no saved config → fatal error path | | `test_standby_saved_config_no_primary` | `test_config_repl.c` | Priority-1 + fetch fails + saved config exists → WARN + proceed | | `test_conf_stale_timeout` | `test_config_repl.c` | Saved config file mtime older than `conf_stale_timeout` → WARN logged, `config_stale` flag set; saved config within timeout → no WARN, flag clear | | `test_cert_expiry_warning` | `test_ha_role.c` (extend) | Certificate expiring within `CERT_WARN_DAYS` → WARN logged; certificate far in future → no WARN (§11.1) | **Phase 2 tests** (require `cluster_init()` Phase 2 to be complete — iomgr, kvmgr, persist): | Test | File | Covers | |---|---|---| | `test_upgrade_config_sha256_mismatch` | `test_config_repl.c` | `OP_upgradeConfig` with wrong caller SHA-256 → rejected before disk write | | `test_upgrade_config_invalid_capnp` | `test_config_repl.c` | `OP_upgradeConfig` with corrupt blob → Cap'n Proto decode fails → rejected | | `test_upgrade_config_generation_incremented` | `test_config_repl.c` | `OP_upgradeConfig` with any generation value → primary ignores incoming generation, sets `config_generation = current + 1` (§4.4, §6.4 step 3) | | `test_upgrade_config_replay_harmless` | `test_config_repl.c` | Duplicate `OP_upgradeConfig` (same bytes) → accepted again, generation incremented, restart triggered (§4.4) | | `test_upgrade_config_wal_unknown_id` | `test_config_repl.c` | WAL replay with unknown queue ID → job routed to `lost_and_found` queue; unknown resource ID → job resource cleared, set to PEND; unknown host → routed to `lost_and_found` host | | `test_lost_and_found_queue` | `test_config_repl.c` | `persist_recover()` with job referencing removed queue ID → job placed in `lost_and_found` queue; unknown resource ID → job resource cleared, state set to PEND | | `test_lost_and_found_counter` | `test_config_repl.c` | `persist_recover()` routes jobs to `lost_and_found` → `lost_and_found_job_count` incremented; startup WARN summary logged if counter > 0 (§6.5) | ### 14.2 Integration test (requires two controller processes) | Test | Covers | |---|---| | `test_standby_receives_config_on_first_boot` | Fresh standby with no conf/ → receives config → `conf/current` pointer file created, SHA-256 verified | | `test_standby_skips_fetch_on_tag_match` | Standby reconnects with matching tag → `up_to_date=true` → no file I/O | | `test_standby_updates_config_on_tag_mismatch` | Standby reconnects with stale tag → receives new config → pointer file updated → previous version retained | | `test_primary_missing_config_aborts` | Start primary with no conf/ → process exits non-zero | | `test_primary_recomputes_hash_before_send` | Primary's handler double-checks SHA-256 before sending; mismatch detected | | `test_upgrade_config_pushes_to_standby` | Admin sends `OP_upgradeConfig` → primary ACKs → connected standby receives push → both have new tag before primary restarts | | `test_upgrade_config_offline_standby_catches_up` | Admin upgrades config while standby is offline → standby restarts → detects tag mismatch on reconnect → pulls new config | | `test_upgrade_config_no_standbys` | Admin upgrades config on single-node cluster (no standbys) → save succeeds → restart applies new config | ### 14.3 Regression All existing HA tests (`test_ha_cluster.c`, `test_ha_role.c`, `test_ha_redirect.c`, `test_p1_integration.c`) must continue to pass unchanged. These tests manually zero-init `cluster_t` and call `ha_init()` directly, bypassing `cluster_init()` — the phase split does not affect them. ### 14.4 Future Work Tracked in `FUTURE_WORK.md`. Summary: | Test | Covers | |---|---| | `test_peers_cross_validation` | `OP_heartbeat` includes `--peers` hash → mismatch between two candidates → ERROR logged (§11.2 future work) | | `test_lost_and_found_drain` | `xcecli lost-and-found drain` command → reassign or cancel sidelined jobs → `lost_and_found_job_count` reset to 0 (§6.5 observability) | | `test_upgradeconfig_redirect` | `OP_upgradeConfig` sent to standby → error reply with primary address → admin tool retries (§10.12) |