myxce/src/docs/persistence-and-ha-design.md

XCE Controller: State Persistence & High Availability Design

Status: Reviewed Last Updated: 2026-05-07 Target: src/controller/ (C daemon)

---

Table of Contents

1. Part 1: Unused Code Cleanup
2. Part 2: Single-Controller Persistence
3. Part 3: Checkpoint — Dual zarray COW
4. Part 4: Deferred Entity Cleanup at Checkpoint
5. Part 5: HA Failover — LSF-Style Priority Election
6. Part 6: Fan-Out TCP Replication
7. Part 7: Mixed Sync/Async Replication Protocol
8. Part 8: Hook-Driven Architecture
9. Part 9: Worker Connection Strategy
10. Part 10: libxce Smart Discovery
11. Implementation Phases
12. Appendix: Data Structures & Signatures

---

Part 1: Unused Code Cleanup

The controller has no database code. It uses kvmgr (in-memory hash tables) for all state. The following are dead stubs and unused code that should be removed.

1.1 Dead Code in job.c

Location	Code	Action
job.c:18	#define NJOBS (100)	Remove — unused constant
job.c:20	uint64_t jobid = 100;	Remove — unused global
job.c:63-89	load_jobs() body inside #if 0	Remove dead block, keep function signature (will be rewired in Part 2)

1.2 Unused Function in worker.c

Location	Code	Action
worker.c:367-391	restore_worker()	Keep — currently unused but will be the persistence restore entry point (Part 2). The existing restore_worker(void* cls, xce_worker_t host) signature is unchanged. A new adapter function worker_from_capnp() (see P0-12) will bridge Cap'n Proto deserialization to restore_worker() — the recovery path calls worker_from_capnp(), not restore_worker() directly.

1.3 Empty Function in cluster.c

Location	Code	Action
cluster.c:400	cluster_build_groups() — empty body (return 0;)	Keep for now (groups are a future feature), but add a /* TODO: implement group building */ comment

1.4 Stale Frontend References

The Go API server already removed PostgreSQL (lib/pq removed from go.mod). Remaining stale references:

• src/frontend/docs/ — Update any references to lib/pq or PostgreSQL
• src/frontend/apiserver/store/audit_test.go — Already has forbidden-pattern checks for DBConnPool, DBHandle, InitDB, QueryFromDB; keep these as regression guards

---

Part 2: Single-Controller Persistence

2.1 Architecture: WAL + Periodic Checkpoint

                    ┌──────────────┐
  State mutations ──▶│  Hook System │
                    └──────┬───────┘
                           │ AFTER_EVENT
                    ┌──────▼───────┐
                    │  persist.c    │
                    │  (WAL Writer) │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
       ┌──────────┐ ┌──────────┐ ┌──────────────┐
       │ WAL File │ │ Checkpoint│ │ Replication  │
       │ (append) │ │ (COW snap)│ │ (Part 6)     │
       └──────────┘ └──────────┘ └──────────────┘

The design draws from etcd's architecture, simplified for XCE's single-writer model:

etcd Concept	XCE Adaptation	Why Simplified
WAL (append-only log)	Adopted directly	Proven durability pattern
Chained CRC	Adopted (links each record to previous)	Detects torn/truncated writes
Pre-allocated segments	Adopted (64 MB segments, rotate)	Avoids filesystem allocation latency
8-byte alignment	Adopted	Direct I/O compatibility
BoltDB snapshot	Not adopted — use dual zarray COW	Application-level COW avoids fork() pitfalls
MVCC key-value store	Not adopted	Single-writer, no concurrent readers needing versioning
Raft consensus	Not adopted — LSF-style priority (Part 5)	~300 LOC vs ~5000 LOC

2.2 WAL Entry Format

/* WAL entry header — 40 bytes, 8-byte aligned */
typedef struct {
    uint64_t term;          /* Election term (incremented on failover) */
    uint64_t lsn;           /* Log sequence number within this term */
    uint64_t timestamp;     /* Epoch microseconds */
    uint32_t type;          /* wal_entry_type enum */
    uint32_t payload_len;   /* Bytes following this header */
    uint32_t crc;           /* CRC32 of (header + payload) */
    uint32_t prev_crc;      /* Previous record's CRC (chained); WAL_CRC_NONE (0xFFFFFFFF) for first record in segment */
} wal_entry_hdr_t;
static_assert(sizeof(wal_entry_hdr_t) == 40, "wal_entry_hdr_t must be 40 bytes for 8-byte alignment and on-disk format stability");

/* Sentinel value for prev_crc: no predecessor (first record in segment).
 * 0xFFFFFFFF is used instead of 0 because CRC32 can legitimately produce 0,
 * which would be indistinguishable from the sentinel. */
#define WAL_CRC_NONE 0xFFFFFFFF

/* Bytes covered by CRC in the header — all fields except crc and prev_crc.
 * Used for both on-disk WAL validation and replication CRC verification. */
#define WAL_HDR_CRC_SIZE 32  /* sizeof(wal_entry_hdr_t) - 2*sizeof(uint32_t) */

2.3 WAL Entry Types

enum wal_entry_type {
    /* Job lifecycle — the high-frequency path */
    WAL_JOB_SUBMIT    = 1,  /* SYNC replication */
    WAL_JOB_START     = 2,  /* ASYNC */
    WAL_JOB_FINISH    = 3,  /* ASYNC */
    WAL_JOB_FAIL      = 4,  /* ASYNC */
    WAL_JOB_STOP      = 5,  /* ASYNC */
    WAL_JOB_RESUME    = 6,  /* ASYNC */
    WAL_JOB_UPDATE    = 7,  /* ASYNC */
    WAL_JOB_RUSAGE    = 8,  /* ASYNC */
    WAL_JOB_CKPT      = 9,  /* ASYNC */
    WAL_JOB_PREASON   = 10, /* ASYNC */

    /* Queue and host control — infrequent */
    WAL_QUEUE_CTRL    = 20, /* ASYNC */
    WAL_WORKER_CTRL   = 30, /* ASYNC */

    /* Marker */
    WAL_CHECKPOINT    = 100 /* Marker — no payload */
};

Note: There are no WAL_JOB_DELETE, WAL_WORKER_DELETE, or WAL_QUEUE_DELETE entry types. Entity cleanup is deferred to checkpoint (see Part 4). This means WAL never contains delete records — only creates and updates — simplifying replay logic.

WAL_CHECKPOINT replication skip — code boundary: WAL_CHECKPOINT (type=100) is written to the local WAL only via persist_write_checkpoint_marker() — a dedicated function in persist.c that calls pwrite()/fsync() directly and never calls repl_broadcast(). By contrast, persist_wal_async() and persist_wal_sync_append() call repl_broadcast() after writing. The invariant persist_wal_async(WAL_CHECKPOINT, ...) MUST NOT be called — enforce with assert(type != WAL_CHECKPOINT) at the top of both persist_wal_async() and persist_wal_sync_append() in debug builds. This makes the code-path boundary explicit rather than relying on callers to never pass WAL_CHECKPOINT to the replication path. The last_replicated_crc field in persist_t (distinct from last_crc which covers all entries including checkpoints) is updated only by the replication send paths, never by persist_write_checkpoint_marker() — this is the mechanism that maintains a continuous CRC chain for standbys that skip checkpoint entries. The standby's mirror of this field is ha_state_t.last_received_crc (Appendix A.4), which tracks the CRC of the last replicated entry the standby received; together these two fields implement the chain validation described in Part 6.4.

Mapping to existing hook detail enums:

Hook Detail (cluster_def.h)	WAL Entry Type	Replication	Notes
HOOK_DETAIL_JOB_NEW	WAL_JOB_SUBMIT	SYNC
HOOK_DETAIL_JOB_START	WAL_JOB_START	ASYNC
HOOK_DETAIL_JOB_EXIT	WAL_JOB_FINISH	ASYNC
HOOK_DETAIL_JOB_FAIL	WAL_JOB_FAIL	ASYNC
HOOK_DETAIL_JOB_STOP	WAL_JOB_STOP	ASYNC	Not in current enum — must be added (see footnote 1)
HOOK_DETAIL_JOB_RESUME	WAL_JOB_RESUME	ASYNC	Not in current enum — must be added (see footnote 2)
HOOK_DETAIL_JOB_UPDATE	WAL_JOB_UPDATE	ASYNC
HOOK_DETAIL_JOB_CKPT	WAL_JOB_CKPT	ASYNC
HOOK_DETAIL_JOB_RUSAGE_UPDATE	WAL_JOB_RUSAGE	ASYNC
HOOK_DETAIL_SETPREASON	WAL_JOB_PREASON	ASYNC	Both map to same type — see footnote 3
HOOK_DETAIL_CLRPREASON	WAL_JOB_PREASON	ASYNC	Both map to same type — see footnote 3
HOOK_DETAIL_QUEUE_UPDATE	WAL_QUEUE_CTRL	ASYNC
HOOK_DETAIL_WORKER_JOIN	WAL_WORKER_CTRL	ASYNC
HOOK_DETAIL_WORKER_UPDATE	WAL_WORKER_CTRL	ASYNC
HOOK_DETAIL_WORKER_DEAD	WAL_WORKER_CTRL	ASYNC

Footnotes:

1. HOOK_DETAIL_JOB_STOP: Does not exist in the current cluster_def.h enum. It must be added as a prerequisite for P0-6. Codebase reality: on_job_stop() currently calls mark_job_update() which fires HOOK_DETAIL_JOB_UPDATE — it does NOT fire HOOK_DETAIL_JOB_EXIT. The implementation in P0-4a must update on_job_stop() to call xce_run_hooks directly with HOOK_DETAIL_JOB_STOP, bypassing mark_job_update() for this event. Search grep mark_job_update src/controller/job.c to find the current call sites.

2. HOOK_DETAIL_JOB_RESUME: Does not exist in the current cluster_def.h enum. It must be added as a prerequisite for P0-6. The existing on_job_resume() function in job.c must be updated to call hooks with this new detail.

3. Pending reason deduplication: Both HOOK_DETAIL_SETPREASON and HOOK_DETAIL_CLRPREASON map to WAL_JOB_PREASON without payload differentiation. This is acceptable because pending reasons are transient — after a controller restart or standby promotion, the scheduler re-evaluates pending jobs and publishes fresh pending reasons when scheduling resumes. On recovery, jobs in JOB_STAT_PEND state are assigned a default pending reason ("Job is waiting to be scheduled") until the scheduler updates them. The critical invariant is that the job's pend/run/exit state is preserved, not the specific pending reason text. Ambiguity logging: When apply_wal_entry() processes a WAL_JOB_PREASON entry during WAL replay, it logs at DEBUG level: "WAL_JOB_PREASON replay for jobid %lu — pending reason text is not preserved across recovery" to make the ambiguity visible in logs rather than silent.

WAL payload serialization format: Each WAL entry's payload is serialized as Cap'n Proto using the controller.capnp schemas. The mapping from entry type to schema is:

WAL Entry Type	Cap'n Proto Schema	Key fields serialized
WAL_JOB_SUBMIT	Job	All fields: jobid, name, queue, num_proc, command, environment, submit_time, stat, resource limits
WAL_JOB_START	Job (partial)	jobid, stat, start_time, exec_host
WAL_JOB_FINISH	Job (partial)	jobid, stat, end_time, exit_code
WAL_JOB_FAIL	Job (partial)	jobid, stat, end_time, fail_reason
WAL_JOB_STOP	Job (partial)	jobid, stat
WAL_JOB_RESUME	Job (partial)	jobid, stat
WAL_JOB_UPDATE	Job (partial)	jobid, changed fields only
WAL_JOB_CKPT	Job (partial)	jobid, ckpt_time
WAL_JOB_RUSAGE	Job (partial)	jobid, rusage fields
WAL_JOB_PREASON	Job (partial)	jobid, preason
WAL_QUEUE_CTRL	QueueConf	All fields for the affected queue
WAL_WORKER_CTRL	Worker	All fields for the affected worker
WAL_CHECKPOINT	(none)	No payload — payload_len = 0

"Partial" payloads serialize only changed fields using Cap'n Proto's standard optional field encoding — unset fields are omitted. apply_wal_entry() applies partial payloads by merging into the existing in-memory object (not replacing it). Merge semantics (CRITICAL): For partial payloads, apply_wal_entry() sets only the fields present in the Cap'n Proto payload. Fields that are not set (absent/optional in the Cap'n Proto message) are not touched in the in-memory object — they retain their previous value. This is naturally supported by Cap'n Proto's optional field encoding: a field that was never set in the message has has<Foo>() == false, and the merge logic skips it. Implementers must NOT use a "replace entire struct" approach, which would zero out absent fields and corrupt in-memory state. The merge rule is: if (payload.has<Field>()) in_memory->Field = payload.get<Field>(); for each optional field.

2.4 WAL Segment Management

effective_dir/data/wal/
├── seg-0000000000000000.wal   # Pre-allocated 64 MB
├── seg-0000000000000064.wal   # LSN offset in filename
└── ...

• Segments are pre-allocated to 64 MB (configurable)
• When a segment is full, rotate to the next
• fsync() at configurable interval (wal_sync_ms, default 10ms) — batched, not per-write. If fsync() fails, persist_wal_sync_append() and persist_wal_async() return an error. For sync replication, the failed write is not sent to standbys and the client receives an error reply. For async replication, the error is logged and the controller continues operating — the WAL entry is in the OS page cache but not on disk, so it may be lost on crash. Monitoring should alert on persistent fsync failures.

  Implementation note: delaymgr uses sleep(precision) with precision=2 seconds (see cluster.c:604), giving 2-second minimum granularity — far too coarse for wal_sync_ms=10. The WAL fsync timer must use a dedicated pthread with nanosleep() or a timerfd_create(CLOCK_MONOTONIC) loop, not delaymgr. The periodic checkpoint timer (P0-3) can still use delaymgr since snap_interval_s is measured in seconds.

• WAL trim guard: WAL segments are only deleted up to min(acked_lsn) across all connected standbys that are not in catch-up mode (standby_conn_t.acked_lsn — updated on each OP_replAck received, but only from standbys with in_catch_up == 0). For catching-up standbys, use their OP_replJoin.applied_lsn (the LSN they reported on reconnect) as the floor — WAL segments from that LSN forward must be retained until the standby sends OP_replCatchUpDone. In-catch-up standbys have stale acked_lsn values (from before their catch-up started); using stale acked_lsn as the trim point could trim entries the standby is about to request, forcing snapshot transfer when only a short WAL replay was needed. Concretely: trim_lsn = min(min_non_catchup(acked_lsn), min_catchup(join_applied_lsn), checkpoint_lsn). This prevents trimming segments that a lagging standby still needs for catch-up. When a standby reconnects after a gap, it reports its last applied LSN, and the primary streams missing entries from that point. If the standby's LSN is older than the primary's oldest surviving WAL segment, a full snapshot transfer is required (see Part 6 standby catch-up). Each standby heartbeat includes the standby's applied_lsn so the primary always knows the safe trim point. Empty-standby fallback: When no standbys are connected (replicator.nstandbys == 0), use the primary's own checkpoint_lsn as the trim point — WAL segments before the latest confirmed checkpoint can be safely deleted. Standby disconnect trim delay: When a standby disconnects, the primary should delay WAL trimming by one checkpoint interval before falling back to checkpoint_lsn-based trimming. This gives the standby a window to reconnect and resume catch-up without requiring a full snapshot transfer. The implementation uses per-slot tracking in replicator_t.last_disconnected_lsn[] and replicator_t.last_disconnect_time[] (see Appendix A.3). When a standby disconnects, its acked_lsn and current time are recorded in its slot. The trim point is computed as min(min_over_disconnected_slots(last_disconnected_lsn[i]), checkpoint_lsn) for one checkpoint interval after each disconnect, then falls back to checkpoint_lsn per slot. Slot expiry (CRITICAL): A disconnected slot's last_disconnected_lsn[i] hold on the trim guard expires exactly when now >= last_disconnect_time[i] + snap_interval_s. At that point, last_disconnected_lsn[i] is reset to 0 and last_disconnect_time[i] is cleared, and that slot no longer contributes to the trim guard min(). Permanent decommission: When a standby is removed from the --peers candidate list (its slot index is no longer valid), its last_disconnected_lsn[i] and last_disconnect_time[i] must be zeroed immediately in replicator_remove_candidate(). Without explicit slot invalidation, a decommissioned standby's disconnect record holds the WAL trim guard indefinitely — a WAL growth leak with no upper bound other than wal_max_size_mb.
• WAL growth bound during standby disconnect: The worst-case WAL accumulation when no standbys are connected is: wal_growth_MB ≈ snap_wal_size_mb + (submit_rate_per_sec * avg_WAL_submit_KB * snap_interval_s * 2 / 1024). The * 2 accounts for the one-interval trim delay before checkpoint_lsn-based trimming takes effect. At default settings (snap_interval_s = 60, snap_wal_size_mb = 64) with 10 submits/sec and ~1 KB average WAL_JOB_SUBMIT payload, worst-case growth is ≈65 MB — well within a single segment. At pathological rates (1000 submits/sec, 4 KB average payload), worst-case growth is ≈535 MB, requiring ~9 segments. Mitigation: If WAL disk usage exceeds wal_max_size_mb (default: 2 GB), log an ERROR and trigger an immediate checkpoint to advance checkpoint_lsn and enable trimming. The one-interval trim delay is skipped when replicator.nstandbys == 0 and WAL usage exceeds wal_max_size_mb / 2 — in this scenario, preserving disk space takes priority over the reconnection catch-up window.
• Segments before the latest snapshot are deleted during recovery, subject to the trim guard above
• WAL integrity model (explicit boundary): CRC32 (chained) detects accidental corruption — bit flips, torn writes, truncated segments. It is NOT tamper-resistant: an attacker with write access to the WAL directory can modify payload bytes and recompute a valid CRC32. Tamper resistance would require HMAC-SHA256 keyed on a controller-local secret. XCE's integrity model relies on OS-level filesystem permissions (mode 0700, controller-UID-owned) as the access control boundary, not on cryptographic authentication. Implementers must not assume WAL CRC provides any security guarantee beyond accidental-corruption detection. If tamper resistance is required for a deployment, use encrypted storage (LUKS/dm-crypt) which protects both confidentiality and integrity.
• Chained CRC detects corruption: if prev_crc doesn't match previous record's CRC, stop replay at last valid entry. The first record in a WAL segment uses prev_crc = WAL_CRC_NONE (defined as 0xFFFFFFFF, not 0 — CRC32 can legitimately produce 0 for certain inputs, which would be indistinguishable from the "no predecessor" sentinel if 0 were used). Recovery validates by checking that each record's prev_crc matches the previous record's crc field, starting from WAL_CRC_NONE for the first record. Segment boundary handling: When recovery encounters prev_crc == WAL_CRC_NONE at a position that is NOT the absolute first record in the file (i.e., at the start of a rotated segment), this is treated as a valid chain break — the CRC chain restarts from this point. The recovery algorithm must NOT treat WAL_CRC_NONE at any record position that is 8-byte aligned and at the beginning of a segment file as corruption. Each segment file is independently CRC-chained from its first record. Writer-side obligation (ADV-11): persist_wal_sync_append() and persist_wal_async() MUST set hdr.prev_crc = WAL_CRC_NONE when writing the first record of a new segment (i.e., when persist_t.seg_offset == 0 after segment rotation). The implementation MUST NOT carry persist_t.last_crc across a segment boundary — the new segment starts a fresh chain. This is the mirror of the recovery-side rule; without it, recovery would see a non-WAL_CRC_NONE prev_crc at a segment boundary and enter 8-byte linear scan mode unnecessarily.
• Truncated segment recovery: If a crash occurs mid-write, the last WAL segment may contain a partial record. The recovery algorithm handles two cases:
  1. Valid header CRC, truncated payload: Read the full 40-byte header, validate hdr_crc against WAL_HDR_CRC_SIZE bytes. If the header CRC is valid but payload_len exceeds the remaining bytes in the segment, truncate at the previous valid record. Do NOT use the declared payload_len to advance the read position — a partially-written payload_len could be corrupted, causing the scanner to jump past valid records.
  2. Invalid header CRC (corrupted header): Enter 8-byte linear scan mode — skip forward 8 bytes at a time, checking for a valid hdr_crc and matching prev_crc at each alignment boundary. This ensures that a corrupted header cannot cause the scanner to skip past valid data.
• WAL trim and total_wal_bytes accounting: After a successful checkpoint, the WAL trim step deletes segment files whose last LSN is ≤ checkpoint_lsn. On each segment deletion, total_wal_bytes is decremented by the segment file size (fstat(fd).st_size). On each WAL write, total_wal_bytes is incremented by the bytes written. On segment rotation, the new segment file size is not counted until bytes are written to it. This ensures total_wal_bytes accurately reflects the on-disk WAL size at all times. In both cases, recovery stops at the end of the WAL segment (the segment file size) when it cannot find a valid record. The seg_offset field in persist_t tracks the last fsync point at runtime but is in-memory and lost on crash — recovery must NOT rely on it as the trust boundary. Instead, recovery validates entries up to the physical end of the segment file using chained CRC: records with invalid CRC or prev_crc mismatch are treated as corruption. The recovery algorithm scans from the beginning of the segment and stops when it encounters an unrecoverable CRC failure at or near the end of the file. seg_offset is only used during normal operation to track how far the fsync thread has flushed — it is updated to the post-write position only after fsync() succeeds, ensuring that on crash + restart (before the next fsync), the runtime can still distinguish flushed from unflushed data if needed.

Data at rest: WAL segments and snapshot files contain job submission payloads (which may include environment variables, command lines, and user-specified data). If the storage directory is on a shared or compromised filesystem, this data is accessible. XCE relies on OS-level filesystem permissions (effective_dir/data/wal and effective_dir/data/snap directories should be owner-read-only, mode 0700, where effective_dir is derived from --workdir and -n name CLI flags) and physical host security rather than encryption at rest. If encryption at rest is required, it should be provided by the underlying filesystem (e.g., LUKS, encrypted EBS volumes) rather than by the controller — adding application-level encryption would double the encryption cost with filesystem-level encryption already in place for most production deployments.

Data retention and scrubbing: Sensitive data (environment variables, job command lines) persists in WAL files until those segments are trimmed (after checkpoint), and in snapshot files until the next checkpoint replaces them. The retention window is bounded by finished_retain_s (default 1h) for job payloads: once a job is GC'd at checkpoint, its data no longer appears in new snapshots, and its WAL segment is trimmed after the checkpoint that removed it. Old WAL segments and snapshot files are deleted with unlink() — no explicit overwrite/scrubbing of file data is performed. If the storage device requires explicit data destruction (e.g., SSDs with wear leveling where unlink does not guarantee physical erasure), this must be handled at the storage layer (encrypted volumes with key rotation, or secure erase tooling). XCE does not implement application-level data scrubbing of WAL or snapshot files.

WAL payload logging policy: The persistence and replication modules must never log WAL payload content. Only entry metadata (type, LSN, term, payload_len, CRC) may appear in log messages. This is a security hardening invariant: WAL_JOB_SUBMIT payloads contain job environment variables which frequently hold API keys, database passwords, and other credentials. All logging call sites in persist.c and repl.c that reference WAL entries must log only hdr->type, hdr->lsn, hdr->term, hdr->payload_len, and CRC values — never the payload pointer or its contents. Code review must verify this as a gating check before merge. The CRC mismatch error path already follows this pattern: ERROR("replication CRC mismatch: expected %u, got %u, lsn %lu", hdr.crc, computed_crc, hdr.lsn). Structural enforcement (required — P0-1 deliverable): Define a persist_log_entry_metadata(const wal_entry_hdr_t* hdr) helper in persist.h that accepts only the header struct and logs type, lsn, term, payload_len, and CRC — it has no payload parameter and cannot log payload content. All log call sites in persist.c and repl.c that emit WAL entry information must use this helper. This converts credential-logging from a review-time catch to a build-time enforcement: a developer cannot format-print a payload pointer without explicitly calling an unwrap function that clearly signals intent. Relying solely on code review is a human control that will eventually fail given how frequently debug log statements are added during development. Add persist_log_entry_metadata() to the P0-1 deliverable table.

Filesystem type requirement: effective_dir/data/wal and effective_dir/data/snap must be on a local filesystem (ext4, xfs, etc.), not on NFS or other network filesystems. (where effective_dir is derived from --workdir and -n name CLI flags) The fsync() durability guarantee depends on the OS writing to stable storage — on NFS, fsync() may only flush to the server's cache, not to persistent storage, undermining the WAL's crash-recovery guarantee. Run a startup check using statfs() and reject f_type values matching NFS_SUPER_MAGIC (0x6969) or SMB_SUPER_MAGIC (0x517B) — log ERROR and abort if detected.

2.5 Recovery Sequence

On startup, cluster_init() calls persist_recover():

1. Find latest snapshot: Scan effective_dir/data/snap/ for the snapshot with the highest LSN
2. Load snapshot: Read the snapshot file into memory 2a. Verify snapshot checksums: (i) Recompute CRC32 over the header fields (lsn, timestamp, term, version, nJobs, nQueues, nWorkers) and compare against SnapshotHeader.headerChecksum. If mismatch, the header has been tampered or corrupted — log an error and try the next-oldest snapshot. (ii) Recompute CRC32 over the Cap'n Proto payload and compare against SnapshotHeader.checksum. If mismatch, log an error and try the next-oldest snapshot. If all snapshots fail verification and the WAL contains entries that predate any valid snapshot (i.e., WAL segments have already been trimmed), the controller must abort with a fatal error — silently booting with empty state would silently discard live job/worker state and is data loss. The fatal error message must include: the number of corrupt snapshots found, the LSN range of surviving WAL segments, and the instruction "Contact support or restore from backup before restarting". Do not fall back to empty state in this scenario. The empty-state fallback is only valid on first boot (see below). Snapshot version check: After verifying both checksums, check SnapshotHeader.version. If the version is higher than the current controller's WAL_FORMAT_VERSION (defined in persist.h as 1), log a fatal error and abort — a snapshot written by a newer controller cannot be safely loaded by an older one. This catches version mismatches earlier than encountering unknown WAL entry types during replay. Fallback WAL verification: When falling back from the latest snapshot to an older snapshot (due to checksum failure), the recovery code must first verify that WAL segments exist from older_snapshot.lsn + 1 forward. If WAL segments have already been trimmed past the older snapshot's LSN (i.e., only segments covering [latest_snapshot.lsn, ...] exist), the fallback cannot replay the gap and the controller must abort with a fatal error. Do NOT silently accept an older snapshot with no WAL to replay — this would start the controller with stale state, silently discarding all changes between the older snapshot and the trimmed WAL. 2b. Deserialize: Deserialize Cap'n Proto data, rebuild kvmgr instances. Set capnp::ReaderOptions.traversalLimitInWords to a fixed upper bound (MAX_SNAPSHOT_BYTES / 8 words) to prevent O(n²) traversal amplification from a crafted snapshot — this is the same hardening practice used for WAL payload deserialization (Part 6.4). Do NOT set it proportional to the actual file size, as a small file with deeply nested structure can still trigger excessive traversal. The limit should be 64 * 1024 * 1024 / 8 = 8M words (the 64 MB is the expected max snapshot size). If the snapshot file is larger than MAX_SNAPSHOT_BYTES (default: 256 MB — 4× the expected 64 MB for 10K jobs × 1 KB + overhead), log an ERROR and reject the snapshot.
   • Jobs → makeJob(c->jobmgr, ...) (reuse existing create_job())
   • Workers → restore_worker(cls, host) (already exists in worker.c:367)
   • Queues → create_queue(c, qconf) (already exists)
   • Resources → makeResource(c->resmgr, ...) (already exists)
   • Configs → add_config(c, key, val) (already exists)
3. Replay WAL: Starting from the snapshot's LSN, read and apply WAL entries forward via apply_wal_entry(). Hooks are not yet registered during recovery — persistence hooks are registered only after persist_recover() completes in cluster_init(), so no bypass flag is needed. This avoids re-persisting (WAL write during recovery) and prevents replication sends during replay. For WAL_JOB_SUBMIT entries, replay calls restore_job(c, jobid, payload) — a dedicated function that inserts the job at its original jobid without allocating a new one. Calling job_submit() here would assign a new jobid, breaking replay by creating duplicate or wrong-ID jobs. Job ID allocation note: The controller does NOT allocate jobids — job IDs are assigned by the Go API server via idgen (clib.go) before the submit request reaches the controller. Recovery merely restores jobs at their original IDs from the WAL; no idgen re-seeding is required in the controller. Recovery is single-threaded: persist_recover() runs before cluster_init() starts the iomgr, vcpu pool, or any other threads — all restore_job()/xkvmgr_add() calls during recovery are serialized and require no additional locking beyond what the functions already provide. Worker behavior during recovery: iomgr is not yet listening during recovery, so workers attempting to connect receive "connection refused". libxce's xce_reconnect() backoff+jitter loop handles this transparently — no protocol-level "not ready" response is needed. Workers connect normally once iomgr starts after recovery completes.
4. Trim WAL: Delete WAL segments fully covered by the loaded snapshot
5. Retention pass: After replay, run the retention/GC check on all entities (equivalent to a thaw pass) to clean up any entities whose TTL has expired during the controller's downtime. This handles the case where a controller was down for longer than finished_retain_s or worker_retain_s — expired entities are cleaned immediately on recovery.
6. Restore election term: Set ha.term = max(snapshot_header.term, max_term_from_replayed_wal). This is a security-critical invariant: a controller must never start with a term lower than any term it has previously used, as this would undermine split-brain fencing. On first boot (empty state), ha.term is initialized to 0. P0 scope note: ha_state_t and its term field are defined in Part 5.3 (HA design), which is implemented in P1. P0 implementers must nonetheless allocate a uint64_t placeholder for the term in the snapshot header (SnapshotHeader.term) and in WAL entry headers (wal_entry_hdr_t.term) — these fields exist in the serialized format even before HA is wired up. During P0 (single-controller), ha.term is always 0; the restore step is a no-op. The field is read and written correctly in P0 so that P1 can wire ha_state_t.term to it without format changes. Snapshot version backward compatibility: If SnapshotHeader.version is lower than the current WAL_FORMAT_VERSION, this indicates a snapshot written by an older controller. New schema fields must be optional with defaults to ensure backward compatibility. If the schema has breaking changes between versions, a migration function (e.g., migrate_snapshot_v1_to_v2()) must be written before bumping WAL_FORMAT_VERSION. Do not accept older versions without verifying field compatibility — the default behavior of loading an older snapshot with missing fields is safe only if all new fields are optional with sensible defaults.

Forward compatibility warning — unknown WAL entry types during recovery: apply_wal_entry() logs a WARN and returns 0 (skips) unrecognized wal_entry_type values. This allows a controller running an older version to recover from WAL segments written by a newer version — the old controller boots, skips entries it cannot interpret, and continues operating. However, the state changes intended by those skipped entries are silently lost. This is a downgrade risk: if a newer controller writes entry types the old controller does not understand, and the old controller is started on that WAL, the cluster state will be incomplete (missing features, incorrect job/worker state for data written by the newer controller). Deployments should never roll back to an older controller version after a newer version has written persistent state. If rollback is required, the only safe procedure is: drain all jobs, stop all controllers, delete WAL and snapshot directories on all candidates, then start the old controller in empty-state-first-boot mode.

WAL entry type version guard: To convert silent data loss into a loud failure, persist_recover() tracks the maximum wal_entry_type value encountered during WAL replay. If any entry has type > WAL_ENTRY_TYPE_MAX (defined in persist.h as the highest known entry type in the current version — initially WAL_CHECKPOINT = 100), persist_recover() logs a FATAL error and aborts: "WAL contains entry type %u beyond current version's maximum %u — controller version is too old to safely recover; upgrade to a newer version or drain and reset WAL". This mirrors the snapshot version check pattern (step 2a) and prevents an older controller from silently booting with incomplete state. Operators who genuinely need to force-boot against a newer WAL (e.g., emergency recovery) can use the --force-wal-version startup flag, which downgrades the abort to a WARN per unknown entry type.

Empty state (first boot after upgrade): If both effective_dir/data/snap/ and effective_dir/data/wal/ are empty (no snapshots, no WAL segments), persist_recover() returns 0 (success) with no state loaded — equivalent to today's in-memory-only boot. This ensures backward compatibility: existing deployments upgrading from in-memory-only operation need no migration steps. The controller simply starts with empty state and begins accumulating WAL entries.

Valid snapshot, empty WAL (clean shutdown): If effective_dir/data/snap/ contains a valid snapshot file but effective_dir/data/wal/ has no segments, persist_recover() loads the snapshot with no WAL replay. This is the expected state after a clean shutdown where the last checkpoint has already trimmed all WAL segments. No error or warning is generated.

Last-known-good snapshot retention: When a new checkpoint completes successfully, the controller retains the previous snap_keep_n - 1 snapshots on disk (default snap_keep_n = 2, i.e., current + one previous). When this fallback works: If the controller crashes mid-checkpoint (before the new snapshot is fsync'd and the WAL is trimmed), the previous snapshot and its follow-on WAL segments are intact — recovery falls back to the previous snapshot and replays the bridging WAL. When this fallback does NOT work: If a checkpoint completed successfully (new snapshot fsynced, WAL trimmed up to new checkpoint_lsn), and the resulting snapshot is corrupt (e.g., silent disk error), the WAL entries covering the previous→new snapshot gap are already trimmed and unavailable. In this case the fallback fails at the WAL-existence check (see step 2a) and the controller aborts with a fatal error — snap_keep_n provides no protection against completed-checkpoint corruption. The fallback chain walks snapshots from newest to oldest, stopping at the first valid one where WAL covers the gap. Snapshot rotation: after writing a new snapshot, delete snapshots beyond snap_keep_n, oldest first.

2.6 Configuration

typedef struct {
    /* Paths are derived from --workdir and -n name CLI flags at startup.
     * Not stored as config fields. Derivation rules (name is mandatory):
     *   effective_dir = workdir + "/" + name
     *   wal_path      = effective_dir + "/data/wal"
     *   snap_path     = effective_dir + "/data/snap"
     *   cluster_cfg   = effective_dir + "/conf/current/cluster.bin"
     *   pid_file      = effective_dir + "/run/controller.pid"
     */
    uint32_t wal_sync_ms;      /* fsync interval (default 10) */
    uint32_t snap_interval_s;  /* checkpoint interval (default 60) */
    uint32_t snap_wal_size_mb; /* checkpoint when WAL exceeds this (default 64) */
    uint32_t snap_keep_n;      /* number of snapshots to retain (default 2 — current + one previous) */
    uint32_t transient_job_timeout_s; /* seconds after failover before JOB_STAT_TRANSIENT jobs are
                                         auto-terminated (default 300 = worker_reconnect_timeout_s).
                                         0 = no auto-termination (operator must intervene manually).
                                         After timeout: job → JOB_STAT_EXIT with reason
                                         "worker did not reconnect after failover"; worker →
                                         WORKER_UNAVAILABLE. Increments ctrl_transient_jobs_stuck
                                         metric for each auto-terminated job. */
    uint32_t finished_retain_s;/* retain finished jobs for N sec (default 3600) */
    uint32_t worker_retain_s;  /* retain unavailable workers for N sec (default 86400) */
    uint32_t sync_timeout_ms;  /* sync replication ACK timeout (default 50) */
} persist_config_t;

Working Directory Layout (unified for dev and production):

${XCE_WORKDIR}/                    e.g. /var/lib/xce (prod) or /tmp/xce (dev)
├── ctrl-1/                        effective_dir for -n ctrl-1
│   ├── conf/
│   │   ├── current/
│   │   │   └── cluster.bin        controller-specific config (WAL params, HA timeouts)
│   │   └── previous/
│   │       └── cluster.bin        rollback target (atomic rename)
│   ├── data/
│   │   ├── wal/                   mode 0700
│   │   └── snap/                  mode 0700
│   └── run/
│       └── controller.pid
│
├── ctrl-2/                        effective_dir for -n ctrl-2
│   └── ...
│
└── ctrl-3/                        effective_dir for -n ctrl-3
    └── ...

Controller startup:

xce_controller \
  --workdir /var/lib/xce \
  -n ctrl-1 \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000

The -n flag is mandatory. Candidate list is passed via --peers CLI flag (no config file for peers). cluster.bin contains only controller-specific config (WAL/HA parameters), not the candidate list.

Added to cluster_config_t in the Cap'n Proto schema and myconf.c YAML parsing.

2.7 Controller Connection Context

The controller only handles connections from three peer types:

1. Workers — compute node daemons reporting job status and receiving dispatch
2. Scheduler — job placement service
3. API servers — Go REST frontend (1–3 instances)

End users never connect directly to the controller — the API server provides traffic control, rate limiting, and request validation. This has important implications:

• Sync replication: Only triggered by job submit from API servers. At 1–3 API server connections, vcpu pool exhaustion from blocking repl_wait_ack() is not a concern — the wait is bounded by sync_timeout_ms (default 50ms) and only affects the vcpu handling that specific API server's request.
• Connection count: Workers (100s–1000s) + scheduler (1–2) + API servers (1–3). The scheduler and API server connections are long-lived and stable.
• All connections use SSL: The controller's iomgr enforces SSL on all connections (see netio.c). Replication channels must also use this pipeline, not raw TCP.

---

Part 3: Checkpoint — Dual zarray COW

3.1 Why Not fork()?

The original design considered fork() + COW for checkpoint snapshots. This is problematic for a multithreaded controller:

• After fork(), only the calling thread survives. If another thread held a mutex (e.g., the WAL writer held persist.lock), the child inherits a locked mutex with no owner — instant deadlock.
• kvmgr's internal my_rwlock_t is not fork-safe. The child must not take any lock, but serializing kvmgr state likely requires traversal that touches locked structures.
• Linux memory overcommit: if the parent modifies many pages during the child's serialization, the OS may OOM-kill the child (or parent).

3.2 Why Not Online Compaction or Snapshot-and-Continue?

Two alternatives to the dual-zarray COW approach were considered:

Online compaction (e.g., log-structured merge, shadow paging with page-level COW): requires per-page dirty tracking or a WAL-driven snapshot mechanism. XCE's kvmgr uses pointer-based structs (job_t*, worker_t*) — page-level COW would require every hot pointer to be remapped across page boundaries, adding complexity the codebase does not support today.

Snapshot-and-continue (serializing while holding a read lock the whole time): viable but blocks mutations for the duration of serialization. For 10K jobs at ~1KB each, serialization takes ~10–100ms. Blocking all job submits for 100ms at checkpoint time is worse than the brief freeze/thaw window (≤100μs for 10K entries — see Part 3.8).

Decision: dual-zarray application-level COW. The "stop-the-world" cost is only at freeze (O(1) pointer swap, ~ns) and thaw (O(shadow_caps) pointer copies, ~100μs). Serialization of the shadow arrays happens with no locks held, so mutations continue unblocked. This gives checkpoint-without-blocking at the cost of ~2× peak memory during COW. At XCE's scale (10K jobs × ~1KB = ~10 MB), this is negligible.

3.3 Design: Application-Level COW via Dual zarray

XCE's kvmgr uses a two-level structure: kvtab (hash table for key→index lookup) + data (xzarray — dense pointer array for iteration and index-based access). This structure makes application-level COW practical.

Key property: The hash table maps keys to stable indices in the zarray. COW only needs to happen at the zarray level. The kvtab never needs to be duplicated.

kvmgr structure:
  kvtab:  key ──hash──▶ index (stable, never changes for a given key)
  data:   xzarray of { inuse, data_ptr } indexed by index

COW operation:
  1. Swap: data → shadow (frozen, immutable)
  2. New data = empty array with same capacity, all inuse=0
  3. During COW: reads check data first, fall through to shadow
  4. After serialization: merge shadow back into data, apply GC

3.4 xzarray COW Extensions

xzarray_t is file-private in zarray.c: the struct definition is internal to the RTTI type system (line 353–369), and external code only accesses it through the typeinfo_t xzarray_type interface. Rather than expose the struct, COW state is stored in xzarray_private_t (the RTTI private section, allocated by makeInstance() alongside each xzarray instance):

/* In zarray.c — extend xzarray_private_t */
typedef struct {
    mylock_t          lock;
    xzarray_item_t*   shadow;      /* Frozen snapshot (NULL when not in COW) */
    size_t            shadow_caps; /* Shadow array capacity */
} xzarray_private_t;

New freeze/thaw operations are exposed by extending the RTTI intf_varray_t interface. The rtti library is maintained in-house (/nix/store/...-rtti-0.0.1/), so adding new function pointers is a controlled change — version bumped to 0.1.0 (see Appendix A.9 for the full rtti enhancement proposal).

Add two new function pointer types and slots to the intf_varray_t interface in rtti/typeinfo.h:

/* retain_fn_t is defined in rtti/typeinfo.h (see Appendix A.9) */
typedef int (*vfreeze_fn_t)(meta_obj_t* obj);
typedef int (*vthaw_fn_t)(meta_obj_t* obj, retain_fn_t should_retain, void* arg);

typedef struct {
    vstore_fn_t   vstore;
    vget_fn_t     vget;
    vget_vec_fn_t vget_vec;
    vput_fn_t     vput;
    vput_vec_fn_t vput_vec;
    vclear_fn_t   vclear;
    vfreeze_fn_t  vfreeze;  /* NEW — freeze array for COW snapshot */
    vthaw_fn_t    vthaw;    /* NEW — thaw + merge + GC after snapshot */
} intf_varray_t;

These map to xzarray_freeze() and xzarray_thaw() implementations in zarray.c, which access shadow state through RTTI_OBJ_TO_PRIV().

static int xzarray_freeze(meta_obj_t* obj) {
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);
    xzarray_t*         a    = RTTI_OBJ_TO_DATA(xzarray_t, obj);

    priv->shadow      = a->data;
    priv->shadow_caps = a->caps;

    a->data = calloc(a->caps, sizeof(xzarray_item_t));
    if (a->data == NULL) {
        /* Rollback: restore original data from shadow.
         * a->count still holds the pre-freeze value (not modified above),
         * so no restoration needed for count. */
        a->data = priv->shadow;       /* restore active pointer */
        a->pos = 0;
        priv->shadow = NULL;
        priv->shadow_caps = 0;
        return -ENOMEM;
    }
    a->count = 0;
    a->pos   = priv->shadow_caps;  /* start past shadow to prevent vstore index collisions */
    return 0;
}

When shadow != NULL (i.e., priv->shadow != NULL), the array is in COW mode. The existing xzarray_item_t is unchanged:

typedef struct {
    _Atomic _Bool inuse; /* C11 atomic flag — true when slot holds data.
                          * Changed from unsigned int :1 bitfield because
                          * C11 atomic operations require an addressable
                          * object, and bitfields are not addressable.
                          * _Atomic _Bool is lock-free on all supported
                          * platforms. */
    void*       data;
} xzarray_item_t;

3.5 Modified zarray Operations During COW

Critical constraint: During COW mode, vclear is never called (see Part 4). This means inuse=0 in the active array always means "slot not yet touched" — there is no tombstone ambiguity.

Runtime enforcement: xzarray_clear() must assert priv->shadow == NULL at entry. If vclear is invoked while COW is active, the array's invariant is violated (a hole in the active array would cause vget to fall through to stale shadow data). The assertion converts a silent data corruption bug into a loud crash during development:

static void xzarray_clear(meta_obj_t* obj) {
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);
    assert(priv->shadow == NULL && "xzarray_clear called during COW — deferred cleanup required");
    /* ... existing clear logic ... */
}

vget — Two-Level Lookup

static int xzarray_get(meta_obj_t* obj, int idx, void** data) {
    xzarray_t*         a    = RTTI_OBJ_TO_DATA(xzarray_t, obj);
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);

    /* COW mode: check active first, fall through to shadow */
    if (priv->shadow != NULL) {
        if (idx < a->caps && atomic_load_explicit(&a->data[idx].inuse, memory_order_acquire)) {
            *data = a->data[idx].data;
            return 0;
        }
        if (idx < priv->shadow_caps && atomic_load_explicit(&priv->shadow[idx].inuse, memory_order_acquire)) {
            *data = priv->shadow[idx].data;
            return 0;
        }
        return -ENOENT;
    }

    /* Normal path: unchanged */
    /* ... existing code ... */
}

vput — Write-Through to Active at Same Index

/* vput during COW: write to active at the same index.
   The kvtab still maps key→same index, so the next vget
   finds the updated value in active first. */
static int xzarray_put_vec(meta_obj_t* obj, int count, int* idxs, void** data) {
    xzarray_t* a = RTTI_OBJ_TO_DATA(xzarray_t, obj);

    /* Ensure active has capacity */
    int maxid = 0;
    for (int i = 0; i < count; i++)
        if (idxs[i] > maxid) maxid = idxs[i];
    xzarray_ensure_items(a, maxid + 1);

    for (int i = 0; i < count; i++) {
        int idx = idxs[i];
        if (atomic_load_explicit(&a->data[idx].inuse, memory_order_acquire) && a->killer)
            a->killer(a->data[idx].data);
        a->data[idx].data  = data[i];
        atomic_store_explicit(&a->data[idx].inuse, true, memory_order_release);
    }
    return 0;
}

Since indices are stable (hash table maps key→same index always), a vput(42, new_data) writes to active[42]. The hash table still says key→42. Reads check active[42] first and find the updated value. The shadow's shadow[42] is silently masked.

Lock protocol design: The rtti intf_lock_t provides a thin abstraction — lock() / unlock() function pointers that each type implements. Whether vget acquires a read lock internally is an implementation decision, not an rtti contract. The current design uses the kvmgr my_rwlock_t as the outer coordination lock: callers acquire the kvmgr READ lock before vget and release it after they are done with the returned pointer. The xzarray mylock_t mutex is used for individual array mutations (vstore, vput, vclear). This two-level locking (kvmgr rwlock for container-level coordination, xzarray mutex for array-level mutations) is the established pattern — see handle_jobsubmit() in job.c:455-525 for the reference implementation. vget does not acquire the xzarray mutex because the caller already holds the kvmgr READ lock, which prevents concurrent freeze/thaw from modifying the array structure.

Memory ordering during COW serialization (CRITICAL — must be implemented): During the no-locks-held serialization phase (Part 3.8), vput() writes to active[idx] while the serialization thread reads shadow[] (which is immutable). These two accesses are to different memory regions, so there is no data race on the shadow array. However, concurrent vput() and vget() on the active array require careful memory ordering. vget() reads active[idx].inuse then active[idx].data as separate fields — a vput() that writes .data before setting .inuse = 1 could cause vget() to see inuse = 1 with a stale .data pointer if the compiler reorders the stores or the CPU reorders their visibility.

Why x86-64 alone is NOT sufficient: The x86 memory model prevents the CPU from reordering stores to the same location, but .data and .inuse are different fields. The C compiler is free to reorder stores to different fields of a struct regardless of target architecture — the volatile qualifier on the struct would prevent this but imposes performance costs on all accesses. Without volatile, the compiler may emit the store to .inuse before the store to .data, and a concurrent vget() would see inuse = 1 with an uninitialized or stale .data pointer — a use-after-free.

Required implementation: Use C11 atomics for the .inuse field to enforce release-acquire ordering between vput() (writer) and vget() (reader):

• Change inuse from unsigned int :1 bitfield to _Atomic _Bool (C11 atomics cannot operate on bitfields — the address-of operator is undefined for bitfields, and _Atomic _Bool is lock-free on all supported platforms).
• vput() writes .data first, then stores .inuse using atomic_store_explicit(&a->data[idx].inuse, true, memory_order_release).
• vget() reads .inuse using atomic_load_explicit(&a->data[idx].inuse, memory_order_acquire), and only accesses .data if .inuse == true.
• The release-acquire pair ensures .data is visible to vget() when it sees .inuse == true.

This is required on ALL architectures, not just ARM — compiler reordering is sufficient to cause the race even on x86-64. The kvmgr READ lock in vget() callers provides happens-before between vput() and vget() on the same vcpu thread (serialized by lock/unlock), but does not prevent the compiler from reordering stores within a single vput() call. The atomic store on .inuse is the only reliable barrier against both compiler and CPU reordering.

Concurrent vstore race under READ lock (CRITICAL): During COW mode, multiple vcpu threads may hold the kvmgr READ lock simultaneously and call vstore(). The kvmgr READ lock does NOT serialize these calls — it allows concurrent readers. Two concurrent vstore() calls can both see inuse==0 at the same idx during the scan phase, and the second silently overwrites the first's .data — silent data loss. Fix: During COW mode, the entire xzarray_store() find-slot-and-write sequence is serialized via the inner xzarray mutex (mylock_lock(&priv->lock)). The mutex is acquired at the top of the COW-mode path and released on all return paths. This is the ONLY place in the COW protocol where the inner mutex is required — normal-mode vput and vstore rely on the kvmgr WRITE lock for serialization (unchanged). See the full xzarray_store() implementation below (R-22 fix).

vstore — Must Skip Shadow Indices

Design invariant: After xzarray_freeze(), the new active array is empty (count=0, pos=shadow_caps). The vstore scan during COW mode must use a linear scan bound to [shadow_caps, a->caps) — it must NOT wrap around to index 0 when reaching a->caps. If all slots in [shadow_caps, a->caps) are occupied, vstore grows the array via xzarray_ensure_items() and assigns from the new capacity. This prevents index reuse below shadow_caps, which would cause silent data corruption when vget falls through to the stale shadow entry.

Why circular scan is unsafe during COW: The existing xzarray_store() at zarray.c:456-490 wraps from idx >= a->caps to idx = 0. With pos = shadow_caps, after enough vstore calls fill [shadow_caps, a->caps), the next store wraps to index 0 — but index i < shadow_caps may have an inuse=0 slot which vstore reclaims. The kvtab now maps new_key → i, while shadow[i] still holds the old entity's pointer. A vget(i) finds active[i].inuse == 0 (recently assigned but not yet visible to a concurrent reader) or active[i] holds the new entity while shadow[i] holds stale data — either case produces wrong results.

Fix: Add a COW-mode flag check in xzarray_store(): when priv->shadow != NULL, scan only the range [pos, a->caps) without wrapping. If no free slot is found, grow the array:

After thaw, pos is reset to the normal low-water scan (or left at the current high-water mark — the existing vstore scan-forward logic handles this correctly).

vstore — Linear Scan During COW (No Wrap)

static int xzarray_store(meta_obj_t* obj, void* data) {
    xzarray_t*         a    = RTTI_OBJ_TO_DATA(xzarray_t, obj);
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);
    int idx;

    if (priv->shadow != NULL) {
        /* COW mode: linear scan from pos without wrapping.
         * Must not reuse indices < shadow_caps — those belong to shadow.
         * inuse must be read with atomic_load_explicit(acquire) to pair
         * with vput's atomic_store_explicit(release) — both vstore and
         * vput hold the kvmgr READ lock concurrently, so the atomic
         * ordering (not the lock) provides inter-reader visibility.
         *
         * CRITICAL: The entire find-slot-and-write sequence is serialized
         * via the inner xzarray mutex (priv->lock). Without this, two
         * concurrent vstore callers can both see inuse==0 at the same idx
         * during the scan, and the second silently overwrites the first's
         * .data — a silent data loss bug. The mutex is only held during
         * COW mode; normal-mode vput/vstore rely on the kvmgr WRITE lock
         * for serialization (unchanged). */
        mylock_lock(&priv->lock);
        for (size_t idx = a->pos; idx < a->caps; idx++) {
            if (atomic_load_explicit(&a->data[idx].inuse,
                                     memory_order_acquire) == 0) {
                a->data[idx].data  = data;
                atomic_store_explicit(&a->data[idx].inuse, true, memory_order_release);
                a->count++;
                a->pos  = idx + 1;
                mylock_unlock(&priv->lock);
                return idx;
            }
        }
        /* No free slot in [shadow_caps, a->caps) — pre-grown range is full.
         * Do NOT call xzarray_ensure_items() (realloc) here: realloc moves
         * a->data, invalidating pointers held by concurrent vget() callers
         * that only hold the kvmgr READ lock (not the inner xzarray mutex).
         * The 1.5× pre-grow at freeze time is designed to make this path
         * rare (< 100ms COW window at normal submit rates). Return -ENOMEM
         * so the caller logs WARN and the client retries after thaw. */
        mylock_unlock(&priv->lock);
        return -ENOMEM;
    }

    /* Normal path: unchanged circular scan */
    /* ... existing xzarray_store() code ... */
}

Iteration — Merge Walk of Both Arrays

static int xzarray_iter_next(meta_obj_t* obj, void* ctx, void** data) {
    xzarray_t*         a    = RTTI_OBJ_TO_DATA(xzarray_t, obj);
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);
    size_t*            pos  = (size_t*)ctx;  /* iterator position — caller-allocated cursor */

    if (priv->shadow != NULL) {
        /* COW mode: walk both arrays, active takes precedence */
        size_t max_caps = (a->caps > priv->shadow_caps) ? a->caps : priv->shadow_caps;
        for (size_t i = *pos; i < max_caps; i++) {
            if (i < a->caps && atomic_load_explicit(&a->data[i].inuse, memory_order_acquire)) {
                *data = a->data[i].data;         /* active wins — shadow[i] intentionally skipped */
                *pos = i + 1;
                return RTTI_ITER_OK;
            }
            if (i < priv->shadow_caps && atomic_load_explicit(&priv->shadow[i].inuse, memory_order_acquire)) {
                *data = priv->shadow[i].data;    /* shadow only (no active entry at i) */
                *pos = i + 1;
                return RTTI_ITER_OK;
            }
        }
        return RTTI_ITER_NO_DATA;
    }

    /* Normal path: unchanged */
    /* ... existing code ... */
}

Serialization must iterate shadow only: During the no-locks-held serialization phase (Part 3.8), the iterator must only walk the shadow array (priv->shadow[0..shadow_caps-1]), since shadow is immutable and active[] may be concurrently modified by vput. The xzarray_item_t struct has a bitfield (inuse : 1) and a pointer (data) — these are not atomically written, so reading active[] without locks is a data race. New entries written to active[] during serialization are covered by the WAL and will be included in the next checkpoint's shadow. The serialization code should use a shadow-only iterator (bound: 0..shadow_caps-1), not the merge-walk iterator above. The merge-walk iterator is for normal-mode queries where callers hold the kvmgr READ lock. The kvmgr READ lock acquisition provides the necessary memory ordering to see vput's release-ordered stores to active[].inuse. Iterating without the READ lock is a data race.

COW invariant — new keys may hash below shadow_caps: New entities added during COW mode (via vstore) are allocated at indices >= shadow_caps (the linear scan starts at pos = shadow_caps). However, a vput for a new hash table key can map to any index, including indices below shadow_caps. This is safe because: (1) vput writes to active[i] at the same index, (2) vget checks active first and finds the new entry, (3) thaw merges shadow and detects active[i].inuse == 1, taking the active-wins branch and freeing the stale shadow entry. The COW vstore linear scan only prevents index reuse from the free list below shadow_caps — it does not prevent new hash-to-index mappings in that range via vput.

3.6 Freeze — Start COW Phase

static int xzarray_freeze(meta_obj_t* obj) {
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);
    xzarray_t*         a    = RTTI_OBJ_TO_DATA(xzarray_t, obj);

    /* Current data becomes shadow (frozen, immutable) */
    priv->shadow      = a->data;
    priv->shadow_caps = a->caps;

    /* Allocate new active with 1.5× capacity (pre-grown to avoid realloc during COW).
     *
     * WHY OVER-ALLOCATE: During COW, vstore() scans [shadow_caps, a->caps) for free
     * slots. If a->caps == shadow_caps (typical case right after freeze), that range
     * is empty and any vstore() would need to call xzarray_ensure_items() → realloc().
     * realloc() moves a->data, invalidating any pointer that a concurrent vget() caller
     * holds into the array (vcpu threads can hold READ lock simultaneously, so the
     * growth lock — the inner mutex — does not block concurrent vget() access).
     *
     * By pre-allocating 1.5× shadow_caps slots, the range [shadow_caps, 1.5×shadow_caps)
     * is immediately available for vstore(). For a typical COW window (< 100ms), this
     * range is almost always sufficient (new entity arrival rate is bounded by the
     * job submit rate). If the pre-grown range fills up, vstore() returns -ENOMEM and
     * the caller logs a WARN — the submit is rejected transiently and the client retries
     * after the COW window closes (< 100ms). This avoids the realloc-during-COW race.
     *
     * CALLER PRE-ALLOCATION (P3 optimization for large deployments): For clusters
     * with >10K entities, this calloc() can allocate 10–25 MB while the kvmgr WRITE
     * lock is held, blocking all readers for the duration of the kernel TLB operation.
     * To keep the write lock hold time at ~nanoseconds (pointer-swap only), a P3
     * improvement adds `xzarray_freeze_with_preallocated(obj, new_active, cow_caps)` —
     * a NON-RTTI direct C function (NOT routed through vfreeze_fn_t, which has the
     * single-argument signature `int (*)(meta_obj_t*)` and cannot accept extra params).
     * The pre-allocation pattern:
     *
     *   size_t cow_caps = a->caps + a->caps/2 + a->delta;  // read caps without lock
     *   xzarray_item_t* new_active = calloc(cow_caps, sizeof(xzarray_item_t));
     *   if (!new_active) return -ENOMEM;
     *   rwlock_wlock(kvmgr);
     *   // Direct call — bypasses RTTI dispatch intentionally:
     *   xzarray_freeze_with_preallocated(obj, new_active, cow_caps);
     *   rwlock_wunlock(kvmgr);
     *
     * If a->caps grew between the caps read and the lock acquisition, the function
     * falls back to inline calloc() and frees the pre-allocated buffer.
     * For P0, the inline calloc() path inside xzarray_freeze() is correct and
     * acceptable; the pre-allocation optimization is deferred to P3. */
    size_t cow_caps = priv->shadow_caps + (priv->shadow_caps / 2) + a->delta;
    a->data = calloc(cow_caps, sizeof(xzarray_item_t));
    if (a->data == NULL) {
        /* Rollback: restore original data pointer from shadow.
         * a->count still holds the pre-freeze value (not modified above),
         * so no restoration needed for count. */
        a->data = priv->shadow;
        a->pos = 0;
        priv->shadow = NULL;
        priv->shadow_caps = 0;
        return -ENOMEM;
    }
    a->caps  = cow_caps;  /* active array is larger than shadow — update caps */
    a->count = 0;  /* calloc zeroes the allocation; count reflects zero live entries */
    a->pos   = priv->shadow_caps;  /* start past shadow to prevent vstore index collisions */
    return 0;
}

This is a pointer swap — O(1), no data copied. The write lock is held only for the duration of the swap (~nanoseconds). The pre-grown active array (cow_caps = 1.5 × shadow_caps + delta) eliminates realloc during the COW window, avoiding the concurrent-vget dangling-pointer race.

3.7 Thaw — End COW Phase with Merge and GC

/* xzarray_thaw — merge shadow back into active, apply optional GC.
 * should_retain: callback to decide entity survival. NULL = keep all,
 * no GC (used when aborting a checkpoint — merge shadow back unchanged).
 *
 * TWO-PASS DESIGN (required for correct GC):
 * Pass 1: Merge all shadow entries into active[] (pointer copies only,
 *         no retention decisions, no frees). This ensures all entities
 *         are visible in active[] before any GC check runs.
 * Pass 2: Run should_retain() on merged active[] entries, freeing those
 *         that fail retention. This is correct because job_has_dependents()
 *         can now see ALL entities (including ex-shadow ones) in active[].
 *
 * Single-pass designs where retention checks happen during merge are
 * INCORRECT: when should_retain(shadow[i]) calls job_has_dependents(),
 * dependents at shadow[j] (j > i) have not yet been merged and are
 * invisible in active[]. This causes premature GC of jobs with dependents. */
static int xzarray_thaw(meta_obj_t* obj, retain_fn_t should_retain, void* arg) {
    xzarray_private_t* priv = RTTI_OBJ_TO_PRIV(obj);
    xzarray_t*         a    = RTTI_OBJ_TO_DATA(xzarray_t, obj);

    /* Pass 1: Merge all shadow entries into active. No retention checks. */
    for (size_t i = 0; i < priv->shadow_caps; i++) {
        if (!atomic_load_explicit(&priv->shadow[i].inuse, memory_order_relaxed)) continue;  /* empty slot, skip */

        if (i < a->caps && atomic_load_explicit(&a->data[i].inuse, memory_order_relaxed)) {
            /* Active was written during COW — free shadow's stale object */
            if (a->killer) a->killer(priv->shadow[i].data);
        } else {
            /* Copy shadow pointer to active (no retention check yet) */
            xzarray_ensure_items(a, i + 1);
            /* Use explicit field copies — cannot struct-copy xzarray_item_t
             * because it contains _Atomic _Bool inuse (C11 prohibits plain
             * assignment of atomic fields). Thaw runs under kvmgr WRITE lock,
             * so relaxed ordering is sufficient. */
            a->data[i].data = priv->shadow[i].data;
            atomic_store_explicit(&a->data[i].inuse,
                atomic_load_explicit(&priv->shadow[i].inuse, memory_order_relaxed),
                memory_order_relaxed);
            a->count++;
        }
    }

    /* Free shadow — all live pointers are now in active[] */
    free(priv->shadow);
    priv->shadow      = NULL;
    priv->shadow_caps = 0;

    /* Pass 2: Run retention/GC checks on the fully-merged active[].
     * All entities are now visible, so job_has_dependents() finds dependents
     * regardless of their original position in the shadow array. */
    if (should_retain != NULL) {
        for (size_t i = 0; i < (size_t)a->caps; i++) {
            if (!atomic_load_explicit(&a->data[i].inuse, memory_order_relaxed)) continue;
            if (!should_retain(arg, a->data[i].data)) {
                if (a->killer) a->killer(a->data[i].data);
                atomic_store_explicit(&a->data[i].inuse, false, memory_order_relaxed);
                a->data[i].data  = NULL;
                a->count--;
                /* Caller must also remove kvtab entry for this index */
            }
        }
    }

    /* Reset scan position for normal vstore operation.
     * After merge, some indices < shadow_caps have inuse=0 (GC'd entities)
     * and new entries at indices >= shadow_caps have inuse=1 (added during COW).
     * Starting from 0 ensures vstore reuses freed slots instead of growing
     * the array monotonically across checkpoints. */
    a->pos = 0;
    return 0;
}

3.8 Checkpoint Flow

persist_checkpoint() fires on timer thread
       │
       ▼
  For each kvmgr (jobs, workers, queues, resources, configs):
    rwlock_wlock()
    xzarray_freeze()          ← O(1) pointer swap
    record checkpoint_lsn
    rwlock_wunlock()
       │
       ▼
  Serialize all shadows → Cap'n Proto → disk
  (no locks held, controller serving requests normally,
   reads go through COW two-level lookup transparently)

  **CRITICAL: kvtab access during serialization**: The serialization phase
  reads shadow data without locks (shadow is immutable after freeze), but
  MUST NOT access the kvmgr hash table (kvtab) during this phase. The kvtab
  is shared mutable state — concurrent vstore calls can insert new keys while
  serialization is iterating the hash table, creating a data race on the
  hash chain pointers. Instead, serialization enumerates shadow entries by
  walking the shadow zarray indices [0..shadow_caps) directly, without
  consulting kvtab. Each shadow slot at index i contains an inuse flag and
  a data pointer; serialization reads shadow[i].inuse and shadow[i].data
  for each slot — these are immutable (no lock needed). The kvtab key
  lookup is only needed during the freeze phase (under write lock) to build
  a snapshot of key→index mappings, which is stored alongside the shadow
  array for serialization use:

  **Serialization index snapshot (built during freeze, used during serialize)**:
  During `xzarray_freeze()`, after the pointer swap but before releasing the
  write lock, the checkpoint code captures key→index mappings from kvtab for
  each live entity. For each kvmgr, a local struct array is allocated:

  ```c
  typedef struct {
      int    index;    /* shadow slot index */
      char   key[XCE_MAXNAME_LEN]; /* key string — deep-copied during capture (not a borrowed
                                    * pointer into kvtab). kvtab can rehash and move string storage
                                    * after the write lock is released; owned copies remain valid
                                    * throughout the lock-free serialization phase. */
      size_t key_len;  /* key length (includes null terminator) */
  } slot_key_t;

xzarray_freeze() returns after the pointer swap. While still holding the kvmgr write lock (before releasing it), the caller captures key→index mappings by walking the kvtab and building a slot_key_t[] array:

/* Caller-side step: after vfreeze(obj) returns, still under kvmgr write lock */
vfreeze(obj);  /* xzarray_freeze — O(1) pointer swap */
slot_key_t* keys = NULL;
size_t n_keys = 0;
kvmgr_capture_keys(kvmgr, &keys, &n_keys);  /* walks kvtab under write lock;
                                              * deep-copies each key string into
                                              * slot_key_t.key via strncpy */
/* Now release write lock. Serialization reads keys[] (no locks held).
 * keys[] holds owned copies — safe against concurrent kvtab rehash. */

This approach avoids threading output parameters through the RTTI interface. The vfreeze_fn_t signature remains int (*vfreeze_fn_t)(meta_obj_t* obj). Key capture is a separate caller-side function that walks the kvtab hash table while the write lock is still held. The serialization phase reads the keys[] array (no locks held) to map shadow slots to their keys in the snapshot. After serialization completes and thaw runs, each slot_key_t in the array holds an owned copy — no separate per-key free is needed. free(keys) releases the array. │ ▼ After serialization confirmed on disk: For each kvmgr: rwlock_wlock() xzarray_thaw(should_retain): - Merge shadow → active (pointer copies for untouched entries) - Apply retention policy → free expired/hidden entities (GC) - Free shadow kvtab_gc(): ← remove hash entries for GC'd indices rwlock_wunlock()

CRITICAL: kvtab_gc must run inside the same write-lock section as thaw. kvtab_gc_keys() MUST be called before releasing the kvmgr write lock. Freed zarray indices (where inuse was set to 0 by thaw's GC pass) become available for reuse by the next vstore() call. If kvtab_gc_keys() runs after the write lock is released, a concurrent vstore() (holding only a READ lock) can allocate the freed index for a new entity before kvtab_gc removes the stale key→index mapping, causing the old key to resolve to the new entity's data — a hash→zarray inconsistency. The checkpoint flow above (freeze → serialize → thaw → kvtab_gc → unlock) is correct because all four operations run under the same write lock. │ ▼ Delete WAL segments covered by this checkpoint

**Cross-kvmgr freeze ordering and referential consistency**: The checkpoint flow freezes kvmgrs sequentially (jobs → workers → queues → resources → configs), releasing each kvmgr's write lock after its freeze. This means the shadow snapshots for different kvmgrs capture state at slightly different LSN points. A job serialized in the jobmgr snapshot may reference a worker (via `exec_host`) that was added after the workermgr snapshot was taken. This is acceptable because WAL replay after snapshot loading corrects all inconsistencies — all mutations between the earliest and latest freeze points are captured in the WAL entries that follow the snapshot. On recovery, `persist_recover()` loads the snapshot and then replays WAL entries in strict LSN order, which restores full consistency. The checkpoint_lsn recorded in `persist_t` (used for WAL trimming) is set to the **minimum** LSN across all kvmgrs at the time of each freeze. Using the minimum ensures WAL entries between the earliest and latest kvmgr freeze points are preserved — entries covering LSN [min_freeze..max_freeze] are needed to bring kvmgrs frozen at earlier LSNs up to date during recovery replay. Using the maximum would trim precisely those bridging WAL entries and corrupt the cross-kvmgr recovery.

**Capture order (CRITICAL)**: `checkpoint_lsn` must be set **after all kvmgr freezes complete**, not incrementally during the freeze loop. Record each kvmgr's freeze-time LSN into a local array (`uint64_t freeze_lsns[N_KVMGRS]`), then `checkpoint_lsn = min(freeze_lsns[0..N_KVMGRS-1])` in one assignment after the loop. If `checkpoint_lsn` is updated inside the loop (e.g., `checkpoint_lsn = min(checkpoint_lsn, current_lsn)` at each step), the intermediate values are visible to WAL-trim decisions before all freezes complete — a racing checkpoint-triggered trim could delete WAL entries that a later freeze still needs.

Non-kvmgr zarrays: cluster_t also contains standalone zarray_t fields (endpoints, configs, sharedinst) that are not managed by kvmgr. These are handled differently during checkpoint:

• c->endpoints: Not serialized in the snapshot. Endpoints are transient network connections that are rebuilt on recovery (workers/schedulers reconnect after restart). The deferred endpoint cleanup mechanism (Part 4.9) handles cleanup during normal operation, not checkpoint.
• c->configs: Serialized under a separate write lock (lock(c->configmgr, RTTI_LOCK_READ) for reads, RTTI_LOCK_WRITE for freeze/thaw). Configs are read-only during normal operation and rare writes (admin config changes). Config freezes are serialized as a separate Cap'n Proto array in the snapshot.
• c->sharedinst: Not serialized. Shared instances are reconstructed from config on recovery.

Shadow lifetime and shutdown safety: xzarray_thaw() calls free(a->shadow) and sets a->shadow = NULL. The serialization thread reads shadow without any lock (shadow is "immutable after freeze"). This is safe only if thaw never runs while the serialization thread is still reading. The normal checkpoint flow guarantees this: persist_checkpoint_timer_fn() (running on the delaymgr timer thread) does the freeze and key-capture under the write lock, then spawns a dedicated serialization pthread for the slow I/O (Cap'n Proto serialization + fsync). Thaw is called by that same serialization pthread after the snapshot is confirmed on disk. The delaymgr timer thread returns immediately after spawning, keeping the delaymgr responsive. The serialization pthread ID is stored in persist_t.snap_thread so that cluster_fini() can join it. pthread_cond_timedwait(&persist.checkpoint_done, ...) in cluster_fini() is the normal-path wait; pthread_join(persist.snap_thread, NULL) is the deadline-exceeded path that ensures no use-after-free when the checkpoint is cancelled but the serialization thread is still running. The distinction matters: "same checkpoint timer thread" is intentionally avoided — the timer thread is shared across all delaymgr tasks and must not block on I/O; only the dedicated serialization pthread can block. However, the shutdown path must also observe this constraint: if cluster_fini() or a shutdown signal fires while a checkpoint is in progress, it must wait for the serialization thread to complete before calling xzarray_thaw(). Implement with a persist_t.checkpoint_in_progress atomic flag and a pthread_cond_t checkpoint_done condition variable. The condition variable must use a dedicated pthread_mutex_t checkpoint_mutex — NOT persist.lock (which is a mylock_t). pthread_cond_timedwait() requires a pthread_mutex_t*. mylock_t is typedef pthread_mutex_t (confirmed by mylock.h), so the type concern does not apply. The real reason for using a dedicated checkpoint_mutex is deadlock avoidance: persist.lock is held by vcpu threads during persist_wal_sync_append() (which may block on repl_wait_ack()), so cluster_fini() attempting to acquire persist.lock for the timed wait would deadlock against a vcpu thread holding it. The dedicated checkpoint_mutex avoids this: it is never held during replication waits. On shutdown: cluster_fini() acquires checkpoint_mutex, checks checkpoint_in_progress. If set, it calls pthread_cond_timedwait(&persist.checkpoint_done, &checkpoint_mutex, &deadline) where deadline is now + 5 seconds (configurable). The checkpoint thread signals checkpoint_done after completing serialization and thaw (or after canceling). If the deadline expires, cluster_fini() cancels the in-progress checkpoint (sets a cancel flag) and skips thaw — incomplete checkpoint state is cleaned by the next restart's persist_recover(). This avoids busy-waiting on the atomic flag and provides bounded shutdown latency. IMPORTANT: after the 5-second deadline expires and the cancel flag is set, cluster_fini() must pthread_join() the checkpoint/serialization thread before proceeding to free any cluster state. The serialization thread reads shadow arrays and slot_key_t arrays that hold pointers to kvtab keys — if cluster_fini() proceeds to free the kvmgr and its contents before the serialization thread exits, a use-after-free occurs. The join ensures the serialization thread observes the cancel flag, exits its loop, and returns before any cluster state is destroyed.

Crash safety during checkpoint: If the controller crashes mid-checkpoint (during shadow serialization or mid-thaw), the previous snapshot on disk remains valid. On recovery, persist_recover() loads the last valid snapshot and replays WAL entries from that snapshot's LSN forward — the incomplete checkpoint is simply ignored. WAL segments are only deleted after the new snapshot is fully written and fsync'd to disk. This ensures atomicity: a checkpoint either completes fully (new snapshot + WAL trim) or is treated as if it never happened.

Mid-thaw crash and COW in-memory state: If the controller crashes mid-thaw (some shadow entries merged back to active[], some not yet merged, priv->shadow != NULL), the COW in-memory state is lost on restart — it was never persisted. persist_recover() starts from the last completed snapshot and replays WAL entries from that point. Mutations that were applied to active[] during the COW phase but not yet written to WAL (e.g., a vput that occurred after the freeze but before the WAL write via the hook) are also lost. This is consistent with the WAL durability model: only mutations that have been written to WAL and fsynced are durable. The COW mechanism is a transient optimization for non-blocking checkpoint; it provides no additional durability guarantee beyond the WAL. Implementers must NOT attempt to recover COW state from WAL — on restart, xzarray_freeze() runs anew from the fully recovered state.

Lock hold times:

• Freeze: ~nanoseconds (pointer swap)
• Thaw + GC: O(shadow_caps) pointer copies + retention checks. For 10K entries, ~50-100 microseconds. Note: if should_retain calls job_has_dependents() for job entities, the retention check becomes O(n_terminal_jobs × n_total_jobs) in the worst case because job_has_dependents() scans all jobs per call. At XCE's current scale (typically <1K finished jobs in a retention window), this is fast. For deployments with >10K jobs, consider building a reverse dependency index to make job_has_dependents() O(k) where k is the number of dependents per job, reducing GC to O(shadow_caps + n_terminal_jobs × k).

3.9 Lock Hierarchy for COW

XCE uses a two-lock hierarchy in kvmgr:

kvmgr rwlock (outer)          — protects kvtab + serializes freeze/thaw
  └─ xzarray mutex (inner)    — protects individual vstore/vget/vput/vclear operations

• kvmgr my_rwlock_t: The outer lock. Taken as read-lock for normal lookups (xkvmgr_find). Taken as write-lock for additions (xkvmgr_add) and for COW freeze/thaw coordination.
• xzarray mylock_t: The inner lock. Two modes:
  • Normal mode: NOT taken inside xzarray_put_vec() — callers hold the kvmgr WRITE lock for mutations, which provides serialization; the inner mutex adds no value and is skipped.
  • COW mode: Taken inside xzarray_store() for the entire find-slot-and-write sequence (R-22 fix). During COW, multiple vcpu threads hold the kvmgr READ lock concurrently, so the outer lock does not serialize concurrent vstore() calls. The inner mutex is the only serialization point preventing two threads from claiming the same free slot. vput() in COW mode is still NOT protected by the inner mutex — vput() writes to a known index (never allocates a new slot), so there is no find-slot race.
  • Thaw mode: NOT taken during thaw — the kvmgr WRITE lock is held for the entire thaw+GC pass, which serializes all access.

COW lock protocol:

freeze:
  rwlock_wlock(kvmgr)           ← serialize: no concurrent lookups during swap
    xzarray_freeze(data)         ← O(1) pointer swap, no xzarray mutex needed
  rwlock_wunlock(kvmgr)

serialize shadows:
  (no locks held — shadows are immutable)

thaw + GC:
  rwlock_wlock(kvmgr)           ← serialize: block lookups during merge
    xzarray_thaw(data, retain)   ← merges shadow→active, frees GC'd entities
                                     (kvmgr write lock held; inner mutex not needed)
    kvtab_gc_keys(kvmgr)         ← prune stale hash entries
  rwlock_wunlock(kvmgr)

Per-object locks are NOT nested inside kvmgr lock. The kvmgr lock serializes container-level operations (key→index lookup, COW coordination). Per-object locks (job_t.lock, worker_t.lock) are taken after releasing the kvmgr lock — the lookup returns a referenced pointer, and the caller locks the object separately.

GC use-after-free mitigation — reference-counted object lifecycle (CRITICAL): During thaw+GC, the kvmgr WRITE lock is held, which prevents new kvtab lookups. However, a vcpu thread that previously completed a kvtab lookup (under READ lock) and obtained a pointer to a job_t may still be accessing that job object after releasing the READ lock. If thaw+GC frees that job during the window between the vcpu thread's READ unlock and its next access, the vcpu thread dereferences freed memory — a use-after-free.

This window exists because XCE's lock protocol is: (1) acquire READ lock, (2) kvtab lookup → get pointer, (3) release READ lock, (4) use pointer to access the object. Between steps 3 and 4, the checkpoint thread can acquire the WRITE lock, run thaw+GC, and free the object.

Mitigation — epoch-based reclamation with generation counter: The kvmgr gains a uint64_t epoch counter, incremented atomically at the start of xzarray_thaw() (inside the WRITE lock, before any GC frees). Vcpu threads record kvmgr.epoch at lookup time (step 1). After each per-object operation, the vcpu thread checks whether kvmgr.epoch has changed. If it has, the pointer may be stale and the vcpu must re-lookup the object. This is a lightweight generation counter — no per-object reference counting is needed, just an atomic counter check after each kvmgr lookup.

Alternative (simpler but higher overhead): vcpu threads hold the kvmgr READ lock for the entire duration of their per-object operation (lookup → use → release). This prevents GC from acquiring the WRITE lock until all READ holders are done. The downside is longer READ lock hold times, which delays checkpoint and increases the checkpoint "freeze" pause. At XCE's scale (10K jobs, ~100μs per job operation), holding the READ lock across a per-job operation is acceptable — the checkpoint thread waits at most the duration of one per-job operation for the WRITE lock.

Recommended approach: Use the simpler per-operation READ lock hold for P0, with the epoch optimization as a P3 performance improvement if checkpoints cause detectable latency spikes. The P0 implementation holds the kvmgr READ lock across the entire per-object operation (lookup through use), preventing GC from freeing objects that any vcpu thread is actively accessing. This is safe because per-object operations (job status update, worker state change) are fast (<100μs), and the WRITE lock holder (checkpoint) only needs to wait for the brief overlap.

Why there is no lock race during serialization + thaw: The concern is that concurrent vput calls during the "no locks held" serialization phase might interleave with xzarray_thaw(). This is safe because:

1. During serialization (no locks): vput writes to active[] under the kvmgr READ lock. These writes are to active[i].data — they never touch shadow[].
2. During thaw (WRITE lock held): xzarray_thaw() merges shadow[] → active[]. The write lock blocks all concurrent vput (read-lock) calls. No interleave is possible.
3. Thaw's merge logic: for each shadow[i], if active[i].inuse == 1 (set by a concurrent vput during serialization), thaw takes the "active wins" branch and frees the stale shadow pointer. This is the intended behavior — active is newer.

This matches the existing pattern in handle_jobsubmit() (job.c:455-525): lock(c->jobmgr, RTTI_LOCK_READ) for lookup, release, then per-object lock for modification.

3.10 Checkpoint Timer

The checkpoint thread uses the existing delaymgr timer system. delaymgr will be enhanced to change sleep(precision) to nanosleep(precision_ms) in delay.c, giving sub-second granularity. The delay_task() delay parameter remains in seconds throughout — only the underlying sleep precision changes. The delay.c ts_nextrun = now + delay arithmetic is unchanged (seconds-based). All existing callers continue to pass seconds; the precision reduction only affects how tightly the timer fires relative to its scheduled time.

/* In delay.c — change sleep(precision) to nanosleep with precision_ms granularity.
 * The ts_nextrun / run_period fields remain in seconds (time_t). No caller changes
 * are required. Default precision_ms for the cluster delaymgr: 200ms, set via:
 *   delaymgr_init(&c->delaymgr, 200 /* precision_ms */, run_delay_tasks)
 *
 * Existing callers at cluster.c:756 (heartbeat) continue passing seconds. */

/* In persist_start() — register periodic checkpoint timer (1-second poll) */
int ckpt_timer = delay_task(&(c->delaymgr),
                                    1,   /* 1-second poll interval (seconds) */
                                    -1,  /* recurring */
                                    (delay_task_entry_fn_t)persist_checkpoint_timer_fn,
                                    (void*)c);
/* The periodic checkpoint (snap_interval_s, e.g. 60s) is counted inside
 * persist_checkpoint_timer_fn() via a tick counter, not as a separate delay_task. */

Checkpoint overlap handling: If a timer-driven checkpoint fires while a previous checkpoint is still in progress (checked via persist_t.checkpoint_in_progress atomic flag), the timer callback logs an INFO message ("checkpoint already in progress, skipping") and returns. The size-based inline trigger (Part 3.11) provides a safety valve for rapid WAL growth.

Checkpoint timer for inline-triggered checkpoints: The timer callback checks persist_t.checkpoint_requested on each 1-second tick, responding within ~1 second of the flag being set.

Benefits:

• No raw pthread_create/pthread_join management for the timer thread
• delaymgr already handles timer cancellation in cluster_fini()
• Consistent with how all periodic tasks are managed
• No semantic change to delay_task() callers — all pass seconds as before

P0-3 implementation note: The delaymgr nanosleep() change (delay.c) is a prerequisite for P0-3. The change is: rename precision field to precision_ms and replace sleep(precision) with struct timespec ts = {0, precision_ms * 1000000L}; nanosleep(&ts, NULL);. Update delaymgr_init() call in cluster.c:604 from precision=2 to precision_ms=200. Audit all delaymgr_init() callers to confirm they all pass precision_ms in milliseconds.

3.11 Snapshot File Format

Serialized from shadow arrays while no locks are held:

┌─────────────────────────┐
│ SnapshotHeader (Capnp)   │
│  - lsn: last WAL LSN     │
│  - timestamp             │
│  - term                  │
│  - version: UInt32 (=1)  │
│  - nJobs, nQueues, nWorkers │
│  - checksum: CRC32(body) │
│  - headerChecksum: CRC32(header fields) │
├─────────────────────────┤
│ Job array (Capnp)        │
├─────────────────────────┤
│ Queue array (Capnp)      │
├─────────────────────────┤
│ Worker array (Capnp)     │
├─────────────────────────┤
│ Resource array (Capnp)   │
├─────────────────────────┤
│ Config array (Capnp)     │
└─────────────────────────┘

Snapshot CRC scope: The SnapshotHeader.checksum (CRC32) covers the Cap'n Proto–serialized body (Job array, Queue array, Worker array, Resource array, Config array) excluding the SnapshotHeader struct itself. The header is verified separately via SnapshotHeader.headerChecksum. After headerChecksum passes, nJobs, nQueues, nWorkers are trusted as authoritative counts — a secondary validation against actual deserialized array lengths is a debug assertion only (assert(nJobs_from_header == actual_cap'nproto_list_count)), not a required safety check. Treating the counts as untrusted after a passed headerChecksum would create a contradictory two-truth-source design. This split design avoids the chicken-and-egg problem where the CRC needed for body verification is itself part of the data being verified.

• Time-based: every snap_interval_s (default 60s)
• Size-based: when WAL exceeds snap_wal_size_mb (default 64 MB)
• Shutdown: final checkpoint in cluster_run() before exiting the main loop

Inline size check in WAL append path: The timer-driven size check alone creates a window where WAL can grow unbounded between timer ticks (e.g., 10K submits/sec × ~200 bytes = 2MB/sec, filling 64MB in ~32s, which may fall between two 60s timer ticks). Both persist_wal_async() and persist_wal_sync_append() must check cumulative WAL size after writing and trigger an inline checkpoint if current_segment_bytes > snap_wal_size_mb.

Inline checkpoint mechanism — atomic flag, not direct freeze: The calling thread (which holds the kvmgr READ lock from the hook call chain) cannot directly call persist_checkpoint(), because persist_checkpoint() acquires the kvmgr WRITE lock — and a thread holding a READ lock that attempts to acquire a WRITE lock on the same POSIX rwlock results in undefined behavior (deadlock per POSIX). Instead, the inline trigger sets an atomic flag: atomic_store(&persist->checkpoint_requested, 1). The checkpoint timer thread (running via delaymgr) checks this flag on each wake (every snap_interval_s, or sooner if the flag is set — use a shorter poll interval like 1-2 seconds when the flag is set). When checkpoint_requested is observed, the timer thread calls persist_checkpoint() (it holds no kvmgr lock, so acquiring the WRITE lock is safe). After the checkpoint completes, the timer thread clears the flag. This approach: (1) avoids the rwlock self-deadlock, (2) ensures checkpoint always runs on a thread that does not hold the kvmgr READ lock, and (3) bounds the WAL growth to snap_wal_size_mb + max_wal_per_poll_interval (at 2-second poll, ~4MB additional WAL at 2MB/sec). If an inline checkpoint is already in progress (checked via persist_t.checkpoint_in_progress atomic flag), the WAL append skips setting checkpoint_requested — the in-progress checkpoint will trim the WAL when it completes.

WAL growth during an in-progress checkpoint: When checkpoint_in_progress is set, the inline trigger is suppressed. WAL growth during this window is bounded by the checkpoint duration times the submission rate. For pathological cases (very large shadow state, slow serialization disk), the WAL can grow significantly. Mitigation: if cumulative WAL size exceeds 2 × snap_wal_size_mb while checkpoint_in_progress is set, log a WARN("WAL size %.1f MB exceeds 2x threshold during in-progress checkpoint"). If WAL size exceeds wal_max_size_mb (default 2 GB), trigger an emergency checkpoint even if one is in progress — the existing checkpoint must be completing soon (serialization is lock-free, so a stuck checkpoint indicates an I/O problem). Operators should size snap_wal_size_mb based on expected peak throughput and worst-case checkpoint duration. A rule of thumb: snap_wal_size_mb ≥ peak_submit_rate_KB/s × worst_case_checkpoint_seconds / 1024.

---

Part 4: Deferred Entity Cleanup at Checkpoint

4.1 Design Philosophy

No entity is ever deleted during normal operation. All entities follow the same lifecycle:

User-visible behavior: This is intentionally different from eager deletion. Finished jobs and unavailable workers remain visible in API queries (filtered by their terminal status) for their configured retention period — this is the desired behavior. Users need to see recently completed jobs to check exit status and history. Abruptly removing entities on state transition (e.g., a job disappearing the moment it finishes) would confuse users and break operational workflows that depend on querying recent job results. The retention periods (finished_retain_s, worker_retain_s) control how long this historical data remains visible.

alive state ──▶ terminal state ──▶ [TTL/retention expires] ──▶ cleaned at next checkpoint

This eliminates vclear from the hot path. The del_key callback in kvmgr is still registered as NULL (the existing design in kvmgr.c:286), but the deferred cleanup mechanism replaces it: instead of xkvmgr_del() being called at state transition time, expired entries are removed during checkpoint GC via a kvtab_gc() scan that iterates over all kvmgr entries, calls should_retain() on each, and removes those that fail retention. This is safe because:

1. Checkpoint GC holds the kvmgr write lock (serializing all other access)
2. kvtab removal is O(n) but n = total entries (10K jobs, etc.) — fast enough for the infrequent checkpoint path
3. Eliminates the need for del_key entirely — the existing NULL registration is the intended design, with deletion deferred to checkpoint GC instead of the hot path.

Design invariant: del_key is deliberately NULL. The deferred GC at checkpoint handles entity removal via kvtab_gc(), not through hot-path key deletion. The RTTI interface's del_key = NULL is correct — no change is needed.

This simplifies COW (no tombstones, no three-state flags — see Part 3)

4.2 Terminal States by Entity Type

Entity	Terminal States	Trigger	Cleanup Criteria
Job	JOB_STAT_DONE, JOB_STAT_EXIT, JOB_STAT_PERR	Job finishes naturally	No dependents AND finish_time + finished_retain_s < now
Job	JOB_STAT_TRANSIENT (0x400)	Primary failover — job was RUNNING on old primary but no state update in standby's WAL	Worker reconnection (sets actual state) OR transient_since + transient_job_timeout_s < now — auto-terminate with JOB_STAT_EXIT and reason "worker did not reconnect after failover" (see config transient_job_timeout_s, default 300s)
Worker	WORKER_UNAVAILABLE	Worker daemon dies / disconnects	dead_since + worker_retain_s < now
Worker	WORKER_HIDDEN	Admin explicit remove (off-hours maintenance only)	Immediate cleanup at next checkpoint (no TTL)
Queue	QUEUE_HIDDEN	Admin deletes queue (off-hours maintenance only)	Immediate cleanup at next checkpoint
Config	CONFIG_HIDDEN	Admin deletes config key	Immediate cleanup at next checkpoint
Resource	(never deleted)	N/A	Resources persist permanently

Important: In production clusters, deleting queues or removing hosts while the cluster is running can cause job loss and is not acceptable. WORKER_HIDDEN and QUEUE_HIDDEN are intended for off-hours maintenance windows only, when no jobs are running on the affected resources. The deferred cleanup design exists so that removals during maintenance can be batched and completed efficiently at the next checkpoint.

4.3 How Terminal States Work

Entities in terminal states remain inuse=1 in the zarray and still have their kvtab entries. Business logic ignores them through status-based filtering that already exists:

• Scheduler skips JOB_STAT_DONE/JOB_STAT_EXIT/JOB_STAT_PERR jobs (already happens)
• Scheduler skips WORKER_UNAVAILABLE workers (already happens)
• API queries filter by status (already happens)

The new states WORKER_HIDDEN, QUEUE_HIDDEN, CONFIG_HIDDEN follow the same pattern — business logic and queries skip them, but the data structure entries persist until the next checkpoint.

4.4 New HostState / EntityState Values

Worker status uses bitmask flags (e.g., XCE_WORKER_UNAVAIL = 0x00010000), not a state enum. Add new bitmask flags and a QueueStatus hidden value:

/* In types.h — new bitmask flags for deferred cleanup (XCE_WORKER_UNAVAIL already exists at 0x00010000) */
#define XCE_WORKER_HIDDEN    (0x00200000)  /* Admin explicitly removed — invisible but not yet cleaned up */

/* Job status — add transient bitmask for HA failover.
 * NOTE: 0x200 is already JOB_STAT_WAIT (types.h:351) — do NOT reuse it.
 * JOB_STAT_TRANSIENT uses 0x400 which is currently free. */
#define JOB_STAT_TRANSIENT   (0x400)       /* Job state uncertain after failover — awaiting worker re-join */

/* Queue status — add hidden bitmask */
#define QUEUE_STAT_HIDDEN    (0x0100)       /* Admin deleted — invisible but not yet cleaned up */

In the Cap'n Proto schema, Worker.status is UInt32 (bitmask-compatible). QueueStatus may also need a hidden value added depending on its representation.

XCE_WORKER_UNAVAIL already exists in types.h as 0x00010000. The new additions are XCE_WORKER_HIDDEN, JOB_STAT_TRANSIENT, and QUEUE_STAT_HIDDEN (or equivalent).

4.5 Retention Policy at Checkpoint

The should_retain callback passed to xzarray_thaw() implements entity-specific retention.

Supporting functions (both implemented as part of P0-13):

/* job_is_terminal — returns true if job is in a terminal state */
bool job_is_terminal(const job_t* j);
/* Implementation: return (j->stat & (JOB_STAT_EXIT | JOB_STAT_DONE | JOB_STAT_PERR)) != 0; */

/* job_has_dependents — returns true if any other job depends on this jobid.
 * Dependency sources:
 *   - Array jobs: if j is an array master, any array element with stat != terminal
 *   - Job dependencies: any pending job with pending_ctrl referencing j->jobid
 *     (i.e., held because this job must finish first)
 * Called only on terminal jobs (job_is_terminal() == true) during checkpoint.
 * Caller holds kvmgr write lock (during thaw), so this function must not
 * re-acquire any kvmgr lock.
 *
 * CRITICAL: During thaw, shadow entries are merged into active one at a time.
 * When should_retain() is called for shadow[i], entries at higher indices
 * (shadow[j] where j > i) have NOT yet been merged into active[]. If
 * job_has_dependents() scans only active[], it will miss dependents that
 * reside in not-yet-merged shadow slots, causing premature GC of jobs that
 * have dependents. To avoid this, xzarray_thaw() MUST use a two-pass
 * approach: (1) merge ALL shadow entries into active[] (pointer copies for
 * untouched entries, skip entries where active[i].inuse == 1 and free shadow[i].data),
 * then (2) run should_retain() checks on the fully-merged active[] array.
 * This ensures that all entities are visible in active[] before any retention
 * decision is made. The two-pass approach is safe because merge (step 1) only
 * copies pointers and sets inuse flags — it does not call should_retain() or
 * free any memory.
 *
 * Returns false if the jobid is not found (treated as no dependents).
 *
 * LOCKING REQUIREMENT: should_retain() MUST be called inside the kvmgr WRITE
 * lock section (during thaw). It MUST NOT be called off the critical path
 * (e.g., in a background GC thread) because concurrent kvtab lookups via
 * xkvmgr_find() (holding only a READ lock) would encounter freed entities
 * still referenced by hash entries until kvtab_gc runs. The window between
 * entity free and kvtab cleanup must be atomic with respect to READ-lock
 * lookups. The checkpoint flow (freeze → serialize → thaw/GC → kvtab_gc →
 * unlock) guarantees this atomicity by holding the WRITE lock throughout. */
bool job_has_dependents(cluster_t* c, uint64_t jobid);

/* Job retention: keep if it has dependents OR TTL not expired.
 * Also handles JOB_STAT_TRANSIENT auto-termination via transient_job_timeout_s. */
static bool job_should_retain(void* arg, void* item) {
    cluster_t* c = (cluster_t*)arg;
    job_t*     j = (job_t*)item;

    /* Auto-terminate transient jobs whose timeout has expired.
     * JOB_STAT_TRANSIENT is NOT terminal (job_is_terminal() would return false),
     * so we handle it before the terminal check. transient_job_timeout_s=0 disables
     * auto-termination (operator must intervene manually). j->transient_since is set
     * when the job enters JOB_STAT_TRANSIENT at promotion time. */
    if ((j->stat & JOB_STAT_TRANSIENT) &&
        c->persist.config.transient_job_timeout_s > 0) {
        time_t now = time(NULL);
        if (j->transient_since > 0 &&
            now > j->transient_since + (time_t)c->persist.config.transient_job_timeout_s) {
            /* Convert to terminal state so it falls through to normal GC below.
             * LOCK ORDERING NOTE: Do NOT write a WAL entry here.
             * job_should_retain() is called from xzarray_thaw() which is called
             * from persist_checkpoint() while holding the kvmgr WRITE lock.
             * persist_wal_async() requires persist.lock — acquiring persist.lock
             * while holding the kvmgr WRITE lock would introduce an undocumented
             * lock ordering dependency that can deadlock with vcpu threads that
             * acquire persist.lock first then attempt xkvmgr_find() (READ lock).
             * Instead: record the auto-terminated jobid in a local heap array
             * declared in persist_checkpoint():
             *   uint64_t* auto_term = calloc(MAX_AUTO_TERM_PER_CHECKPOINT, sizeof(uint64_t));
             *   int n_auto_term = 0;
             * (MAX_AUTO_TERM_PER_CHECKPOINT defined in config.h, default 10240.)
             * This is a local variable in persist_checkpoint(), NOT a field of persist_t.
             * After thaw completes and the kvmgr WRITE lock is released, walk the array
             * and call persist_wal_async() for each jobid. On persist_wal_async() failure,
             * log ERROR("auto-term WAL write failed for jobid %lu: %s", jobid, strerror(e))
             * and continue. The job reverts to JOB_STAT_TRANSIENT after recovery, but this
             * does NOT loop forever: the recovery retention pass (Part 2.5 step 5) treats
             * any JOB_STAT_TRANSIENT job older than transient_job_timeout_s as immediately
             * auto-terminatable. If the WAL write keeps failing across restarts, the job is
             * cleaned up on the NEXT successful checkpoint after recovery — bounded by one
             * checkpoint interval (snap_interval_s), not across unbounded restart cycles.
             * Increment ctrl_transient_jobs_stuck for each failed WAL write so monitoring
             * can detect persistent I/O problems that prevent clean GC.
             * If n_auto_term >= MAX_AUTO_TERM_PER_CHECKPOINT, stop recording (remaining
             * jobs survive this checkpoint and are auto-terminated in the next one — bounded
             * delay is acceptable). free(auto_term) after WAL writes complete. */
            j->stat = JOB_STAT_EXIT;
            j->finish_time = now;
            WARN("transient job %lu on worker %s auto-terminated after %us: "
                 "worker did not reconnect after failover",
                 j->jobid, j->exec_host, c->persist.config.transient_job_timeout_s);
            /* ctrl_transient_jobs_stuck metric incremented by caller after lock release */
        } else {
            return true;  /* transient timeout not yet expired */
        }
    }

    if (!job_is_terminal(j))
        return true;                          /* still active, keep */

    if (job_has_dependents(c, j->jobid))
        return true;                          /* other jobs depend on it */

    time_t now2 = time(NULL);
    if (j->finish_time + c->persist.config.finished_retain_s > now2)
        return true;                          /* TTL not expired */

    return false;                             /* safe to clean up */
}

/* Worker retention: keep if TTL not expired (or hidden = immediate) */
static bool worker_should_retain(void* arg, void* item) {
    cluster_t* c = (cluster_t*)arg;
    worker_t*  w = (worker_t*)item;

    if ((w->host.status & XCE_WORKER_HIDDEN) != 0)
        return false;                          /* admin removed, cleanup immediately */

    if (!IS_WORKER_UNAVAIL(w->host.status))
        return true;                          /* still connected, keep */

    time_t now = time(NULL);
    if (w->host.dead_since + c->persist.config.worker_retain_s > now)
        return true;                          /* TTL not expired */

    return false;                             /* safe to clean up */
}

CLOCK_MONOTONIC vs wall-clock time for retention: time(NULL) returns wall-clock time, which is vulnerable to NTP step adjustments (backward clock jumps can cause premature GC of terminal entities whose TTL appears to have expired). For production deployments, either: (a) use clock_gettime(CLOCK_MONOTONIC) for retention comparisons and store a monotonic timestamp alongside finish_time/dead_since, or (b) document the sensitivity to NTP step changes and require slewing-only NTP configuration (e.g., chrony with the smoothtime directive, which never steps the clock backward). The heartbeat module already uses CLOCK_MONOTONIC (Part 5.5) for exactly this reason. Option (a) is preferred but requires storing a struct timespec or time_t from CLOCK_MONOTONIC per entity (one extra 8-byte field per job/worker). Option (b) is acceptable for deployments where NTP is controlled by the cluster administrator.

4.6 kvtab Cleanup at Checkpoint

When xzarray_thaw frees an entity (returns it from should_retain as false), the kvtab still has a key→index mapping for it. After thaw completes, walk the kvtab and prune stale entries:

void kvmgr_gc_keys(meta_obj_t* obj) {
    xkvmgr_t* kvm = RTTI_OBJ_TO_DATA(xkvmgr_t, obj);

    /* Walk all hash chains. For each entry, check if the
       zarray slot at the mapped index is still inuse.
       If inuse=0, the entity was GC'd — remove hash entry. */
    kvtab_gc_stale_indices(kvm->kvtab, kvm->data);
}

This runs while the write lock is still held (during thaw), so it's safe and bounded. For 10K entries, the hash chain scan is ~tens of microseconds.

4.7 Index Reuse After Cleanup

After checkpoint cleanup, freed zarray slots become inuse=0 with their kvtab entries removed. The zarray's circular scan (pos) finds these freed slots, so new vstore calls reuse them. A new job can be assigned index 42 after the old job 42 was cleaned up — the kvtab gets a fresh key→42 entry for the new job. No confusion because the old key→42 entry was removed.

4.8 Benefits of Deferred Cleanup

Aspect	Before (eager delete)	After (deferred to checkpoint)
COW complexity	Requires tombstones or three-state flags	Simple — inuse=0 always means "not touched"
WAL entry types	Need WAL_*_DELETE types	No delete types — WAL is purely additive
WAL replay	Must handle insert-delete-reinsert correctly	Insert or update — always correct
Delete spikes	Random (worker disconnect, admin action)	Bounded — only at checkpoint intervals
Memory overhead	Freed immediately	Jobs retained for up to snap_interval_s; workers retained for up to worker_retain_s (default 86400s = 24h). Memory sizing: a typical worker_t in XCE is ~500 bytes. Retaining 2000 disconnected workers for 24h consumes ~1 MB — negligible. Job retention at 1h (default finished_retain_s = 3600) with 100K finished jobs × ~1 KB/job = ~100 MB at the high end; operators with very high job throughput may reduce finished_retain_s to limit peak memory.
zarray fragmentation	Holes appear unpredictably	Compacted at checkpoint during merge
kvtab fragmentation	Hash chains shrink unpredictably	Pruned at checkpoint — predictable

4.9 Endpoint Deferred Cleanup

Endpoints (the endpoint_t objects in the controller's c->endpoints zarray) also follow the deferred cleanup pattern, but with an important difference: the network resource (fd, SSL object) must be released immediately, while the slot cleanup is deferred.

When a worker or scheduler disconnects:

1. Immediate: Close fd and free SSL* — these are OS resources that cannot be retained
2. Immediate: Set ep->busy = 0 (unblock any pending vtask queue entries)
3. Deferred: Set ep->state = ENDPOINT_DISCONNECTED but do NOT call xzarray_clear() on the slot — defer cleanup to checkpoint

The endpoint slot is cleaned up at checkpoint via the same should_retain mechanism:

static bool endpoint_should_retain(void* arg, void* item) {
    endpoint_t* ep = (endpoint_t*)item;
    /* Disconnected endpoints have fd=-1 and SSL=NULL (already freed).
       The slot just needs to be reclaimed by the zarray. */
    return (ep->state != ENDPOINT_DISCONNECTED);
}

Why not delete immediately? Calling xzarray_clear() during handle_endpoint_error() would set inuse=0 on the active array during COW mode, creating a "hole" that doesn't exist in the shadow. This would cause vget two-level lookup to incorrectly fall through to stale shadow data. Deferring to checkpoint avoids this entirely.

Note on array type: The current c->endpoints uses plain zarray_t. As part of P0-2, zarray_t will be merged into xzarray_t (this merge is already partially completed), which enables endpoints to participate in COW. Until the merge is complete (P0-0a), the immediate-cleanup path is used: handle_endpoint_error() calls zarray_del() as it does today. After the merge, the deferred cleanup flow applies: set ep->state = ENDPOINT_DISCONNECTED, close fd/SSL immediately, and clean up the slot at checkpoint via should_retain. The P0-8a step in the implementation phases enforces this ordering — it depends on P0-0a (zarray merge) being complete first.

Why immediate fd/SSL close is safe during COW: The fd and SSL pointer are on the endpoint object itself (not on the zarray slot). Setting ep->state = ENDPOINT_DISCONNECTED and closing ep->fd + ep->ssl are mutations on the endpoint struct, not on the zarray metadata. The zarray slot remains inuse=1 pointing to the endpoint — the deferred cleanup at thaw/merge will free the struct and set inuse=0. Since vget/vput check inuse on the active array first (which still has inuse=1), reads still find the disconnected endpoint, and business logic filters on ep->state != ENDPOINT_DISCONNECTED.

Connection to vtaskmgr: When endpoint_block_tasks(ep) sets ep->busy = 1, and later endpoint_unblock_tasks(ep) sets ep->busy = 0, this per-endpoint serialization is maintained regardless of whether the endpoint is in COW mode. The busy flag is on the endpoint object itself, not on the zarray slot.

4.10 Existing Code Alignment: Worker Unavailable State

The mark_worker_unavail() function already exists in worker.c:516-526. It currently:

1. Sets the XCE_WORKER_UNAVAIL flag on the worker
2. Runs AFTER_EVENT hooks (which will trigger the WAL write for WAL_WORKER_CTRL)
3. Does not remove the worker from the workermgr — the worker remains in the zarray with inuse=1

This matches the deferred cleanup design: after mark_worker_unavail(), the worker is in WORKER_UNAVAILABLE state and will be garbage-collected at the next checkpoint when worker_retain_s expires.

The new additions needed:

1. Set w->host.dead_since = time(NULL) in mark_worker_unavail() so the TTL check works
2. Add WORKER_HIDDEN state for admin-initiated removal (sets state, never deletes from zarray)
3. The handle_worker_leave() path currently calls mark_worker_unavail(c, w) and sets w->ep = NULL — this is correct and needs no change

---

Part 5: HA Failover — LSF-Style Priority Election

5.1 Design Philosophy

Instead of Raft consensus (~5000 LOC), use LSF's simple priority-based model:

• A fixed list of controller candidates is defined in cluster config, ordered by priority
• The first (highest-priority) candidate that is alive becomes primary
• "Am I the highest-priority alive candidate?" is the entire election algorithm
• Split-brain risk: Under a network partition, a standby may promote itself while the primary is still alive but unreachable. This is the same trade-off LSF makes — priority-based election cannot prevent split-brain without quorum. XCE accepts this risk with the following assumptions and mitigations: (1) controllers share a dedicated management network in the same LAN with low partition probability — primary and standby controllers are expected to be co-located in the same data center, so network partitions between them are abnormal events; (2) term-based fencing prevents stale standbys from accepting new writes; (3) workers always reconnect to the current primary after partition heals. If split-brain is unacceptable, a quorum-based protocol (Raft) would be required instead.

Split-brain impact analysis: Under split-brain, the original primary continues serving its established API server connections. Since end users never connect directly to the controller — only through the API server — user requests continue flowing to the original primary. The promoted standby also accepts work if an API server connects to it, but in practice the API server maintains a long-lived TCP connection to the original primary and does not switch during a partition (this is confirmed by the current RegisterToBackend() implementation in clib.go, which uses nmasters = 1 with a single long-lived connection and no reconnection logic). However, if the partition drops the API server's TCP connection to the original primary, the API server has no reconnection path until P2.5-7 (libxce discovery for API server) is implemented. After P2.5-7, the API server would use xce_cluster_config_t to reconnect to any candidate, which could redirect it to the promoted standby — in that case, user requests would flow to the promoted standby and writes would diverge from the original primary's WAL.

Worker movement during split-brain: If the partition also severs the TCP connection between some workers and the original primary, those workers will reconnect (via libxce discovery) to the promoted standby. These workers then report their job status to the standby, which applies the updates and writes them to its local WAL. When the partition heals, the standby (lower term) must step down, and these workers must reconnect to the original primary. The critical reconciliation step is: after partition heals, the restored primary must receive state updates from workers that were served by the standby during the partition. This is self-healing — workers always re-notify their current job state on reconnection (existing behavior: handle_worker_join re-registers the worker, and the scheduler re-queries job state). No explicit state merge protocol is needed beyond the existing worker re-join flow, because workers are the source of truth for their own runtime state.

The actual risk is replication divergence and potential data loss: the standby's WAL diverges from the primary's WAL during the partition. Writes accepted by the promoting standby during a network partition are at risk of loss — when the partition heals, term-based fencing ensures the lower-term controller steps down, and the diverged WAL entries from the standby are discarded during full re-sync from the primary (which has the authoritative higher-term WAL). This means any job submits ACKed by the standby during the partition are not durable. At XCE's scale (a few thousand entities), full re-sync is bounded by the partition duration and is handled by full re-sync rather than partial merge. If preserving standby-served writes during partitions is required, a quorum-based protocol (Raft) would be needed instead.

Deployment assumption: LAN-only. XCE HA controllers all run on the same LAN (dedicated management network). The following table enumerates the specific failure modes that can cause LAN partitions, their expected recovery times, and the blast radius for split-brain data loss:

Failure Mode	Expected Duration	Blast Radius	Mitigation
Top-of-rack switch failure	5–30 min (auto-failover or manual swap)	All controllers on that switch are isolated	Deploy controllers across separate switches/racks
Switch port failure (single)	1–10 min (STP reconvergence)	One controller isolated; remaining 2 have quorum connectivity	Redundant uplinks per controller
Cable cut (copper/fiber)	30 min – 4 hours (physical replacement)	Controllers on opposite sides of the cut	Diverse cable paths between controllers
NIC failure (single controller)	5–15 min (operator restarts on backup NIC)	One controller down; no partition if remaining 2 can communicate	Dual-NIC bonding (active-passive)
Misconfigured VLAN or ACL	10 min – 2 hours (operator detects and corrects)	Intermittent or partial isolation	Change management controls on switch config
Kernel NIC driver bug (hang)	5–30 min (operator detect + reboot)	One controller appears dead but process is running	Health monitoring with NIC-level liveness checks
STP broadcast storm	1–10 min (storm suppression)	All controllers may see intermittent connectivity	Storm control on management switches

Split-brain window duration: The window opens when the standby's heartbeat timer fires (default heartbeat_timeout_ms = 10000ms) and closes when the partition heals and the higher-term message reaches the old primary. Data at risk: all job submits ACKed by the standby during this window are discarded on partition healing. At typical submit rates (1–10 submits/second), the worst case for a 10-minute partition is ~600–6000 job submissions at risk of loss.

Quantitative risk model: At XCE's expected submit rate of 1–10 jobs/second and a 99.9% LAN availability (8.76 hours downtime/year), the expected annual data loss is: partition_probability × mean_partition_duration × submit_rate. For a well-managed LAN: P(partition per year) ≈ 0.001 (conservative), mean duration ≈ 15 minutes, submit rate ≈ 5/sec → expected lost submits/year ≈ 4.5. Operators must evaluate whether this risk is acceptable for their workload.

Three-controller asymmetric partition scenario: With 3 controllers (ctrl-1 primary, ctrl-2 standby P=1, ctrl-3 standby P=2), consider an asymmetric partition where ctrl-1 can reach ctrl-2 but not ctrl-3, and ctrl-2 can reach ctrl-1 but not ctrl-3. ctrl-3 detects heartbeat timeout from ctrl-1 and probes candidates with priority < 2. If ctrl-3 also cannot reach ctrl-2, it receives no ACK within election_timeout_ms and promotes itself to primary — creating two primaries (ctrl-1 and ctrl-3). This is the same split-brain risk already acknowledged in Part 5.1, with the same mitigation: term-based fencing resolves the conflict when the partition heals. The lower-term primary (determined by which controller incremented term first) steps down. In this specific scenario, ctrl-3 would have a lower term than ctrl-1 (assuming ctrl-1 was already primary before the partition), so ctrl-3 steps down when connectivity resumes. If the partition also allows an API server to connect to ctrl-3, any jobs submitted to ctrl-3 during the partition are at risk of loss (same as the two-controller case). This scenario illustrates why the split_brain_risk_accepted: true config key is mandatory — even single-LAN deployments can produce asymmetric partitions during switch failures or cable cuts (see the failure mode table above).

This is fundamentally different from WAN or cross-DC deployments where partitions can persist indefinitely due to routing protocol convergence, BGP flap dampening, or ISP outages. For cross-site or multi-data-center HA requirements, the recommended approach is to deploy separate XCE clusters per site with cross-cluster job forwarding (out of scope), not a single replicated HA cluster.

Deployment requirements: LSF-style priority election is appropriate under the following conditions: (1) controllers share a dedicated management network on the same LAN — cross-DC or WAN deployments have unacceptably high partition probability; (2) ≤3 controller candidates — with more candidates, cascading promotions during instability become harder to reason about; (3) operators understand that job submits ACKed by a split-brain standby are not durable and are discarded on partition healing. If any of these conditions cannot be met, a quorum-based protocol (Raft) is required instead.

Required operator acknowledgment: To prevent accidental deployment without understanding the split-brain risk, HA mode requires an explicit opt-in config key:

ha:
  split_brain_risk_accepted: true   # REQUIRED — HA refuses to start without this
  candidates: [...]

cluster_init() must refuse to start HA mode if split_brain_risk_accepted != true, logging: "HA split-brain risk has not been acknowledged — set ha.split_brain_risk_accepted: true in cluster.bin (via config tool from cluster.yaml) after reading the HA deployment requirements".

Operator visibility for degraded mode: When sync_timeout_ms expires and the primary falls back to local-only ACK, this is NOT silent:

1. Log at ERROR level (not WARN) — degraded replication is a durability-affecting event
2. Increment ctrl_repl_degraded_count metric (alongside ctrl_split_brain_detected)
3. Early split-brain signal (state-change log, not per-submit): When the primary first enters the combined state of (degraded replication AND all heartbeat sends failing), log once at ERROR: "Primary entering isolated mode — cannot reach any standby for sync ACK or heartbeat. Split-brain risk: if a standby has promoted itself, submitted jobs may be lost. ctrl_isolated_primary_count=%u". Increment ctrl_isolated_primary_count on state entry. When the primary reconnects to any standby, log once at INFO: "Primary exiting isolated mode after %us — standby reconnected". Do not emit this log on every job submit — sustained degradation at 10 submits/sec produces 600 identical ERROR entries per minute, flooding logs and suppressing the signal. Track isolation state with a _Atomic int ha_isolated; flag in ha_state_t; set on entry, clear on reconnect. Also add primary_isolated_since (struct timespec, CLOCK_MONOTONIC) to ha_state_t to report isolation duration on reconnect.
4. The degraded state persists until the standby reconnects — subsequent submits in degraded mode each log at ERROR and increment the counter

Split-brain detection condition: ctrl_split_brain_detected is incremented when a controller receives a message (heartbeat, replication, election probe, or client request) from a peer that believes a DIFFERENT controller is primary. Specifically:

• A standby receives OP_heartbeat from a controller whose term is LOWER than the standby's current term (the sender is a stale primary that hasn't learned of the promotion)
• A primary receives a message (election probe, OP_replJoin) from a controller whose term is HIGHER than its own (a higher-priority candidate has promoted itself)
• A primary receives OP_heartbeat from another controller that claims to be primary with a term EQUAL to or HIGHER than the receiver's term

This detection is passive — it requires network communication, not active probing. During an actual network partition where no communication is possible, split-brain cannot be detected (this is the fundamental limitation of LSF-style election without quorum). The metric alerts the operator once communication is restored and the controller can observe the divergence.

Trade-off: Non-preemptive — if a higher-priority candidate recovers, it does NOT preempt the current primary. This avoids disruptive failback. Admin can trigger manual failover.

5.2 Candidate Configuration

The MasterCandidate struct already exists in controller.capnp with name, host, port. Priority is list position (index 0 = highest priority).

# cluster.yaml — human-editable config, processed by config tool
controller:
  candidates:
    - name: ctrl-1
      host: 10.0.1.1
      port: 7000
    - name: ctrl-2
      host: 10.0.1.2
      port: 7000
    - name: ctrl-3
      host: 10.0.1.3
      port: 7000

Configuration pipeline: The controller does not parse YAML directly. A separate configuration tool (CLI, out of scope for this design) validates cluster.yaml and produces a Cap'n Proto-encoded binary configuration file that the controller reads at startup. The validation tool checks:

• At least one candidate is defined
• No duplicate host:port pairs
• ha.split_brain_risk_accepted: true is set when HA mode is enabled
• ha.split_brain_risk_accepted is true when more than one candidate is configured

The controller reads only the binary Cap'n Proto config file — it has no YAML parsing dependency, no text config parsing, and treats its config as a trusted, validated blob. This eliminates config injection risks and simplifies the controller's dependency tree.

5.3 Role Management

typedef enum { CTRL_PRIMARY = 0, CTRL_STANDBY = 1 } ctrl_role_t;

typedef struct {
    _Atomic uint64_t term;            /* C11 atomic election term (fencing token) — see Part 5.7 */
    _Atomic int32_t role;             /* C11 atomic — use atomic_load/store_explicit, see Part 5.7 */
    int             my_priority;      /* My index in candidates[] */
    replicator_t    repl;             /* Replicate to standbys (primary only) */
    _Atomic uint64_t applied_lsn;     /* Last applied WAL entry (standby only).
                                       * Must be _Atomic because it is written by the iomgr/vtask
                                       * thread during OP_replEntry processing and read by the
                                       * watchdog thread when constructing OP_replJoin for catch-up.
                                       * Use atomic_store_explicit/load_explicit with release/acquire. */
    endpoint_t*     primary_ep;       /* Connection to current primary via iomgr SSL pipeline (standby only).
                                       * LIFECYCLE: set by iomgr accept/connect — iomgr owns the endpoint
                                       * and the ha_state_t struct only holds a borrowed reference.
                                       * On connection loss, iomgr sets primary_ep = NULL _before_ freeing
                                       * the endpoint. All access must check for NULL; no dereference
                                       * after NULL. The standby's catch-up thread reads primary_ep under
                                       * the ha_state_t mutex (ha->lock), which serializes it against
                                       * iomgr's NULL-then-free sequence. The watchdog thread also
                                       * checks primary_ep == NULL under ha->lock before heartbeat
                                       * comparison. This borrow-with-mutex pattern ensures no use-after-free:
                                       * iomgr holds ha->lock while clearing the pointer, so any
                                       * concurrent dereference sees either a valid endpoint or NULL. */
    char            primary_host[XCE_MAXNAME_LEN];
    uint32_t        primary_port;
    struct timespec last_heartbeat;   /* Last heartbeat from primary (CLOCK_MONOTONIC, standby) */
    _Atomic int     promotion_failed;      /* Non-zero if last promotion attempt failed — retry on next watchdog tick.
                                            * Must be _Atomic: written by ha_promote_to_primary() on the watchdog
                                            * thread, read by ha_check_heartbeat() on the same thread but also
                                            * visible to vcpu threads checking promotion status. Relaxed ordering. */
    _Atomic int     promotion_retry_count; /* Number of consecutive failed promotion attempts.
                                            * Must be _Atomic because it is incremented by the
                                            * watchdog thread and read by vcpu threads during
                                            * promotion status checks. Use atomic_load/store
                                            * with relaxed ordering. */
    _Atomic int     in_catch_up;           /* C11 atomic — non-zero while standby is processing
                                            * historical catch-up entries. Must be _Atomic because
                                            * it is written by the iomgr/vtask thread (on
                                            * OP_replJoin receipt and OP_replCatchUpDone) and
                                            * read by vcpu threads during ACK suppression
                                            * (Part 6.4). Use atomic_store/load_explicit with
                                            * release/acquire ordering.
                                            *
                                            * LIFECYCLE (standby-side, this field):
                                            *   Set to 1 by the standby when it sends OP_replJoin
                                            *   with applied_lsn < primary's committed_lsn.
                                            *   Cleared to 0 by the standby when it sends
                                            *   OP_replCatchUpDone (after applying snapshot + all
                                            *   bridging WAL entries up to committed_lsn).
                                            *   The standby initiates completion — it decides when
                                            *   its applied_lsn has reached committed_lsn.
                                            *
                                            * PRIMARY-SIDE MIRROR: standby_conn_t.in_catch_up
                                            *   (Appendix A.3) is cleared by the primary when it
                                            *   receives OP_replCatchUpDone from this standby.
                                            *   See Part 6.6 and P2-5 for the full protocol.
                                            *
                                            * Gates ACK suppression: during catch-up, the standby
                                            * does NOT send OP_replAck for historical entries
                                            * (see Part 6.4). */
    ha_config_t     config;           /* HA configuration (timeouts, intervals) */
    pthread_mutex_t lock;             /* Protects primary_ep and last_heartbeat against concurrent
                                       * iomgr NULL-then-free and watchdog reads. mylock_t is
                                       * typedef pthread_mutex_t (confirmed by mylock.h), but an
                                       * explicit pthread_mutex_t field documents the pthread_cond
                                       * usage intent and avoids any future mylock_t divergence.
                                       * Initialized in ha_init(). */
} ha_state_t;

typedef struct {
    uint32_t        heartbeat_interval_ms;  /* Primary heartbeat interval (default: 3000). Minimum 2000ms — the HA watchdog pthread (P1-3) uses nanosleep() so there is no delaymgr truncation concern. The 2000ms floor is a practical recommendation: values below ~1000ms risk false timeout triggers on loaded management LANs (SSL handshake + scheduling jitter). Enforce heartbeat_interval_ms >= 2000 in ha_init() as a conservative safe floor; operators needing faster heartbeats may lower this with care. */
    uint32_t        heartbeat_timeout_ms;   /* Standby heartbeat timeout (default: 10000) */
    uint32_t        election_timeout_ms;    /* Shared deadline for all parallel election probes (default: 2000) */
    uint32_t        probe_connect_timeout_ms; /* Per-probe TLS connect timeout for am_i_primary() probe threads
                                               * (default: election_timeout_ms / MAX_HA_CANDIDATES, minimum 500ms).
                                               * Each probe thread uses this as the poll() timeout in its
                                               * non-blocking connect loop, so it wakes up at least once per
                                               * probe_connect_timeout_ms to check the shared election_done flag.
                                               * This bounds the worst-case overshoot of election_timeout_ms
                                               * to one probe_connect_timeout_ms interval per thread.
                                               * ha_init() enforces: probe_connect_timeout_ms >= 500ms AND
                                               * probe_connect_timeout_ms <= election_timeout_ms. */
    uint32_t        max_promotion_retries;  /* Max consecutive failed promotion attempts before giving up (default: 5) */
} ha_config_t;

Added to cluster_t as ha_state_t ha;.

5.4 Startup Sequence

Controller starts
       │
       ▼
  Read candidates[] from config
  Find my_priority (my position in list)
       │
       ▼
  ┌──────────────────────────────────┐
  │ am_i_primary()?                  │
  │                                  │
  │ Launch one pthread per candidate │
  │ with priority < my_priority.     │
  │ Each thread makes a synchronous  │
  │ blocking TLS dial (no iomgr yet) │
  │ and sends OP_heartbeat probe.    │
  │                                  │
  │ Single shared deadline:          │
  │   election_timeout_ms (2000ms)   │
  │   enforced via condvar/timedwait │
  │                                  │
  │ If ANY thread gets a live ACK    │
  │ before deadline: → NO (standby)  │
  │ Cancel remaining threads.        │
  │                                  │
  │ If deadline expires, all probes  │
  │ failed/timed out: → YES (primary)│
  └──────┬──────────────┬────────────┘
         │          │
    PRIMARY      STANDBY
         │          │
         ▼          ▼
  cluster_init()  cluster_init()
  persist_init()  persist_init()
  persist_recover()  persist_recover()    ← iomgr NOT yet running
  persist_start()  persist_start()          workers get ECONNREFUSED,
                     connect to primary      retry via xce_reconnect()
  cluster_start()    (via iomgr, now running)  ← iomgr starts here
                   ha_heartbeat_watch()        workers can now connect
                   standby_loop()

Promotion recovery: A standby runs persist_recover() on startup (populating kvmgrs from its local snapshot + WAL) and then receives ongoing replication via apply_wal_entry() calls from OP_replEntry. By the time promotion occurs, in-memory state is already complete from both sources. It must not call persist_recover() again — that would replay the local WAL and double-apply every entry. Instead, the promotion path should: (1) validate CRC chain from the standby's own persist_t.checkpoint_lsn forward (already maintained by startup recovery + replication), (2) flush the local WAL fsync to ensure all replicated entries are on disk, (3) update the local WAL term to the new election term, (4) start replication to remaining standbys from the current LSN. If the promoted standby has gaps in its WAL (async entries not yet replicated before the old primary died), it requests the missing entries from the old primary's WAL on disk via catch-up (see Part 7 catch-up protocol).

Startup ordering note: am_i_primary() runs before cluster_init() (which starts iomgr/SSL/vtaskmgr). The election probe therefore launches one pthread per lower-priority candidate; each thread makes a synchronous blocking TLS dial using the same cert/key pair as the HA pipeline but without the event loop. All threads share a single election_timeout_ms deadline enforced via a pthread_cond_timedwait.

IMPORTANT — thread cancellation safety: pthread_cancel() is NOT used to stop election threads. pthread_cancel() running through OpenSSL shutdown handlers in a thread mid-handshake can deadlock (cancellation points inside TLS negotiation are not safe). Instead, use a shared atomic_int election_done flag (from <stdatomic.h>) with atomic_store() / atomic_load() operations:

• The main thread sets atomic_store(&election_done, 1) when the first ACK or the deadline fires
• Each probe thread checks atomic_load(&election_done) after each probe attempt and exits cleanly if set (closing its TLS connection and freeing resources)
• The main thread calls pthread_join() on all threads (which return promptly since they check the flag)
• If a thread is blocked during TLS dial, use a non-blocking connect() + poll() with a short per-attempt timeout so each thread can check election_done between probe retries
• Total worst-case latch is bounded by one poll() timeout interval per thread

After the election resolves, cluster_init() starts iomgr and all HA traffic uses the iomgr/SSL/endpoint pipeline. This avoids a circular dependency between election and iomgr startup, and parallel probing bounds worst-case startup latency to one election_timeout_ms regardless of cluster size.

ha_init() mutex initialization ordering: ha_init() must be called before am_i_primary() (the election probe), since ha_state_t.lock (the pthread_mutex_t protecting primary_ep and last_heartbeat) is initialized in ha_init(). Step 5.5 starts heartbeats before cluster_start() (via a dedicated pthread) — this pthread accesses ha->lock, which must already be initialized. Promotion (ha_promote_to_primary()) does NOT re-initialize the mutex — the mutex is initialized once in ha_init() and persists for the controller's lifetime across promote→demote→re-promote cycles. Re-initializing a pthread_mutex_t that may have waiters is undefined behavior.

ha_init() startup enforcement: At startup, ha_init() validates: heartbeat_interval_ms >= 2000 (the HA watchdog uses nanosleep() and is not subject to delay_task() integer-second truncation; the 2000ms floor is a conservative practical minimum to avoid false heartbeat-timeout triggers from SSL handshake jitter on loaded management LANs); heartbeat_timeout_ms >= heartbeat_interval_ms; election_timeout_ms >= 500; ncandidates >= 1. Duplicate candidate validation: ha_init() scans candidates[] and rejects any config where two entries share the same host (case-insensitive strcasecmp) or the same host:port pair. Duplicate candidate names cause ambiguous election probes — an entry appearing twice would receive two heartbeat monitors, trigger double replication connections, and confuse the priority ordering. The validation must log a FATAL error naming both duplicate entries and return -EINVAL. See ha_config_t in Part 5.3 for field documentation.

5.5 Monitor Thread (Standby)

The standby uses the controller's iomgr/vtask heartbeat infrastructure to monitor higher-priority candidates. The primary sends periodic heartbeat messages (OP_heartbeat) to connected standbys at ha_config.heartbeat_interval_ms (default 3000ms), and standbys watch for heartbeat timeouts from higher-priority controllers at ha_config.heartbeat_timeout_ms (default 10000ms — approximately 3 missed heartbeats):

/* The monitor is not a separate thread with raw TCP probes.
   Instead, it uses the existing iomgr/vtask/endpoint pipeline:

   1. The primary sends OP_heartbeat to all connected standbys
      at HA_HEARTBEAT_INTERVAL_MS (default 3000ms).
   2. Each standby tracks last_heartbeat_time for the primary endpoint.
   3. If last_heartbeat_time + HA_HEARTBEAT_TIMEOUT_MS < now,
      the primary is considered dead -> promote self.

   This provides application-level liveness (not just TCP connectivity)
   and reuses the SSL/endpoint/vtask pipeline that all controller
   connections already use. */

/* ha_check_heartbeat() — called from the dedicated HA watchdog pthread (NOT delaymgr).
 * P1-3 requires a dedicated pthread with nanosleep() or timerfd_create(CLOCK_MONOTONIC)
 * for sub-second heartbeat detection. delaymgr has 2-second tick granularity
 * (cluster.c initialises it with precision=2), making it unsuitable here.
 * Uses CLOCK_MONOTONIC — immune to NTP jumps and DST changes that would cause
 * spurious promotions (clock jumped forward) or missed timeouts (clock jumped back). */
static void ha_check_heartbeat(cluster_t* c)
{
    ha_state_t* ha = &c->ha;

    if ((ctrl_role_t)atomic_load_explicit(&ha->role, memory_order_acquire) != CTRL_STANDBY)
        return;

    /* Read primary_ep and last_heartbeat under ha->lock — iomgr clears primary_ep
     * (NULL-before-free) while holding this same lock, so we see either a valid
     * pointer or NULL, never a dangling one. */
    pthread_mutex_lock(&ha->lock);
    int no_primary = (ha->primary_ep == NULL);
    struct timespec last_hb = ha->last_heartbeat;  /* snapshot under lock */
    pthread_mutex_unlock(&ha->lock);

    if (no_primary)
        return;

    struct timespec now;
    clock_gettime(CLOCK_MONOTONIC, &now);
    int64_t elapsed_ms = (now.tv_sec  - last_hb.tv_sec)  * 1000LL
                       + (now.tv_nsec - last_hb.tv_nsec) / 1000000LL;
    if (elapsed_ms > (int64_t)ha->config.heartbeat_timeout_ms) {
        /* Primary heartbeat timeout — promote self.
         * If a previous promotion attempt failed (promotion_failed flag set),
         * retry promotion unless the retry limit is exceeded. */
        if (ha->promotion_failed &&
            atomic_load_explicit(&ha->promotion_retry_count, memory_order_relaxed) >= (int)ha->config.max_promotion_retries) {
            ERROR("cannot promote after %d attempts — manual intervention required",
                  atomic_load_explicit(&ha->promotion_retry_count, memory_order_relaxed));
            return;
        }
        ha_promote_to_primary(c);
    }
}

Heartbeat detection latency: The heartbeat watchdog runs on a dedicated pthread with nanosleep() (see P1-3 and ha_check_heartbeat() above), identical to the WAL fsync timer approach in Part 2.4. This achieves sub-second detection granularity. The actual primary-failure detection latency is heartbeat_timeout_ms + up to heartbeat_interval_ms milliseconds — the timeout fires, plus up to one heartbeat interval of jitter before the next check. For the default heartbeat_timeout_ms = 10000 and heartbeat_interval_ms = 3000, worst-case detection is 13 seconds. For HPC workloads requiring sub-10-second failover, reduce heartbeat_timeout_ms and heartbeat_interval_ms accordingly.

5.6 Promotion Sequence

When a standby promotes to primary:

1. Stop WAL replay from old primary
2. Increment term (ha.term++) — fencing token for stale standbys
3. Flush the local WAL (persist_flush() — fsync the last WAL segment to ensure all replicated entries are on disk) and verify WAL consistency (validate CRC chain from the standby's own last snapshot LSN forward — standbys run persist_recover() on startup and maintain their own local WAL and snapshots, so the validation starting point is the standby's persist_t.checkpoint_lsn). Do NOT call persist_recover() again — the standby's kvmgrs are already populated from replication (via apply_wal_entry() during normal OP_replEntry processing) AND from the startup recovery pass. Calling persist_recover() a second time would double-apply entries. Only a WAL flush + CRC chain validation is needed. WAL gap handling: Async-replicated entries that the old primary committed but had not yet replicated before crashing are lost — the dead primary cannot serve catch-up. This is a known limitation of the async replication model: non-sync WAL entries (job starts, finishes, rusage updates) may be lost on failover. The promoted primary's view is consistent but potentially stale; async state gaps self-heal through worker reconnection (see step 9). WAL consistency guarantee: the standby's local WAL is a strict prefix of the old primary's WAL — standbys only write entries received via OP_replEntry from the primary. No divergence is possible if the standby was continuously connected; the standby's WAL cannot contain entries the old primary did not originate. The only split-brain divergence scenario (standby accepted writes from clients during a partition) cannot occur because standbys do not accept writes — they only redirect to the primary. CRC chain restart on promotion (COH-5): After the WAL flush, persist_flush() MUST reset persist_t.last_replicated_crc to WAL_CRC_NONE. This signals that the promoted primary is starting a new replication chain — any remaining standbys have not received entries from this new primary yet, so there is no valid "previous entry CRC" to chain against. The first entry this primary broadcasts will carry prev_crc = WAL_CRC_NONE, which the standby receiver accepts as a valid chain start (see Part 6.4 catch-up and segment-boundary handling). Without this reset, the new primary would broadcast prev_crc = last_replicated_crc (the CRC of the last entry the old primary sent), which no standby would recognize — causing a chained-CRC mismatch disconnect on the first broadcast.
4. Switch role to CTRL_PRIMARY — only after recovery succeeds. If recovery fails, abort this promotion attempt and set ha.promotion_failed = 1. The standby remains in CTRL_STANDBY role, but since the old primary is dead (that triggered this promotion), the next heartbeat watchdog check (ha_check_heartbeat()) will re-fire ha_promote_to_primary(). The promotion_failed flag gates retry: the standby retries promotion on each watchdog tick unless: (a) a new heartbeat is received from a higher-priority candidate (another controller won the race), or (b) a configurable max_promotion_retries (default: 5) is exceeded, at which point the standby logs FATAL "cannot promote after %d attempts — manual intervention required" and stops retrying. This prevents a permanent no-primary state while bounding the retry window.
5. Register persistence hooks — standby did not register persistence hooks during startup (hooks are registered only after persist_recover() completes, and only on the primary path; standbys register them here on promotion). Idempotent registration guard: ha_promote_to_primary() must check whether hooks are already registered before adding them (e.g., via a ha->hooks_registered flag in ha_state_t). Without this guard, a promote→demote→re-promote cycle would accumulate duplicate hooks that fire N times per event, causing double WAL writes and replication sends. 5.5 Start heartbeat immediately — begin sending heartbeat messages to other standbys BEFORE cluster_start() (step 8). The promoted standby must establish its authority as the new primary by sending heartbeats with its incremented term. This prevents other standbys from timing out and promoting themselves during the potentially lengthy replicator initialization steps.

Pre-iomgr mechanism: At this point in the promotion sequence, cluster_start() (and thus iomgr) has not yet run. The heartbeat messages are sent via the same raw TCP/SSL pthread mechanism as the election probes (Part 5.4): one short-lived pthread per lower-priority standby, each making a synchronous blocking TLS dial, sending a single OP_heartbeat frame with the new term and role, and closing the connection. This is intentionally distinct from the iomgr/endpoint pipeline. After cluster_start() (step 8), all subsequent HA traffic (heartbeats, replication) switches to the iomgr/SSL/endpoint pipeline permanently.

The heartbeat thread is launched with the new term and role set to PRIMARY, but cluster_start() (step 8) is deferred until all preparation steps complete. CRITICAL: The heartbeat messages sent during this pre-operational phase contain only term and role — no LSN or state information. Other standbys receiving these heartbeats update their ha.term and primary_host to the new primary, but do not expect any replication data until step 8 is reached. Bounded timeout for iomgr follow-up: After a standby receives a pre-iomgr heartbeat and updates primary_host, it starts a countdown of 2 × heartbeat_timeout_ms. If no iomgr-level OP_heartbeat arrives within this window from the claimed new primary, the standby resets primary_host to empty and re-runs am_i_primary(). This bounds the "stuck on a dead promoted primary" scenario: if the promotion crashes mid-sequence (after step 5.5 pthreads sent heartbeats but before cluster_start() at step 8), other standbys detect the absence of iomgr heartbeats and re-elect independently. Without this timeout, a standby that updated primary_host but never receives an iomgr heartbeat from the promoted primary would wait indefinitely at heartbeat_timeout_ms intervals, never re-running the election. 6. Initialize replicator — establish connections to remaining standbys 7. Scheduler gating is implicit — the scheduler has not reconnected at this point in the promotion sequence (it reconnects in step 9 via xce_reconnect()). There is no active "gate the scheduler" protocol step: the scheduler simply isn't connected, so no scheduling can occur. When the scheduler reconnects (step 9), it self-heals by sending a full state-sync query to enumerate all PEND and RUN jobs, rebuilds its allocation table from scratch, and resumes scheduling from a clean slate — this is the actual gating mechanism. No additional synchronization is required here. Note: the old text said "pause or flush the scheduler's run queue before accepting writes" — this was incorrect; the scheduler's queue is a scheduler-internal data structure that the controller does not directly control. 8. Start normal primary operation — cluster_start(), accept connections 9. Workers and scheduler reconnect automatically — when the old primary's TCP connections drop, libxce's xce_reconnect() backoff+jitter loop engages on each worker and scheduler. They retry through the candidate list until they reach the promoted primary, then send a standard OP_workerJoin / OP_schedulerJoin. No special failover logic is needed in the promoted primary — workers rejoin via the existing handle_worker_join() path and re-report their current job state. The scheduler re-queries job state when scheduling. The promoted primary is functionally equivalent to a freshly booted controller that has replayed its WAL — it has a consistent but potentially stale view of job lifecycle states, which is corrected as workers reconnect and update their status. No "re-push config" logic is needed on promotion: the join response carries runtime config as usual.

This means async-replicated job lifecycle transitions (start, finish, fail, stop, resume, rusage) that were lost when the old primary crashed are not present in the promoted standby's WAL. The promoted standby's view of these jobs may be stale (e.g., showing a job as PEND when it is actually RUNNING). Workers correct this state when they reconnect and re-report their current job status. Until all workers reconnect, the scheduler should treat job state as uncertain and avoid scheduling decisions that depend on stale information.

Job state during the reconnection window: Jobs that were in JOB_STAT_RUN on the old primary but have no matching update in the standby's WAL enter a transient unknown state. The promoted primary marks such jobs with a sentinel flag (e.g., JOB_STAT_TRANSIENT) to signal uncertainty to operators and the scheduler. The scheduler must not schedule new work onto workers that own transient jobs (their status is unknown — the job may have completed, crashed, or still be running). When the owning worker (the specific worker that held the job at promotion time, identified by job_t.exec_host) reconnects via handle_worker_join(), it re-reports its current job status and the job transitions from JOB_STAT_TRANSIENT to its actual state (JOB_STAT_RUN, JOB_STAT_EXIT, JOB_STAT_DONE, etc.). Only the owning worker's reconnect clears JOB_STAT_TRANSIENT for that specific job — reconnection of any other worker does not affect transient jobs on different workers. reconciliation_progress is a monitoring convenience metric (aggregate reconnect count); it is not the mechanism that resolves individual transient jobs. This guarantees eventual consistency: no job state is permanently lost, only temporarily uncertain until the owning worker reconnects. If a worker never reconnects (hardware failure, network isolation), jobs on that worker remain transient until an operator intervenes — this is the same as today's behavior when a worker goes WORKER_UNAVAILABLE.

Stale scheduler state: The scheduler maintains internal state derived from controller state (resource allocations, pending job counts per queue, scheduling constraints). This internal state is also stale after failover. The scheduler heals by re-initializing from scratch when it reconnects to the promoted primary via xce_reconnect(): on reconnect, the scheduler sends a full state-sync query to enumerate all PEND and RUN jobs, rebuilds its allocation table, and resumes scheduling from a clean slate. Admin-initiated changes (queue configuration changes, admin holds on jobs) are WAL_QUEUE_CTRL / WAL_WORKER_CTRL — ASYNC-replicated events and subject to the same loss window as job lifecycle events. If the old primary crashed before these entries were replicated to the standby, they are absent from the promoted standby's WAL. The scheduler must re-query all state (job, queue, and worker configuration) on reconnect — it cannot selectively trust queue/worker configuration as more reliable than job lifecycle state. Both are subject to the async replication lag at the moment of failure.

Reconciliation window definition: The reconciliation window is the interval between promotion (step 2) and the point where all workers that were connected to the old primary have re-joined the promoted primary. Scheduler behavior during the reconciliation window: (1) New job submits are accepted normally — these have authoritative state. (2) Scheduling of PEND jobs onto reconnected workers proceeds normally. (3) Scheduling of PEND jobs onto unreconnected workers is blocked — the scheduler does not know whether those workers still have capacity (transient jobs may still be running). (4) The scheduler tracks a reconciliation_progress metric: (workers_reconnected / workers_at_promotion_time) where workers_at_promotion_time = count of workers in the promoted standby's kvmgr with status NOT including XCE_WORKER_UNAVAIL or XCE_WORKER_HIDDEN at the moment promotion completes (step 2). This count is derived from the standby's own replicated state — no data from the dead primary is needed. (5) Once reconciliation_progress >= 1.0 (all workers have reconnected), normal scheduling resumes on all workers. Expected window duration: 30–120 seconds for a healthy cluster (workers reconnect within one worker_heartbeat_interval, typically 15–30 seconds, plus one scheduling cycle for the scheduler to catch up). If workers are slow to reconnect (e.g., due to DNS or SSL re-handshake latency), the window extends. If a worker never reconnects, its jobs remain in JOB_STAT_TRANSIENT until the operator intervenes — these workers are excluded from reconciliation_progress after a configurable worker_reconnect_timeout_s (default: 300 seconds). Stuck-worker alarm: when any worker hits the worker_reconnect_timeout_s boundary without reconnecting, log at WARN: "Worker %s did not reconnect within %us after failover — %d transient jobs on this worker require manual resolution" and expose ctrl_reconciliation_stuck_workers gauge (count of workers excluded from reconciliation_progress due to timeout). This gives operators a concrete list of workers requiring investigation rather than requiring them to correlate ctrl_transient_jobs_stuck with worker connection state.

5.7 Term (Fencing Token)

The term field prevents split-brain:

• Incremented on every promotion
• Included in every WAL entry and replication message
• A standby receiving a message with a lower term than its own ignores it
• Workers/scheduler reconnect and verify term on join

Partition heal — old primary returns: When a partitioned primary recovers network connectivity, it resumes sending heartbeats and replication messages with its old (lower) term. The promoted primary (higher term) ignores these messages (term < own term rule above). The partitioned primary receives a heartbeat reply or replication message from the promoted primary carrying the higher term. On receiving any message with term > ha.term, the old primary must detect that it has been superseded and demote itself: set ha.role = CTRL_STANDBY, stop accepting writes, and reconnect to the promoted primary as a standby. The handle_msg() path for all incoming messages must check hdr.term > ha.term and call ha_demote_to_standby() when this condition is detected, then re-process the message in standby mode. This prevents the old primary from continuing to accept writes after the partition heals, eliminating the split-brain write window.

Asymmetric partition — old primary never receives higher-term message: If the partition only partially heals (old primary can reach workers and API servers but NOT the promoted standby), the old primary continues accepting job submits indefinitely — it never receives a higher-term message and cannot self-demote. This is the unbounded split-brain scenario acknowledged in Part 5.1. Mitigation: the promoted primary sends heartbeats to the old primary's address (via the pre-iomgr mechanism in step 5.5 and the iomgr pipeline after step 8). If the asymmetric partition severs primary→standby but allows standby→primary traffic, the old primary will eventually receive a heartbeat-reply with the new term. If the partition is bidirectionally severed between the two controllers, no automatic resolution exists — operator intervention is required (SIGUSR1 manual failover on the old primary, or force-restart it). The ctrl_split_brain_detected metric captures this condition once any cross-controller message is received after partition heal.

Hook disabling on demotion: ha_demote_to_standby() must disable the persistence hooks that were registered during promotion (step 5 above). The chosen approach is disable-in-place via xce_disable_hook() (to be added to src/lib/hook.c) rather than true removal.

Memory ordering: hook_item_t.fn must be declared as _Atomic hook_fn_t (C11 atomic function pointer). xce_disable_hook() stores NULL with atomic_store_explicit(memory_order_release). xce_run_hooks() loads fn into a local variable with atomic_load_explicit(memory_order_acquire), then checks and calls the local — two reads of the same non-atomic variable can diverge on weakly-ordered architectures, but a single atomic load into a local is a single read with a visibility guarantee.

Slot reuse to bound list growth: On each promote→demote cycle, a NULL slot is left in the hook list and a new slot is added on re-promote. To prevent unbounded growth, xce_add_hook() must scan for an existing NULL fn slot of the same type before appending a new entry. If a matching NULL slot is found, it reuses it (sets fn and data). This bounds the hook list to max(ever-registered) slots regardless of cycle count.

Disabling must happen before the role switch to CTRL_STANDBY so that no in-flight requests trigger hooks after the role has changed. The hooks_registered idempotency guard (see step 5) is reset to 0 on disable so that the next promote re-registers correctly. The role-check defense-in-depth (checking ha.role == CTRL_PRIMARY inside each hook function before proceeding) provides a second layer: even if a hook entry fires during the disable window, it does nothing because the role has already switched to STANDBY.

In-flight request window during demotion (CRITICAL): Between the higher-term message arriving at handle_msg() and the demotion taking effect, vcpu threads may already be executing handle_jobsubmit() past the AFTER_EVENT hook. These in-flight requests have the following risks:

1. The persistence hook already called persist_wal_sync_append(), which is writing to the local WAL and may have already sent repl_broadcast(). Standbys following the higher-term primary will ignore the lower-term replication message.
2. After ha_demote_to_standby() sets ha.role = CTRL_STANDBY, the vcpu thread continues executing and eventually calls xce_reply(), ACKing the client for a job that is NOT replicated to the new primary's standbys.

Mitigation — role check in hot paths: To bound this window, both the persistence hook and repl_broadcast() must check ha.role before proceeding:

• persist_job_hook_fun(): before calling persist_wal_sync_append(), check (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_PRIMARY. If ha.role != CTRL_PRIMARY, return -E_XCE_NOT_PRIMARY without writing WAL or replicating. The caller (handle_jobsubmit) treats this as a failed submit and replies to the client with an error.
• repl_broadcast(): before sending to any standby, check the primary's (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_PRIMARY. If the role has changed, skip the broadcast and return an error. These checks use atomic_load_explicit(&c->ha.role, memory_order_acquire) — ha.role must be an atomic field (or at minimum guarded by a compiler barrier) so that the vcpu thread sees the demotion promptly. The corresponding write in ha_demote_to_standby() must use atomic_store_explicit(&c->ha.role, CTRL_STANDBY, memory_order_release) — a plain write (ha.role = CTRL_STANDBY) does not pair with the acquire load and may not be visible to vcpu threads in time, especially on weakly-ordered architectures or with compiler optimizations that cache the value in a register. The release-acquire pair is essential for the in-flight window mitigation to work correctly. Note: these checks are defense-in-depth — the primary mechanism is the handle_msg() term check. The in-flight window is bounded by the time between ha_demote_to_standby() setting the role and the vcpu thread reaching the next role check, which is at most one sync replication timeout cycle (sync_timeout_ms, default 50ms). Any client ACK sent during this window is acknowledged as a split-brain risk (see Part 5.1).

5.8 Non-Preemptive Failover

Time ─────────────────────────────────────────────▶

ctrl-1 (P=0):  ████████░░░░░░████████████████████████
                PRIMARY  DEAD     PRIMARY (recovered,
                         │         does NOT reclaim)
                         │
ctrl-2 (P=1):  ░░░░░░░░██████████████████████████████
                STANDBY  PRIMARY (promoted at T1)

When ctrl-1 recovers:

1. It probes for candidates with priority < 0 → none exist
2. It becomes primary? No — it probes ctrl-2 and finds it alive
3. ctrl-1 stays standby and connects to ctrl-2 as primary
4. Admin can trigger manual failover to restore optimal controller placement:

Manual failover protocol: To move primary back to a higher-priority candidate (e.g., ctrl-1 after recovery), the operator sends SIGUSR1 to the current primary (ctrl-2). The SIGUSR1 signal handler must only set an atomic flag: atomic_store(&ha->failover_requested, 1). It does NOT call ha_demote_to_standby() directly — that function performs operations (mutex locks, fsync, SSL writes, memory allocation) that are not async-signal-safe per POSIX. The failover is executed by the dedicated HA watchdog thread (the same pthread running ha_check_heartbeat() — P1-3), which polls ha->failover_requested on each tick. When the flag is observed, the watchdog thread calls ha_demote_to_standby() from its safe thread context, which: (a) unregisters persistence hooks (see Part 5.7 hook unregistration), (b) sets ha.role = CTRL_STANDBY using atomic_store_explicit with memory_order_release, (c) stops accepting new writes, (d) flushes and sync-replicates any pending WAL entries, (e) sends a OP_steppingDown message to all connected standbys with its current term and the election deadline. On receiving OP_steppingDown, the highest-priority connected standby (ctrl-1) immediately begins its election (calls ha_promote_to_primary()). The stepping-down controller then reconnects to the newly promoted primary as a standby. During the transfer window (~1-2 election_timeout_ms), no controller is primary — job submits receive EAGAIN and must retry. This is the same brief unavailability as an automatic failover. The SIGUSR1 approach avoids needing a dedicated admin RPC or CLI command in the initial implementation; a future xce hostcontrol --failover CLI can wrap this by sending the signal remotely via SSH. Add atomic_int failover_requested; to ha_state_t for this purpose.

SIGUSR1 access control note: Signal delivery to a process requires the sender to have the same UID (or be root). The controller runs as a dedicated service account; on HPC management nodes (login nodes, admin nodes), where multiple user accounts may exist, any process running as the controller's service account can trigger a failover. Mitigation (in scope for P1-3a): add --disable-sigusr1-failover as a startup flag (single atomic_int sigusr1_failover_enabled field checked in the signal handler before setting failover_requested). Operators on shared management nodes should deploy with this flag set; xce hostcontrol --failover (SSH-authenticated) becomes the only failover path once available. The flag is one additional line in the signal handler and should ship with P1-3a rather than being deferred to future work.

Split-brain demotion audit log: When ha_demote_to_standby() is triggered by receiving a higher-term message (partition heal path — NOT the SIGUSR1 path), it must log the range of WAL entries this controller generated during the split-brain window before discarding them: ERROR("Split-brain demotion: discarding local WAL entries lsn=[%lu..%lu] (term=%lu) — these %lu job submits were ACKed to clients but are NOT durable. Operators: treat job IDs in this LSN range as lost. ctrl_split_brain_lost_submits+=%lu", divergence_start_lsn, current_lsn, ha->term, entry_count, entry_count). divergence_start_lsn is the LSN at which this controller's term diverged from the promoted primary's term (recorded when this controller first started accepting writes after the partition began). Increment ctrl_split_brain_lost_submits counter by entry_count. This log gives operators a bounded audit trail without requiring a quorum protocol or external log aggregation.

Demotion failure handling: If persist_flush() fails during ha_demote_to_standby() (step d above), the controller has already set ha.role = CTRL_STANDBY and incremented term. The safest recovery is to not rollback the role change — instead, log a FATAL("persist_flush failed during demotion — WAL may be incomplete. Manual investigation required."). The demoted controller will not accept new writes (role is STANDBY), and the WAL will be flushed again on the next checkpoint. A role rollback (setting ha.role = CTRL_PRIMARY on flush failure) would leave the cluster in a split-brain-eligible state where the original primary continues accepting writes while a new primary may already be promoting. Failing forward as STANDBY is safer: the cluster temporarily has no primary until another standby wins election, which is the same state as a normal failover.

5.9 HA Connection Security (Trusted LAN)

XCE controllers run on a dedicated management LAN. All HA connections (primary→standby replication, standby→primary heartbeat, election probes) use the same iomgr/SSL pipeline as worker/scheduler connections, providing transport encryption. No mutual TLS (mTLS) client certificate verification is required between controllers.

Trust model: The management network is the security boundary. Only nodes on the management LAN can reach controller HA ports. This is the same model used by traditional HPC middleware (LSF, SLURM, PBS) — network-level access control (firewall rules, dedicated VLAN) replaces PKI-based peer authentication.

• Election probe: The initial am_i_primary() probe (before iomgr is started — see Part 5.4 startup sequencing note) launches one pthread per lower-priority candidate, each making a synchronous blocking TCP/SSL dial. All threads share a single election_timeout_ms deadline; the first live response wins. Peer identity is confirmed by host:port matching against the --peers candidate list. After the election resolves, all subsequent HA traffic uses the iomgr/SSL pipeline for heartbeat and replication.
• No PKI requirement: HA does not require a cluster CA or controller-specific certificates. The existing SSL infrastructure (used for worker/scheduler connections) is reused for controller-to-controller channels via iomgr. Election probe SSL context: The pre-iomgr election probes (Part 5.4) cannot reuse the iomgr SSL context because iomgr has not yet started. Each probe thread creates a standalone SSL_CTX using the same cert/key pair as the iomgr context. Extract a shared helper ssl_ctx_create(cert_path, key_path) (or equivalent) used by both the election probe threads and iomgr initialization to avoid code duplication. The election probe SSL contexts must be freed after all probe threads join (pthread_join), before cluster_start() initializes iomgr's own SSL context. No mTLS or PKI changes are required — the same cert/key pair is used throughout.
• Optional enhancement — SSL_VERIFY_PEER for controller-to-controller channels: The current design validates peer identity by host:port matching against --peers. Enabling one-way TLS server certificate CN/SAN verification (SSL_CTX_set_verify(SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT)) on the HA client side would convert this network-layer assertion to a cryptographic one — at no additional PKI cost since the same cert/key infrastructure already exists. This is not required for the trusted-LAN model but would be a low-effort P3 hardening step (P3-8): configure SSL_CTX_set_verify on the election probe SSL context and the iomgr HA connection path; verify the server cert's CN or SAN matches the candidate's configured hostname. The --ca-cert flag added for libxce (P2.5-5) would serve this purpose. Deploy only if the management network boundary cannot be enforced via firewall/VLAN.

5.10 New Opcodes and Cap'n Proto Additions

/* In HostRole enum: */
standby @5;  /* NEW — standby controller role */

/* In OPCode enum (existing entries go up to @21 — clrpreason):
 * @22 is intentionally skipped in the OPCode enum to leave room for a future
 * non-HA opcode. The new HA opcodes start at @23. */
replEntry       @23;   /* Replicated WAL entry (primary → standby) */
replAck         @24;   /* ACK for sync-replicated entry (standby → primary) */
redirect        @25;   /* Redirect to current primary (standby → client) */
heartbeat       @26;   /* Primary heartbeat to standbys (primary → standby) */
steppingDown    @27;   /* Primary is stepping down (primary → standby, triggered by SIGUSR1) */
replSnapshot    @28;   /* Snapshot transfer for catch-up (primary → standby, chunked) */
replCatchUpDone @29;   /* Signal that catch-up snapshot is loaded (standby → primary) */

/* In Request struct (struct ordinals are separate from OPCode enum values):
 * @22 here is the Request struct field ordinal — it does NOT conflict with
 * the OPCode @22 gap above. These are different namespaces in Cap'n Proto. */
term @22 :UInt64 $C.mapname("term");  /* NEW — fencing token */

---

Part 6: Fan-Out TCP Replication

6.1 Topology Choice: Fan-Out vs Chain

Factor	Fan-Out	Chain
Latency to all standbys	max(RTT_1..N) — single hop	RTT_1 + RTT_2 + ... RTT_N — cumulative
Primary load	N non-blocking sends	1 send (lowest load)
Failure isolation	Standby failure only affects that standby	One slow standby blocks the chain
Implementation	Simple	Must handle chain reordering
At XCE scale (~3 standbys)	Primary sends 3 copies — trivial	Marginal load savings

Decision: Fan-out. At 3 standbys, the primary does 3 non-blocking send() calls per WAL entry — negligible overhead. The latency advantage (single hop) and failure isolation are decisive.

6.2 Replicator Data Structure

/* MAX_STANDBYS and MAX_HA_CANDIDATES defined in include/config.h (see Appendix A.10) */

**Constant reconciliation**: The existing `src/include/client.h` defines `N_MAX_MASTERS = 3` for the `xce_client_config_t.masters[]` array. Both `MAX_HA_CANDIDATES = 4` and `MAX_STANDBYS = 3` are defined directly in `include/config.h` (see Out of Scope — `MAX_HA_CANDIDATES` reduced from 16 to 4, `MAX_STANDBYS` reduced from 15 to 3). `MAX_STANDBYS` is defined as a standalone constant (`= 3`) in `config.h`, NOT derived as `MAX_HA_CANDIDATES - 1` in `repl.h` — there is one source of truth. The deployment ceiling advisory (≤3 controllers) aligns with `N_MAX_MASTERS = 3` which was already chosen for that purpose. The Go CGo bindings see both constants — they must be clearly separated in header exports.

typedef struct {
    endpoint_t*  ep;            /* Full endpoint (SSL, FSM, outq) — uses iomgr/vtask pipeline */
    uint64_t     acked_lsn;    /* Last LSN this standby has ACKed. ACCESS RULE: on 64-bit
                                   platforms (LP64), acked_lsn is read/written atomically as a
                                   single 8-byte value. On ILP32 or any platform where uint64_t
                                   is not naturally atomic, this field MUST be accessed under
                                   repl.ack_lock. All write paths (repl_on_ack, standby
                                   disconnect tracking) and all read paths (repl_wait_ack
                                   predicate, WAL trim guard min-over-standbys) must use the
                                   same lock or use _Atomic uint64_t with acquire/release
                                   ordering. The recommended implementation is: use C11
                                   _Atomic uint64_t for acked_lsn with relaxed loads for
                                   advisory reads (lag monitoring, trim guard) and
                                   release/acquire for the ack signal path. Alternatively,
                                   gate ALL reads and writes through repl.ack_lock. */
    int          lagging;      /* Flag: standby is behind threshold */
    char         host[XCE_MAXNAME_LEN];
    uint32_t     port;
} standby_conn_t;

typedef struct {
    standby_conn_t  standbys[MAX_STANDBYS];
    int             nstandbys;
    mylock_t        lock;         /* Protects standbys array */
    uint64_t        pending_lsn;  /* LSN we're waiting ACK for (sync mode) */
    pthread_cond_t  ack_cond;     /* Signaled when ACK arrives */
    pthread_mutex_t ack_lock;     /* Protects ack_cond + pending_lsn. MUST be pthread_mutex_t
                                   * (not mylock_t) because pthread_cond_timedwait requires a
                                   * pthread_mutex_t* — same rationale as checkpoint_mutex. */
} replicator_t;

6.3 Replication Flow

Primary                              Standby-1    Standby-2
  │                                      │              │
  │  WAL write + fsync                   │              │
  │──────────────────────────────────────│              │
  │                                      │              │
  │  repl_broadcast(entry)               │              │
  │─── send(WAL_ENTRY) ─────────────────▶│              │
  │─── send(WAL_ENTRY) ─────────────────────────────────▶│
  │                                      │              │
  │  [for SYNC mode only]                │              │
  │  repl_wait_ack(target_lsn, timeout)  │              │
  │                                      │              │
  │◀── ACK(acked_lsn) ──────────────────│              │
  │◀── ACK(acked_lsn) ─────────────────────────────────│
  │                                      │              │
  │  Condition signal → resume           │              │
  │                                      │              │

6.4 Standby WAL Receiver

IMPLEMENTATION GUARD: The pseudocode below is a logical sketch of WAL entry processing on the standby. The actual implementation must use the endpoint/iomgr/vtask SSL pipeline (see Part 6.5 and Part 6.6). Do NOT implement standalone receive-loop threads with raw read()/write() on file descriptors. All controller connections — including replication — go through the iomgr event loop. A raw-socket implementation would bypass SSL enforcement, per-endpoint message ordering, and backpressure; any such implementation must be rejected in code review. The standby_recv_thread() function below is pseudocode only — replace with handle_msg() dispatch for OP_replEntry in the iomgr/vtask pipeline.

Serialization note: apply_wal_entry() on the standby (including xkvmgr_add() calls inside it) is effectively single-threaded. All OP_replEntry messages arrive over a single primary→standby endpoint; the iomgr/vtask pipeline's endpoint_block_tasks()/endpoint_unblock_tasks() mechanism ensures messages on a given endpoint are dispatched to one vcpu thread at a time and in order. Therefore apply_wal_entry() does not require additional locking beyond what the functions it calls already provide for the normal (non-COW) path.

The standby processes replication entries through its iomgr event loop:

/* On standby: receive WAL entries from primary
 * NOTE: This is PSEUDOCODE showing the logical processing steps.
 * The actual implementation uses iomgr/vtask/endpoints — see Part 6.5. */
static void* standby_recv_thread(void* arg)
{
    cluster_t* c = (cluster_t*)arg;
    ha_state_t* ha = &c->ha;

    for (;;) {
        wal_entry_hdr_t hdr;
        void* payload;

        /* Read WAL entry from primary (pseudocode — actual: iomgr recv via ha->primary_ep) */
        if (read(ha->primary_ep->fd, &hdr, sizeof(hdr)) != sizeof(hdr))
            break;  /* Primary connection lost */

        if (hdr.payload_len > MAX_WAL_PAYLOAD_BYTES) {
            ERROR("replication: oversized payload %u > MAX_WAL_PAYLOAD_BYTES", hdr.payload_len);
            break;  /* Treat as protocol error — drop connection */
        }
        /* Guard against malloc(0) for WAL_CHECKPOINT entries (payload_len=0).
         * For zero-payload entries, skip malloc entirely — no read or free needed.
         * For non-zero payloads, always check for OOM. */
        payload = NULL;
        if (hdr.payload_len > 0) {
            payload = malloc(hdr.payload_len);
            if (payload == NULL) {
                ERROR("replication: OOM allocating %u bytes for lsn=%lu",
                      hdr.payload_len, hdr.lsn);
                break;  /* Disconnect — cannot safely process entry */
            }
            if (read(ha->primary_ep->fd, payload, hdr.payload_len) != hdr.payload_len) {
                free(payload);
                break;
            }
        }

        /* Validate term — reject stale entries but allow multi-promotion jumps.
         * A primary term lower than standby's term means the standby has a
         * higher term (possibly from a previous self-promotion or a different
         * primary) — reject the entry, do not advance applied_lsn, and
         * continue processing. The standby should not follow a lower-term primary.
         * NOTE: ha->term is _Atomic uint64_t — all reads use atomic_load_explicit. */
        if (hdr.term < atomic_load_explicit(&ha->term, memory_order_acquire)) {
            free(payload);
            continue;
        }
        /* Accept any term >= ha->term, bounded by MAX_TERM_JUMP. During a
         * multi-promotion partition, the primary's term may have advanced by
         * more than 1 relative to the standby's ha->term. Instead of
         * disconnecting, update ha->term to match the primary's term — the
         * primary is the authoritative term leader.
         *
         * Security: cap term jumps at MAX_TERM_JUMP (default: 10000) to prevent
         * a corrupted or malicious packet from setting ha->term to UINT64_MAX,
         * which would permanently lock out legitimate primaries and cause
         * term wraparound (UINT64_MAX + 1 = 0) on next promotion. This is a
         * liveness invariant: the fencing token must never be driven to an
         * unusable value by a single bad packet.
         *
         * Accepted risk: MAX_TERM_JUMP prevents single-packet term exhaustion
         * but NOT cumulative drift. An attacker on the management LAN (which
         * is the trust boundary — see Part 5.9) could send repeated packets to
         * advance ha->term by MAX_TERM_JUMP per packet, eventually reaching
         * UINT64_MAX. Given the trusted-LAN security model (network-level ACLs
         * restrict access), this cumulative attack requires sustained access to
         * the management network and is accepted as an out-of-scope risk. mTLS
         * peer authentication (Part 5.9) would eliminate this attack surface. */
        #define MAX_TERM_JUMP 10000
        if (hdr.term > atomic_load_explicit(&ha->term, memory_order_acquire) + MAX_TERM_JUMP) {
            ERROR("replication: term too high: got %lu (max allowed: %lu) — "
                  "possible corrupted packet or tampering",
                  hdr.term, atomic_load_explicit(&ha->term, memory_order_acquire) + MAX_TERM_JUMP);
            free(payload);
            break;  /* Disconnect — term is implausible */
        }
        if (hdr.term > atomic_load_explicit(&ha->term, memory_order_acquire) + 1) {
            WARN("replication: term jump from %lu to %lu (multi-promotion partition heal)",
                 atomic_load_explicit(&ha->term, memory_order_acquire), hdr.term);
            atomic_store_explicit(&ha->term, hdr.term, memory_order_release);
        }

        /* Validate LSN sequence — entries must arrive in strictly increasing order.
         * A repeat means the primary re-sent without a term bump (unexpected).
         * A backward LSN is always an error.
         * A forward gap is expected for WAL_CHECKPOINT (type=100) entries, which are
         * local-only markers not replicated to standbys — they create gaps in the
         * LSN sequence from the standby's perspective. Therefore the check is
         * strictly increasing (hdr.lsn > applied_lsn), NOT strictly consecutive
         * (hdr.lsn == applied_lsn + 1). A missing-entry gap (network loss) is
         * detected by CRC chain validation, not LSN sequence — and a true gap
         * (more than one missing entry) indicates a network issue requiring
         * reconnection and catch-up. A single-entry gap (typical for a skipped
         * WAL_CHECKPOINT) is benign and must be accepted.
         *
         * Note: when applied_lsn == 0 (standby has not yet applied any entry),
         * both guards are skipped — the first replicated entry is accepted with
         * no LSN validation whatsoever. This is correct because (a) the first
         * entry arrives via catch-up which establishes a known starting LSN, and
         * (b) catch-up mode already performs CRC chain validation on the
         * transferred entries. Any gap in the initial stream is caught by the
         * catch-up protocol, not by the normal-mode LSN check. */
        uint64_t applied = atomic_load_explicit(&ha->applied_lsn, memory_order_acquire);
        if (applied != 0 && hdr.lsn <= applied) {
            ERROR("replication: LSN sequence error: got %lu, expected > %lu",
                  hdr.lsn, applied);
            free(payload);
            break;
        }
        if (applied != 0 && hdr.lsn > applied + 1) {
            /* LSN gap — distinguish single-skip (checkpoint marker) from multi-skip (data loss).
             *
             * Single-skip (hdr.lsn == applied + 2): most likely a skipped WAL_CHECKPOINT
             * (type=100) which is local-only and not replicated. Log WARN and continue —
             * the CRC chain validation below provides a second safety net.
             *
             * Multi-skip (hdr.lsn > applied + 2): indicates actual entry loss (network
             * packet loss, buffer overflow, or protocol bug). WAL_CHECKPOINT is a single
             * entry; two consecutive checkpoint skips would still produce a gap of 2 per
             * checkpoint, not a gap > 2 in a single replication step. Disconnect and
             * trigger catch-up — accepting multi-entry gaps silently would allow data loss
             * to go undetected even if the CRC chain restarts (e.g., at a segment boundary).
             */
            if (hdr.lsn == applied + 2) {
                WARN("replication: single LSN skip: expected %lu, got %lu (likely WAL_CHECKPOINT marker)",
                     applied + 1, hdr.lsn);
                /* continue — CRC chain below provides second validation */
            } else {
                ERROR("replication: multi-entry LSN gap: expected %lu, got %lu — possible entry loss, disconnecting for catch-up",
                      applied + 1, hdr.lsn);
                goto disconnect_for_catchup;
            }
        }

        /* Validate chained CRC — detect missing/reordered entries that LSN alone
         * cannot catch (since LSNs are "strictly increasing, not consecutive").
         * Each entry's prev_crc must match the CRC of the immediately preceding
         * received entry. On the first entry (applied_lsn == 0), prev_crc must
         * be WAL_CRC_NONE (established by catch-up protocol).
         * This mirrors the on-disk chained-CRC validation (Part 2.4) but operates
         * on the live replication stream. Without it, a reordered or replayed entry
         * with a valid per-entry CRC but wrong position in the chain would pass
         * the LSN check (since LSN gaps are expected) and corrupt the standby's
         * in-memory state and local WAL.
         *
         * On segment boundaries the primary sends entries with prev_crc ==
         * WAL_CRC_NONE as the first entry of the new segment. The standby treats
         * this as a valid chain break (same as on-disk recovery, Part 2.4). */
        if (atomic_load_explicit(&ha->applied_lsn, memory_order_acquire) == 0) {
            /* First entry after catch-up: prev_crc must be WAL_CRC_NONE */
            if (hdr.prev_crc != WAL_CRC_NONE) {
                ERROR("replication: first entry after catch-up has non-NULL prev_crc %u",
                      hdr.prev_crc);
                free(payload);
                break;
            }
        } else {
            /* Normal mode: prev_crc must match the CRC of the last received entry.
             * NOTE: WAL_CHECKPOINT entries are written to the primary's WAL but NOT
             * replicated. The primary maintains two separate CRC trackers: last_crc
             * (updated for every WAL entry including checkpoints) and
             * last_replicated_crc (updated only for replicated entries, never for
             * WAL_CHECKPOINT). The post-checkpoint replicated entry therefore carries
             * prev_crc = last_replicated_crc, which equals the CRC of the last
             * non-checkpoint WAL entry — not the CRC of the checkpoint entry itself.
             * Since ha->last_received_crc on the standby also tracks only replicated
             * entries, the chain remains unbroken across checkpoint boundaries.
             * See Part 2.3 for last_replicated_crc field documentation. */
            if (hdr.prev_crc != ha->last_received_crc && hdr.prev_crc != WAL_CRC_NONE) {
                ERROR("replication: chained CRC mismatch: expected %u, got %u, lsn %lu "
                      "(possible reordered/missing entry)",
                      ha->last_received_crc, hdr.prev_crc, hdr.lsn);
                free(payload);
                break;  /* Disconnect; primary may have diverged — reconnect + catch-up will resync */
            }
            /* WAL_CRC_NONE in normal mode (applied_lsn > 0): accepted as a valid
             * chain break. The code above does NOT disconnect when prev_crc ==
             * WAL_CRC_NONE — it passes the `if` condition silently. There are three
             * legitimate sources of WAL_CRC_NONE in normal mode:
             *
             * 1. Segment rotation: the primary MUST emit prev_crc = WAL_CRC_NONE for
             *    the first entry of each new WAL segment (per Part 2.4 writer-side
             *    obligation). This creates a chain break in the replication stream
             *    whenever a segment rotation occurs during live replication.
             *
             * 2. Post-promotion: when a standby promotes to primary, it resets
             *    last_replicated_crc = WAL_CRC_NONE (per Part 5.6 step 3). The first
             *    entry it broadcasts carries prev_crc = WAL_CRC_NONE. Already-connected
             *    standbys with applied_lsn > 0 accept this as a term-change chain break.
             *
             * 3. After catch-up: the first live entry after snapshot application is
             *    handled by the applied_lsn == 0 check above (not this branch).
             *
             * The term check earlier in this function guards against accepting entries
             * from a stale primary. WAL_CRC_NONE itself is not an error indicator —
             * the CRC chain exists to detect missing or reordered entries, and
             * WAL_CRC_NONE explicitly signals a legitimate restart of the chain. */
        }

        /* Validate CRC — detect corruption in replication path.
         * WAL_HDR_CRC_SIZE is defined in persist.h (Part 2.2) — covers all
         * header fields except crc and prev_crc.
         * A static_assert ensures sizeof(wal_entry_hdr_t) == 40 so this
         * computation remains correct regardless of field ordering. */
        uint32_t computed_crc = crc32(0, (void*)&hdr, WAL_HDR_CRC_SIZE);
        computed_crc = crc32(computed_crc, payload, hdr.payload_len);
        if (computed_crc != hdr.crc) {
            /* Corrupted entry — log and skip */
            ERROR("replication CRC mismatch: expected %u, got %u, lsn %lu",
                  hdr.crc, computed_crc, hdr.lsn);
            free(payload);
            continue;
        }

        /* Write to local WAL.
         * WAL_JOB_SUBMIT (SYNC) entries must be fsynced before ACK so the
         * primary's durability guarantee holds on the standby as well — if the
         * standby crashes before fsync, the primary's ACK to the client was
         * premature. All other entry types use async (fire-and-forget) writes.
         *
         * LSN handling: replicated entries use the PRIMARY's LSN and CRC — the
         * standby does NOT assign local LSNs. This is critical because:
         *   (a) After promotion, the former standby must serve catch-up to other
         *       standbys using the original LSN space (applied_lsn).
         *   (b) The WAL trim guard uses min(acked_lsn) across standbys — if
         *       LSNs were remapped locally, the trim guard would trim entries
         *       that a lagging standby still needs.
         *   (c) The chained CRC chain must match the primary's CRC values.
         *
         * The standby's persist_t.current_lsn must be updated to hdr.lsn after
         * each replicated write so the next locally-generated entry (after
         * promotion) gets hdr.lsn + 1. See persist_wal_replicated_entry() below.
         *
         * CRITICAL: Always check the return value of persist_wal_sync_append().
         * If the standby's local WAL write fails (disk full, I/O error), the
         * entry must NOT be applied, applied_lsn must NOT advance, and NO ACK
         * may be sent. A failed fsync with an ACK would violate the sync
         * replication durability guarantee — the primary would believe the
         * standby has the entry durably, but it is not on disk. */
        if (hdr.type == WAL_JOB_SUBMIT) {
            int rc = persist_wal_replicated_entry(&c->persist, &hdr, payload);
            if (rc != 0) {
                ERROR("replication: standby WAL write failed lsn=%lu rc=%d — dropping entry, not sending ACK",
                      hdr.lsn, rc);
                free(payload);
                break;  /* Disconnect; the primary will re-send on reconnect */
            }
        } else {
            if (persist_wal_replicated_entry(&c->persist, &hdr, payload) != 0)
                WARN("replication: async WAL write failed lsn=%lu — entry in page cache only", hdr.lsn);
        }

        /* Apply to in-memory state — MUST NOT fire AFTER_EVENT hooks.
         * apply_wal_entry() mutates state directly (same as persist_recover()).
         * If hooks fired here, the persistence hook would write another WAL
         * entry and send another replication message — an infinite loop.
         * Hooks are not registered on standbys (registered only on the
         * primary path after persist_recover() completes), so no bypass
         * flag is needed — the hook system is simply empty.
         *
         * Note: hdr.payload_len was already validated against MAX_WAL_PAYLOAD_BYTES
         * above (line 1210) before malloc. apply_wal_entry() applies the same bound
         * internally as defense-in-depth — it reads from the local WAL during
         * recovery (also user-controllable if disk is compromised).
         *
         * CRITICAL: Check the return value of apply_wal_entry(). If it fails
         * (OOM during deserialization, malformed payload), do NOT update
         * applied_lsn or last_received_crc, and do NOT send an ACK for sync
         * entries. The replication stream must be broken on apply failure —
         * the primary must re-establish with catch-up to ensure the standby's
         * state is consistent. */
        int rc = apply_wal_entry(c, &hdr, payload);
        if (rc != 0) {
            ERROR("replication: apply_wal_entry failed lsn=%lu rc=%d — breaking replication stream",
                  hdr.lsn, rc);
            free(payload);
            break;  /* Disconnect; catch-up will resync on reconnect */
        }

        atomic_store_explicit(&ha->applied_lsn, hdr.lsn, memory_order_release);
        ha->last_received_crc = hdr.crc;  /* Update chained CRC tracker */

        /* Send ACK for sync-replicated entries — only when not in catch-up mode.
         * During catch-up the standby is replaying historical entries; sending ACKs
         * then would prematurely signal readiness to the primary (see Part 6.6).
         * ha->in_catch_up is the STANDBY-LOCAL field in ha_state_t (set by the
         * standby when it enters catch-up, cleared when it sends OP_replCatchUpDone).
         * This is distinct from the PRIMARY-SIDE per-slot standby_conn_t.in_catch_up
         * (Appendix A.3), which is the primary's mirror of this field. */
        if (hdr.type == WAL_JOB_SUBMIT && !atomic_load_explicit(&ha->in_catch_up, memory_order_acquire)) {
            send_repl_ack(ha->primary_ep, hdr.lsn);
        }

        free(payload);
    }

    /* Primary connection lost — start monitor to check for promotion */
    ha_monitor_thread(c);
    return NULL;
}

6.5 Replication Uses the Endpoint/Iomgr/Vtask SSL Pipeline

All controller connections use SSL — the iomgr (netio.c) enforces this. Replication between controllers is no exception. The raw read/write calls in the standby receiver pseudocode above are simplified; the actual implementation must use the full endpoint/iomgr/vtask pipeline:

Standby                              Primary
  │                                     │
  │  iomgr_monitor(ep)                 │
  │  ─── endpoint established via SSL ──│
  │                                     │
  │  io_worker (epoll)                  │
  │    └─ read_fsm (SSL_read)           │
  │       └─ handle_msg()               │
  │          └─ vtask_handle_msg()      │
  │             └─ OP_replEntry          │
  │                └─ apply_wal_entry()  │
  │                └─ persist_wal_async()│
  │                                     │
  │  For SYNC: vtask sends OP_replAck   │
  │  └─ outq (per-endpoint serialized)  │
  │     └─ write_fsm (SSL_write)        │
  │       ──────────────────────────────▶│

Why the full pipeline?

1. SSL enforcement: All connections in netio.c go through SSL_new()/SSL_set_fd(). Creating a raw TCP socket for replication would bypass this requirement.
2. Per-endpoint serialization: The vtaskmgr endpoint_block_tasks()/endpoint_unblock_tasks() mechanism ensures that messages on a given endpoint are processed in order. This guarantees that WAL entries from the primary are applied in order on the standby.
3. Backpressure: If the standby is slow (e.g., during checkpoint serialization), the endpoint's outq fills up, and io_worker naturally applies backpressure via the write FSM. Raw TCP would require a separate flow-control mechanism.
4. Connection management: Reconnection, heartbeat, and error handling all reuse the existing handle_connection()/handle_endpoint_error() code path.

standby_conn_t update: Replace int fd with endpoint_t* ep — the standby connection is a full endpoint managed by iomgr, not a raw file descriptor.

typedef struct {
    endpoint_t*  ep;            /* Full endpoint (SSL, FSM, outq) — not raw fd */
    uint64_t     acked_lsn;    /* Last LSN this standby has ACKed */
    int          lagging;      /* Flag: standby is behind threshold */
    char         host[XCE_MAXNAME_LEN];
    uint32_t     port;
} standby_conn_t;

The primary initiates the replication connection as a client (using iomgr's outgoing connection support), and the standby accepts it via its own iomgr. The OP_replEntry and OP_replAck opcodes flow through the same SSL/endpoint/vtask pipeline as all other controller messages.

The standby_recv_thread pseudocode above would instead be handled by the iomgr event loop: the standby's handle_msg() dispatches OP_replEntry to vtask_handle_msg(), which applies the WAL entry and sends OP_replAck back via the endpoint's outq.

6.6 OP_replAck Must Bypass vtaskmgr (ACK Deadlock Prevention)

OP_replAck is the only opcode that must be handled directly in the io_worker read callback (handle_msg()) instead of being dispatched to vtaskmgr. All other opcodes go through vtaskmgr.

Why: When sync replication is active, vtask threads executing handle_jobsubmit() block inside repl_wait_ack() waiting for an ACK. If OP_replAck were also dispatched via vtaskmgr, the incoming ACK message would sit in the vtask queue — but no vcpu thread would be free to process it (they are all blocked waiting for ACKs). This is a classic deadlock.

Implementation: handle_msg() must check the opcode before calling vtask_handle_msg():

/* handle_msg is the io_worker read callback. It receives a protocol message,
 * NOT a WAL entry header. The first field is the OPCode (see Part 5.10 for the
 * opcode enum: OP_replEntry=23, OP_replAck=24, OP_redirect=25, OP_heartbeat=26).
 * For OP_replEntry, the payload is a WAL entry (wal_entry_hdr_t + data).
 * Do not confuse the protocol opcode with wal_entry_type values (Part 2.3). */
void handle_msg(cluster_t* c, endpoint_t* ep, msg_hdr_t* hdr, void* payload) {
    if (hdr->opcode == OP_replAck) {
        /* Handle directly in io_worker — do NOT dispatch to vtaskmgr */
        repl_on_ack(&c->ha.repl, decoded_lsn);
        return;
    }
    vtask_handle_msg(c, ep, hdr, payload);
}

repl_on_ack() is safe to call from io_worker: it only updates acked_lsn and calls pthread_cond_signal() under repl.ack_lock, with no vtaskmgr interaction.

Standby catch-up after reconnection: When a standby reconnects to the primary (either after a network disruption or on initial startup), it must tell the primary its last applied_lsn so the primary can stream missing WAL entries. The catch-up protocol is:

1. Standby sends OP_replJoin with its applied_lsn and term
2. Primary validates the peer: The primary verifies that the peer's host:port appears in its candidate list (from --peers CLI flag). An OP_replJoin from an unknown endpoint is rejected with an error reply.
3. Primary checks if the standby's applied_lsn is still within available WAL segments (not yet trimmed by a checkpoint). If yes, stream entries from applied_lsn + 1 forward via OP_replEntry messages on the existing SSL endpoint pipeline
4. If the standby's applied_lsn is too old — i.e., applied_lsn < oldest_surviving_segment_start_lsn (the WAL segments needed for catch-up have been trimmed) — the primary transfers the latest snapshot via the same SSL endpoint pipeline using a new OP_replSnapshot opcode. The snapshot is chunked at MAX_WAL_PAYLOAD_BYTES per message: the first chunk includes a snapshot header { snapshot_lsn, total_size, chunk_seq=0, final=false }, subsequent chunks carry { chunk_seq, data }, and the last chunk sets final=true and includes a CRC32 of the full snapshot payload. The standby reassembles chunks into a buffer, verifies the CRC32, then applies the snapshot to its kvmgrs. Snapshot size validation on standby receive: The total_size field in the first chunk must not exceed MAX_SNAPSHOT_BYTES (default: 256 MB). If total_size > MAX_SNAPSHOT_BYTES, the standby logs an ERROR and tears down the replication connection — a compromised or buggy primary could otherwise exhaust the standby's memory by sending an unbounded snapshot. This mirrors the local recovery path's snapshot size validation (Part 2.5 step 2b). After snapshot application, the primary streams WAL entries from snapshot_lsn + 1 forward. No separate file transfer channel or scp is required — all data flows through the established SSL endpoint.
5. If the standby's term is higher than the primary's, the primary has stale data — the standby should not follow this primary (fencing)

Replication during catch-up: While the primary is streaming historical WAL entries (WAL replay or snapshot + WAL) to a catching-up standby, the primary pauses live OP_replEntry broadcasts to that standby until the standby sends OP_replCatchUpDone. This is the definitive design choice: live entries are NOT buffered in the catching-up standby's sendq during catch-up. Buffering live entries during a 256 MB snapshot transfer (10–30 seconds at disk speed) at 1000 submits/sec would require 10–30 MB of in-flight WAL in the sendq — unbounded memory growth with no application-level control. The simpler and safer design: the catching-up standby is excluded from repl_wait_ack() via in_catch_up gating until it sends OP_replCatchUpDone. After catch-up, the standby requests any WAL entries it missed during the catch-up window via a second WAL-only catch-up phase (the primary's WAL covers this gap since WAL trim is held back by the standby's join LSN). The standby does not send OP_replAck during catch-up — only after fully caught up and processing live entries in normal mode.

Catch-up receive ordering: (1) OP_replSnapshot chunks arrive first and must be fully received and CRC-verified before any new WAL entries are applied to the kvmgrs. (2) The primary must NOT send live OP_replEntry messages to the catching-up standby until the snapshot transfer is complete — the standby sends OP_replCatchUpDone (or equivalent signal) after successfully loading the snapshot. (3) After snapshot application, the primary streams WAL entries from snapshot_lsn + 1 forward, and the standby applies them in strict LSN order. Only after reaching the primary's committed_lsn does the standby begin sending OP_replAck and transition to normal replication. This ensures the standby's state is self-consistent at each phase boundary: snapshot-loaded → historical-WAL-replayed → caught-up.

Snapshot catch-up flow control: The OP_replSnapshot transfer sends the full snapshot in chunks of MAX_WAL_PAYLOAD_BYTES through the SSL endpoint pipeline. The primary sends chunks as fast as the endpoint pipeline allows, and the standby must receive, buffer, and reassemble all chunks before applying the snapshot. The existing iomgr backpressure mechanism (endpoint write FSM, outq buffering, SSL-level flow control via SSL_write partial writes) provides implicit flow control: if the standby's receive path is slow, the primary's endpoint outq fills, and subsequent SSL_write calls block or return partial writes, naturally throttling the sender. No additional application-level flow control is needed beyond the iomgr/SSL pipeline's built-in backpressure. io_worker pool sizing caveat: if the io_worker pool is too small, a slow snapshot transfer (256 MB at disk speed, 10–30 seconds) can starve the OP_replAck path for healthy standbys — sync replication ACKs may experience queue delays that exceed sync_timeout_ms, triggering spurious degraded-mode for job submits that should have been durably replicated. Operators should size the io_worker pool to at least MAX_STANDBYS + 2 workers (default: MAX_STANDBYS=3 catch-up endpoints + scheduler + 1 reserve). Add a startup log at INFO level that reports the configured io_worker count and the recommended minimum. A P3 improvement would be to rate-limit snapshot chunk sends per-endpoint using a configurable snapshot_bandwidth_limit_kbps parameter, leaving headroom for sync replication ACK traffic. However, operators should be aware that a snapshot transfer (up to MAX_SNAPSHOT_BYTES = 256 MB) temporarily increases the standby's memory usage for reassembly. The standby allocates a receive buffer of total_size bytes (validated against MAX_SNAPSHOT_BYTES at first chunk, per Part 6.6) before receiving subsequent chunks — this memory is freed after snapshot application. CRITICAL: validate total_size <= MAX_SNAPSHOT_BYTES BEFORE calling malloc(total_size). Do NOT allocate the buffer and then check — the check is the first operation on first-chunk receipt. Aggregate memory budget: With MAX_STANDBYS = 3, up to 3 standbys may catch up concurrently on the primary side. Each primary-side sendq for an in-flight snapshot transfer holds at most one pending chunk (MAX_WAL_PAYLOAD_BYTES ≈ 1 MB), so the primary's total per-catch-up memory is negligible. On the standby side, only one standby receives a snapshot at a time (a standby can only be catching up from one primary), so the worst-case standby-side allocation is a single total_size ≤ MAX_SNAPSHOT_BYTES buffer. The 768 MB = 3 × 256 MB scenario (all standbys catching up simultaneously) can only occur on the primary if it were somehow buffering outbound snapshots for all standbys at once — but the primary streams each snapshot through an already-established SSL endpoint, holding at most MAX_WAL_PAYLOAD_BYTES per chunk in the endpoint outq. Therefore the realistic aggregate memory bound is MAX_SNAPSHOT_BYTES per standby (one reassembly buffer) plus MAX_WAL_PAYLOAD_BYTES × MAX_STANDBYS on the primary for in-flight chunks.

Catch-up trigger: When a standby reconnects (or initially joins), it includes its applied_lsn in the heartbeat to the primary. The primary compares this against its available WAL segments and streams missing entries from applied_lsn + 1 forward. If the standby's applied_lsn is older than the primary's oldest surviving WAL segment, the primary initiates snapshot transfer (see OP_replSnapshot framing above) before streaming WAL entries.

MAX_WAL_PAYLOAD_BYTES: The maximum payload size per WAL entry (and per OP_replEntry message) is 1 MB (1,048,576 bytes). This bounds the memory allocation in the replication receive path (line payload = malloc(hdr.payload_len) above) to a safe maximum. The largest expected payload is a job submit with a large environment (~64 KB typical, ~512 KB pathological). The 1 MB cap provides 2× headroom over the pathological case while preventing unbounded allocation from a corrupted or malicious entry. Defined in persist.h as MAX_WAL_PAYLOAD_BYTES (see Appendix A.1). A #ifndef include guard allows test override.

Cap'n Proto traversal limit: apply_wal_entry() must set a Cap'n Proto traversal limit when constructing the reader for WAL payloads. Set capnp::ReaderOptions.traversalLimitInWords proportional to MAX_WAL_PAYLOAD_BYTES / 8 (the word count for the maximum payload). This prevents O(n²) or deeper traversal amplification from a crafted payload that is within the 1 MB byte limit but triggers excessive Cap'n Proto structural nesting. This is standard Cap'n Proto hardening practice and is a one-line change per deserialize call site.

This catch-up mechanism is analogous to etcd's snapshot + WAL catch-up, simplified for XCE's single-writer model. The OP_replJoin, OP_replSnapshot opcodes and catch-up protocol will be added as part of P2-5 (repl.c — standby catch-up implementation; distinct from P2.5 which is the libxce smart discovery phase).

---

Part 7: Mixed Sync/Async Replication Protocol

7.1 Design Rationale

The user's mental model is: "if the API server ACKed my job submit, the job survives a crash." This requires sync replication for job submits only.

All other events (job start, finish, fail, rusage, queue/worker control) are self-healing:

• If a job start is lost, the worker will re-notify on reconnect
• If a finish is lost, the job appears running until the worker re-reports
• Queue/host control is infrequent and can be re-sent by the admin

Sync replication = WAL write + wait for at least one standby ACK before ACKing the client. Async replication = WAL write + fire-and-forget send to standbys.

7.2 Sync Mode: Job Submit Flow

Sync replication sends WAL entries to all standbys in parallel, then waits for at least one ACK:

API Server             Controller (Primary)       Standby-1      Standby-2
    │                           │                      │               │
    │  job submit request       │                      │               │
    │──────────────────────────▶│                      │               │
    │                           │  BEFORE_EVENT hook   │               │
    │                           │  job_submit()        │               │
    │                           │  AFTER_EVENT hook ─┐ │               │
    │                           │                    │ │               │
    │                           │  persist_wal_sync()│ │               │
    │                           │    WAL write + fsync│ │               │
    │                           │    send(entry) ──────────────────────▶│
    │                           │    send(entry) ──────────────────▶│   │
    │                           │    wait_any_ack()   │               │
    │                           │◀─── ACK(ack_lsn) ───────│           │
    │                           │                    │ │               │
    │                           │  ← hook returns     │ │               │
    │                           │                      │               │
    │  reply(jobid)             │                      │               │
    │◀──────────────────────────│                      │               │
    │                           │                      │               │

Parallel fan-out: The primary sends the WAL entry to all connected standbys in parallel (fan-out, not sequential). It then waits for at least one ACK within the timeout. The first ACK received satisfies the durability guarantee — remaining standby ACKs are tracked for lag monitoring but do not block the client reply. This maximizes replication durability without adding latency proportional to the number of standbys.

Partial fan-out failure handling: If repl_broadcast() encounters a send failure on one or more standby endpoints (e.g., SSL_write returns an error, or the endpoint's write FSM reports a broken connection), the broadcast continues sending to remaining standbys — a single failed send does not abort the broadcast. Failed standbys are marked for disconnection: repl_remove_standby() is called after the broadcast completes (not during, to avoid mutating the standbys[] array while iterating it). If no standby ACKs arrive within sync_timeout_ms, persist_wal_sync_append() returns -ETIMEDOUT (degraded mode, Part 7.4). If all standbys fail to send, the result is the same as having zero connected standbys: the entry is locally durable (WAL + fsync) but not replicated.

Key change: xce_reply() is moved after AFTER_EVENT hooks in handle_jobsubmit(). This allows the persistence hook to complete sync replication before the client is ACKed.

7.3 Async Mode: All Other Events

Worker                    Controller (Primary)         Standby
    │                           │                         │
    │  job start notify         │                         │
    │──────────────────────────▶│                         │
    │                           │  mark_job_start()       │
    │                           │  AFTER_EVENT hook ──┐   │
    │                           │                     │   │
    │                           │  persist_wal_async()│   │
    │                           │    WAL write        │   │
    │                           │    send(entry) ─────────▶│
    │                           │    return immediately   │
    │                           │                     │   │
    │                           │  ← hook returns     │   │
    │                           │                         │
    │  reply(OK)                │                         │
    │◀──────────────────────────│                         │

7.4 Degraded Mode

When no standby ACKs within the timeout (sync_timeout_ms, default 50ms):

1. Do NOT block the client indefinitely — timeout is hard
2. ACK the client based on local WAL durability alone — the primary's local WAL is already fsynced
3. Log a warning — monitoring tooling should alert on degraded replication
4. Self-heal later — when the standby reconnects, it will catch up via WAL replay or full snapshot transfer (if WAL was trimmed during the degraded window — see Part 6.6 catch-up protocol)

Degraded mode is transient: In a production cluster, replication degradation only occurs during cluster upgrade, cold boot (standbys not yet connected), or network partitions. A monitoring tool or operator should watch for sustained degraded-mode warnings and investigate immediately. The cluster health system (see Part 5.5 heartbeat monitoring) tracks which standbys are connected and their replication lag. Sustained degraded operation indicates a problem requiring intervention, not a normal operating state.

Out of scope: repl_degraded_timeout_s and restricted mode are removed from scope (see Out of Scope section). Do not implement repl_degraded_timeout_s, E_XCE_REPL_DEGRADED, or restricted-mode state transitions. The text below describes the concept for context only. Monitoring and operator escalation handle sustained degradation.

Degraded mode timeout (concept only — not in scope): repl_degraded_timeout_s would add a configurable window after which the primary transitions to restricted mode (refusing new WAL_JOB_SUBMIT with E_XCE_REPL_DEGRADED). This prevents an unbounded accumulation of locally-durable-only submissions that would be lost if the primary itself fails. The escape hatch (repl_degraded_timeout_s = 0) would disable restricted mode for planned maintenance windows.

Client-visible behavior during split-brain — intentional CAP trade-off: API server clients do not receive any explicit error or warning during a split-brain event — the controller continues to ACK job submits normally. However, job submits ACKed by the lower-term controller during a split-brain window are silently lost when the partition heals and the lower-term controller steps down (its diverged WAL is discarded). There is no application-level notification to the client.

This silence is a deliberate consequence of choosing availability over consistency under partition (AP in CAP terminology). During a network partition, a CP system (like Raft) would refuse to accept writes until quorum is re-established — guaranteeing consistency at the cost of availability. XCE chooses the opposite: the system continues to accept job submits during a partition (availability), accepting that those submits may be lost on healing (consistency sacrifice). Real-time split-brain detection is fundamentally impossible without quorum — any mechanism that detects split-brain in real-time at the client level is equivalent to implementing distributed consensus, which is exactly what Raft provides at ~5000 LOC cost. The split_brain_risk_accepted: true config key is the operator's acknowledgment of this CAP position.

Post-partition, the ctrl_split_brain_detected counter and the ERROR-level degraded replication log provide operator visibility. Operators who require at-most-once or exactly-once job submission semantics must either use quorum-based HA (Raft) or implement client-side deduplication using an idempotency key.

Split-brain is unlikely in practice: In a typical deployment, API servers maintain a persistent connection to a single controller. During a network partition, the API server stays connected to its original controller. Since API servers are not replicated and do not run their own election, they cannot become "split-brained" themselves — they simply become isolated from the primary if the primary fails. The split-brain risk applies to scenarios where multiple API servers connect to different controllers during a partition, which requires a configuration where API servers are load-balanced across controllers. The LSF-style election + redirect mechanism (Part 10.3) ensures that API servers connected to a standby are redirected to the current primary, minimizing the window where an API server could submit to a lower-term controller.

7.5 Sync Timeout Tuning

Setting	Latency Impact	Durability Guarantee
10 ms	Low — client sees ~10ms extra	Strong if standby disk latency < 10ms; degrades to local-WAL-only if standby cannot fsync within the window (the protocol requires fsync-before-ACK, so any ACK received is already fsynced on the standby)
50 ms	Moderate	Good — likely one standby has persisted
100 ms	Higher	Strong — high probability all standbys persisted
No timeout	Unbounded	Strongest — but blocks on standby failure

Recommended default: 50ms. At ~1000 submits/sec, average inter-submit gap is 1ms. A 50ms timeout adds 5% to average submit latency while providing strong durability. If all standbys are healthy (RTT < 5ms), ACKs arrive well within the timeout and there's no waiting.

Standby WAL sync analysis: The standby calls persist_wal_sync_append() for every WAL_JOB_SUBMIT entry received via replication, which performs a local fsync before sending ACK. In the primary's 50ms sync timeout window, the standby must: (1) receive the entry via SSL/vtask, (2) write to local WAL, (3) fsync, (4) send ACK. If the standby's WAL is on a slow disk (HDD, shared storage), the fsync alone could take 10-30ms, consuming most of the 50ms budget. Operators should configure sync_timeout_ms based on the slowest standby's WAL disk latency — on HDD-backed standbys, consider raising the timeout to 100ms. The interaction between standby checkpoint (which holds the WAL lock) and sync replication fsync is additive; if both are on the same disk, the worst-case latency is fsync_time + checkpoint_hold_time.

---

Part 8: Hook-Driven Architecture

8.1 Existing Hook Pattern

XCE's hook system already integrates with the controller. Examples from apiserver.c and sched.c:

/* apiserver.c — existing pattern */
static int apiserver_job_hook_fun(int type, int situation, int detail,
                                  void* data, va_list ap)
{
    cluster_t* c = va_arg(ap, cluster_t*);
    /* ... handle job events ... */
}

/* Registration in apiserver_init() */
xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
             (hook_fn_t)apiserver_job_hook_fun, NULL);

8.2 Persistence Hook Functions

Persistence follows the exact same pattern:

/* persist.c — new hook functions */
int persist_job_hook_fun(int type, int situation, int detail,
                         void* data, va_list ap)
{
    cluster_t* c;
    persist_t* p;

    /* Only act on AFTER_EVENT — state mutation is complete */
    if (situation != HOOK_AFTER_EVENT)
        return 0;

    c = va_arg(ap, cluster_t*);
    p = &c->persist;

    switch (detail) {
    case HOOK_DETAIL_JOB_NEW:
        /* SYNC: WAL write + replicate + wait ACK */
        return persist_wal_sync_append(p, WAL_JOB_SUBMIT,
                                        data, compute_payload_len(data));
    case HOOK_DETAIL_JOB_START:
        return persist_wal_async(p, WAL_JOB_START, data, ...);
    case HOOK_DETAIL_JOB_EXIT:
        return persist_wal_async(p, WAL_JOB_FINISH, data, ...);
    case HOOK_DETAIL_JOB_FAIL:
        return persist_wal_async(p, WAL_JOB_FAIL, data, ...);
    /* ... etc ... */
    default:
        return 0;
    }
}

int persist_queue_hook_fun(int type, int situation, int detail,
                           void* data, va_list ap)
{
    /* ... same pattern for queue events ... */
}

int persist_worker_hook_fun(int type, int situation, int detail,
                            void* data, va_list ap)
{
    /* ... same pattern for worker events ... */
}

8.3 Critical Change: Reply After Hooks

Current code in handle_jobsubmit() (job.c:455-525):

/* CURRENT — reply BEFORE hooks */
xce_run_hooks(BEFORE_EVENT, HOOK_DETAIL_JOB_NEW, c, submit);
jobid = job_submit(c, submit);

reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = jobid;
xce_reply(ep, c->myhost, &reply);       /* ← ACK before hooks */

/* Collect jobs into list... */
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist);  /* ← persistence happens here */

Modified code — move xce_reply() after AFTER_EVENT:

/* MODIFIED — reply AFTER hooks */
xce_run_hooks(BEFORE_EVENT, HOOK_DETAIL_JOB_NEW, c, submit);
jobid = job_submit(c, submit);

if (jobid == 0) {
    reply.status = -errno;
    reply.data_which = RequestReply_data_nodata;
    xce_reply(ep, c->myhost, &reply);   /* Error reply is immediate */
    return -errno;
}

/* Collect jobs into list... */
/* AFTER_EVENT hook errors are tolerated per XCE hook design: pre-hook (BEFORE_EVENT)
 * failures abort the operation; post-hook (AFTER_EVENT) failures are logged but do not
 * require transaction rollback.
 *
 * WAL result is communicated via a PER-REQUEST stack variable passed via xce_run_hooks
 * extra argument. Using a shared persist_t field would race: multiple vcpu threads can
 * execute handle_jobsubmit() concurrently (one per API server connection), and each
 * would overwrite the shared field with its own result, causing thread A to read
 * thread B's -EIO and reject a durable job, or vice versa.
 *
 * Implementation: persist_job_hook_fun() receives &sync_rc via va_list as the last
 * argument (after cluster_t* c and the jlist pointer), writes the WAL result there,
 * and returns 0. handle_jobsubmit() reads sync_rc after hooks return. */
int sync_rc = 0;    /* per-request WAL result — not shared, no race */
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist, &sync_rc);
/* ↑ SYNC replication happens inside the persistence hook.
 * The hook always returns 0 (AFTER_EVENT errors tolerated at the hook level).
 * WAL write result is in sync_rc (stack-allocated, this vcpu thread only):
 *   0           — WAL written + fsynced (+ standby ACKed if standbys connected)
 *  -EIO         — local WAL write failed (disk full, permanent)
 *  -ETIMEDOUT   — wrote locally but no standby ACKed within sync_timeout_ms */
if (sync_rc == -EIO) {
    /* Permanent WAL failure — do not ACK to client.
     * The job was inserted into kvmgr by job_submit() but is not durable.
     * CRITICAL: mark the job terminal so checkpoint GC cleans it up. Without
     * this, the job survives every GC pass (job_is_terminal() returns false)
     * and accumulates as a phantom entry the client was told doesn't exist.
     *   job_mark_wal_error_exit(c, jobid);   <- sets JOB_STAT_EXIT + finish_time=now
     * This makes it eligible for GC after finished_retain_s and prevents
     * the scheduler from dispatching work to a nonexistent job. */
    job_mark_wal_error_exit(c, jobid);   /* see job.h — sets stat=JOB_STAT_EXIT */
    reply.status = -EIO;
    reply.data_which = RequestReply_data_nodata;
    xce_reply(ep, c->myhost, &reply);
    ERROR("job submit: WAL write failed with -EIO — job %lu marked terminal for GC", jobid);
    return -EIO;
}
if (sync_rc == -ETIMEDOUT) {
    /* Locally durable but no standby ACK — degraded durability. Accept. */
    WARN("job submit: no standby ACK within timeout — accepting with degraded durability");
}

/* Reply AFTER hooks — client ACK implies durability */
reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = jobid;
xce_reply(ep, c->myhost, &reply);

Why this works: AFTER_EVENT hook failures are tolerated (no transaction rollback needed). The persistence hook uses the per-request stack variable sync_rc (passed via va_list as an extra argument to xce_run_hooks()) to communicate WAL write results back to handle_jobsubmit() without relying on hook return values. Because sync_rc is stack-allocated in handle_jobsubmit(), it is thread-private — concurrent vcpu threads each have their own sync_rc with no sharing or race conditions. This decouples the sync replication result from the hook system's return-value semantics.

No reply reordering concern: The controller processes requests from each API server connection one at a time — a request is fully handled (including hooks and xce_reply()) before the next request on the same connection is processed. SYNC replication adds latency to the AFTER_EVENT hook, which delays the xce_reply() for that specific request, but it does not cause out-of-order replies because the next request on that connection doesn't begin processing until the current one completes. Replies to different API servers may have different latencies, but each API server sees its own replies in request order.

Lock safety analysis: The current handle_jobsubmit() acquires lock(c->jobmgr, RTTI_LOCK_READ) for job lookup, then operates on the per-job object. No kvmgr write lock is held during the hook call or xce_reply() for job submit. The change to move xce_reply() after hooks is therefore safe (note: handle_worker_join() in join.c:58-69 is the exception — it holds the workermgr WRITE lock during AFTER_EVENT hooks, but this is acceptable because worker join uses async replication only):

1. job_submit() takes the jobmgr READ lock for lookup, then per-object lock for modification — no change here
2. AFTER_EVENT hooks (including persist_wal_sync_append()) acquire their own locks internally (persist.lock for WAL, repl.lock for replication) — no nesting with jobmgr
3. xce_reply() writes to the endpoint's outq — no kvmgr lock interaction
4. handle_worker_join() in join.c:58-69 takes the workermgr WRITE lock for add_worker(). Worker join is always async-replicated (never blocks on repl_wait_ack), so holding the write lock during the hook call is safe — the hook only does a non-blocking WAL write and fire-and-forget replication send. The xce_reply() for worker join is also inside the write lock (at line 65), which risks blocking on endpoint outq backpressure under extreme conditions. add_worker() is infrequent enough that this is acceptable in practice, but if endpoint backpressure becomes a concern, xce_reply() should be moved after unlock(c->workermgr).

Enforcement: The persistence hook for HOOK_DETAIL_WORKER_JOIN must use persist_wal_async() — never persist_wal_sync_append(). To make this self-documenting and prevent a future refactor from accidentally switching to sync replication while the WRITE lock is held, add a persist_wal_async_only() wrapper that asserts type != WAL_JOB_SUBMIT at runtime (under persist.lock), or log a CRITICAL error and abort if a sync-type entry is attempted through the worker path. This is a belt-and-suspenders guard: the hook mapping in Part 2.3 already maps worker events to async-only WAL types, but the runtime assertion prevents regression.

Replication scope: Sync replication (repl_wait_ack) only triggers for WAL_JOB_SUBMIT entries, which originate from API server connections. The controller only handles connections from workers, scheduler, and API servers — never end users. At 1–3 API server connections, the vcpu pool is not at risk of exhaustion from blocking repl_wait_ack() calls.

Array job sync replication latency bound (CRITICAL): Array jobs submit multiple sub-jobs in a single handle_jobsubmit() call. With AFTER_EVENT firing once per sub-job (the existing hook dispatch model), 10,000 sub-jobs each triggering repl_wait_ack() would block the vcpu for 500 seconds. Mitigation — accumulation buffer: For array submits (submit->array_size > 1), handle_jobsubmit() allocates a persist_batch_ctx_t on the heap and passes it as a second extra va_list argument (after &sync_rc), always NULL for scalar submits:

/* Scalar submit: sync_rc only (batch_ctx = NULL) */
int sync_rc = 0;
persist_batch_ctx_t* batch_ctx = NULL;
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist, &sync_rc, batch_ctx);

/* Array submit: sync_rc + batch_ctx (non-NULL) */
int sync_rc = 0;
persist_batch_ctx_t* batch_ctx = persist_batch_ctx_alloc(submit->array_size, 100 /* batch_size */);
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist, &sync_rc, batch_ctx);
/* ... iterate sub-jobs, each fires AFTER_EVENT ... */
free(batch_ctx);

The hook reads both arguments: int* sync_rc_p = va_arg(ap, int*); persist_batch_ctx_t* batch = va_arg(ap, persist_batch_ctx_t*);. When batch != NULL, the hook accumulates WAL entries and flushes every 100 sub-jobs. The hook sets batch->sub_jobs_done++ and writes the final sync_rc into *sync_rc_p on the last sub-job (batch->sub_jobs_done == batch->total_sub_jobs). This bounds latency to (array_size / 100) × 50ms — 5 seconds for a 10K array. The accumulation buffer is per-request, not shared, so concurrent submits are fully independent.

Async hook error handling: For async-replicated events (all events except WAL_JOB_SUBMIT), persist_wal_async() may return a non-zero error code. These errors fall into two categories: (1) -EIO (local WAL write failure — disk full, I/O error) and (2) other errors. For all async events, the caller (e.g., handle_worker_join(), on_job_start(), on_job_finish()) must follow this error handling convention:

• -EIO (local WAL write failure): Log an ERROR with the event type, LSN (if available), and error code. The in-memory state mutation has already completed — the entity exists in the kvmgr but is not durable on local WAL. The error is not recoverable without operator intervention (disk full or permanent I/O failure). Do NOT undo the in-memory mutation — it is the source of truth for the scheduler and API queries. Do NOT reply to the client with an error for async events (they have no reply expected from the persistence path — xce_reply() for async events has already happened before the hook).
• Async replication failure (send to standby failed): Log a WARN. The entry is locally durable (WAL + fsync). The standby will receive it on reconnection catch-up. No action needed — this is the expected behavior for async replication.
• Summary: Async hook callers should log errors but never undo in-memory state or block the vcpu thread. The self-healing property of async events (workers re-notify on reconnect) means that a missed async WAL entry is recoverable without manual intervention unless the local WAL itself has failed.

8.4 Registration in cluster_init()

Hook registration is conditional on role and happens after persist_recover() completes. This resolves two constraints:

• Part 2.5: "hooks not registered until after persist_recover() completes" — recovery replay must not trigger WAL writes
• Part 6.4: apply_wal_entry() on standbys must not fire hooks — standbys must never register persistence hooks

/* In cluster_init(), after xce_install_hooks(): */

if (persist_config_enabled(&c->myconf->cluster_config->persist)) {
    rc = persist_init(&c->persist, &c->myconf->cluster_config->persist, c);
    if (rc < 0) return rc;

    /* Step 1: Recover BEFORE registering hooks.
       persist_recover() replays WAL via the same mutation functions
       (job_submit, mark_worker_unavail, etc.) but hooks are not yet
       registered, so no WAL writes occur during replay. */
    rc = persist_recover(&c->persist, c);
    if (rc < 0) return rc;

    /* Step 2: Register persistence hooks ONLY on primary path.
       Standbys apply WAL entries via apply_wal_entry() — hooks are
       not registered on standbys, so no infinite loop occurs.
       Hook registration happens here for primary, or in
       ha_promote_to_primary() step 5 when a standby promotes. */
    if ((ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_PRIMARY) {
        xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
                     (hook_fn_t)persist_job_hook_fun, NULL);
        xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_QUEUE,
                     (hook_fn_t)persist_queue_hook_fun, NULL);
        xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_WORKER,
                     (hook_fn_t)persist_worker_hook_fun, NULL);
    }
    /* Standbys: no hooks registered here. apply_wal_entry() mutates state
       directly — persistence hooks are registered only after cluster recovery
       completes, and only on the primary path. */
}

---

Part 9: Worker Connection Strategy

9.1 Current Design: Worker-to-Controller Long TCP

Workers maintain a persistent TCP connection to the controller. This provides:

1. Fast failure detection: If a worker dies, the controller gets an immediate TCP RST — no polling needed
2. Easy push from controller: The controller can push job dispatches, config updates, and heartbeats over the existing connection without initiating a new one
3. Bidirectional heartbeat: Both sides can detect liveness via the same connection

The main concern is the OS connection limit: each TCP connection consumes a file descriptor and kernel socket buffer. However, for typical HPC cluster scales (100s–low 1000s of workers), this is a configuration issue, not an architectural one:

# /etc/sysctl.conf — sufficient for a 2000-node cluster
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
fs.file-max = 200000

# In the controller process startup:
# ulimit -n 65535

Linux handles tens of thousands of idle TCP connections with minimal overhead. The practical limit is ~65K connections per IP/port tuple, which is far beyond any single XCE cluster.

9.2 Decision: Workers Connect Directly to Primary

Three options were evaluated:

Option A — Direct-only: Workers connect to primary only.

Option B — Standby as full proxy: Workers connect to any candidate; standbys forward all traffic (both directions) to primary.

Option C — Standby as discovery + redirect: Workers can connect to any candidate at startup; standbys send a redirect response pointing to the primary; worker reconnects to primary.

Aspect	A (Direct-only)	B (Full proxy)	C (Redirect)
Normal latency	1 hop	2 hops (doubles)	1 hop
Complexity	Minimal	High — bidirectional relay, reply correlation, read-after-write consistency	Low — just a redirect opcode
Failover reconnection	Workers must discover new primary	Workers on promoting standby: zero cost; others: must reconnect	Workers reconnect with libxce auto-retry
Worker startup	Must know primary	Pick any candidate	Pick any candidate, get redirected
Read-after-write	No issue	Critical issue — standby forwards write to primary, but its own state is stale until replicated back	No issue — all writes go to primary directly
Connection load	Primary only	Distributed	Primary only (discovery is transient)

Decision: Option C (Redirect). It gives the "worker doesn't need to know the primary" benefit at startup, without paying the 2-hop latency and read-after-write consistency cost on every message. The redirect is a one-time cost during connection setup — amortized to zero over a long-lived connection.

This pattern is used by etcd (proxy mode), Redis Sentinel, and MongoDB replica sets.

9.3 Full Communication Topology

                     ┌─────────────────┐
                     │  Controller     │
                     │  Candidates     │
                     │  (--peers flag) │
                     └────────┬────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
        ┌──────────┐   ┌──────────┐   ┌──────────┐
        │ ctrl-1   │   │ ctrl-2   │   │ ctrl-3   │
        │ PRIMARY  │   │ STANDBY  │   │ STANDBY  │
        │ P=0      │   │ P=1      │   │ P=2      │
        └────┬─────┘   └────┬─────┘   └────┬─────┘
             │              │              │
    ┌────────┼──────┐       │              │
    │        │      │       │              │
    ▼        ▼      ▼       ▼              ▼
  Worker   Worker  Worker  (accepts discovery
   W1       W2      W3     connections only —
                            sends REDIRECT)
                                │
                         New Worker W4
                         startup flow:
                         1. Try ctrl-2 (random)
                         2. Get REDIRECT → ctrl-1
                         3. Connect to ctrl-1

9.4 Standby Redirect Handling

When a standby receives a join request from a non-controller peer (worker, scheduler, API server), it sends a redirect response instead of accepting the connection:

/* In handle_join() — modified for HA redirect */
int handle_join(cluster_t* c, endpoint_t* ep, xce_request_t* request)
{
    xce_peerinfo_t* peer = request->body.joinmsg->peer;
    int role = peer->role;

    /* If I'm a standby and the peer is a worker/scheduler/apiserver,
       redirect them to the primary */
    if ((ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_STANDBY) {
        switch (role) {
        case HostRole_worker:
        case HostRole_scheduler:
        case HostRole_apiserver: {
            xce_reply_t reply;
            init_reply_by_req(&reply, request);
            /* Guard: primary_host is zero-initialized on startup and only populated
             * after the standby receives the first heartbeat from the primary.
             * If we have never seen a primary (cold start, all controllers starting
             * simultaneously), return E_XCE_NOMASTER instead of a garbage redirect. */
            if (c->ha.primary_host[0] == '\0' || c->ha.primary_port == 0) {
                reply.status = E_XCE_NOMASTER;
                xce_reply(ep, c->myhost, &reply);
                return 0;
            }
            reply.status     = E_XCE_REDIRECT;
            reply.data_which = RequestReply_data_redirect;
            reply.data.redirect.host = c->ha.primary_host;
            reply.data.redirect.port = c->ha.primary_port;
            xce_reply(ep, c->myhost, &reply);
            return 0;
        }
        default:
            break;
        }
    }

    /* Normal join processing (I am primary) */
    /* ... existing code ... */
}

Redirect authentication: All controller connections use SSL enforced by iomgr (netio.c), which means a redirect response can only come from a peer that has completed the SSL handshake with a valid certificate. A rogue endpoint that has not established an SSL connection cannot inject redirect responses. Additionally, the RedirectInfo contains only host/port of a known candidate — the receiving peer validates that the redirect target is in its candidate list (from --peers CLI flag) before following it.

9.5 Thundering Herd Mitigation on Failover

When the primary dies, all workers connected to it get TCP RST simultaneously. If they all reconnect to the new primary at once, the listen backlog and accept loop could be overwhelmed.

Mitigation: Exponential backoff with random jitter in libxce (detailed in Part 10). This spreads worker reconnections over a time window instead of a single instant.

For 2000 workers with 100ms base backoff + 50% jitter:

• First wave: ~0-150ms — ~500 workers reconnect
• Second wave: ~150-300ms — ~300 workers
• Third wave: ~300-600ms — remaining workers
• Total reconnection window: ~600ms — well within the new primary's capacity

9.6 Worker Disconnect — Deferring to Terminal State

When a worker disconnects, instead of immediately removing it from data structures, the controller transitions it to WORKER_UNAVAILABLE:

/* In handle_worker_disconnect() — modified for deferred cleanup */
void handle_worker_disconnect(cluster_t* c, worker_t* w) {
    /* Instead of: destroy(w); kvmgr_del(...); */
    w->state = WORKER_UNAVAILABLE;
    w->host.dead_since = time(NULL);

    /* Worker remains in zarray (inuse=1) and kvtab.
       Scheduler already ignores UNAVAILABLE workers.
       Actual cleanup happens at next checkpoint if
       worker_retain_s has expired. */
}

9.7 Redirect Cap'n Proto Schema

/* New struct for redirect response */
struct RedirectInfo $C.mapname("xce_redirect_t") {
    host @0 :Text $C.mapname("host");
    port @1 :UInt32 $C.mapname("port");
}

/* In RequestReply union: */
redirect @8 :RedirectInfo $C.mapname("redirect");

/* New error code in common/ */
#define E_XCE_REDIRECT (-100)  /* Not an error — redirect instruction */

---

Part 10: libxce Smart Discovery

10.1 Problem

Currently, workers and other components are configured with a single controller address. With HA, they need to:

1. Discover which candidate is the current primary
2. Follow redirects from standbys
3. Reconnect automatically on failover with appropriate backoff
4. Verify they're talking to the primary after connecting

10.2 Configuration API

/* libxce — xce_cluster.h */

typedef struct {
    char     host[XCE_MAXNAME_LEN];
    uint32_t port;
} xce_candidate_t;

typedef struct {
    char     ca_cert_path[PATH_MAX];  /* CA cert path for SSL_VERIFY_PEER (empty = skip) */
} xce_tls_config_t;

typedef struct {
    xce_candidate_t  candidates[MAX_HA_CANDIDATES];
    int              ncandidates;           /* Number of candidates configured */
    uint32_t         connect_timeout_ms;    /* Per-candidate connect timeout (default 2000) */
    uint32_t         retry_base_ms;         /* Backoff base (default 100) */
    uint32_t         retry_max_ms;          /* Backoff cap (default 5000) */
    int              max_retries;           /* Max reconnection attempts (default 10) */
    xce_tls_config_t tls;                  /* TLS options — CA cert path for SSL_VERIFY_PEER.
                                            * Populated from --ca-cert CLI flag (P2.5-5/P2.5-6). */
} xce_cluster_config_t;

/* Connect to the cluster — returns SSL* (not raw fd) connected to primary.
 * The caller must NOT call close() on the returned pointer — use
 * xce_disconnect_cluster() to tear down the SSL connection and free resources.
 * All controller connections use SSL enforced by iomgr; this function performs
 * the TLS handshake (see P2.5-2). */
SSL* xce_connect_cluster(xce_cluster_config_t* conf);

/* Connect + join as specific role (worker, scheduler, apiserver) */
SSL* xce_join_cluster(xce_cluster_config_t* conf, xce_peerinfo_t* myinfo);

/* Handle disconnection — auto-reconnect with backoff + redirect following.
 * On success, *ssl is updated to the new SSL connection. On failure, *ssl is
 * set to NULL and the caller must treat the connection as lost. */
int xce_reconnect(SSL** ssl, xce_cluster_config_t* conf);

/* Disconnect from cluster — SSL shutdown + close + free.
 * Safe to call on NULL or already-disconnected SSL*. */
void xce_disconnect_cluster(SSL* ssl);

/* REMOVED: xce_load_cluster_config() — no YAML parsing in libxce.
 * xce_cluster_config_t is populated by caller (worker main, scheduler main, etc.)
 * from --peers and --ca-cert CLI flags. */

10.3 Connection State Machine

                    ┌─────────────┐
                    │    INIT     │
                    └──────┬──────┘
                           │ Load candidates[] from config
                           ▼
                 ┌─────────────────┐
           ┌────▶│   CONNECTING    │◀──────────────┐
           │     └────────┬────────┘               │
           │   timeout or │                        │
           │   connection │                        │
           │     failed   │ connected              │
           │        │     ▼                        │
           │        │  ┌─────────────┐             │
           │        │  │  CONNECTED  │             │
           │        │  └──────┬──────┘             │
           │        │         │ send join request   │
           │        │         ▼                     │
           │        │  ┌─────────────┐              │
           │        │  │  HANDSHAKE  │              │
           │        │  └──────┬──────┘              │
           │        │         │ receive reply        │
           │        │         ▼                     │
           │        │  ┌──────────────┐             │
           │        │  │ CHECK REPLY  │             │
           │        │  └───┬──────┬───┘             │
           │        │      │      │                 │
           │        │  ACCEPTED  REDIRECT            │
           │        │      │      │                 │
           │        │      ▼      ▼                 │
           │        │  ┌───────┐ ┌────────────┐     │
           │        │  │ READY │ │ Follow      │     │
           │        │  │(prime)│ │ redirect →  │─────┘
           │        │  └───┬───┘ │ reconnect  │
           │        │      │     │ to primary  │
           │        │      │     └────────────┘
           │        │      │
           │        │      │ TCP RST / connection lost
           │        │      ▼
           │        │  ┌────────────┐
           └────────┴──│ RECONNECT  │
              backoff   │ (backoff + │
              + jitter  │  jitter)   │
                        └────────────┘

10.4 Candidate Selection Strategy

Two phases of selection:

Initial connection — Try candidates in priority order:

• The highest-priority alive candidate is most likely the primary
• This minimizes redirects on first connection

Reconnection after failure — Try candidates in round-robin with priority bias:

• Start from the candidate after the one we were connected to (it may be down)
• Round-robin distributes reconnection load across candidates after a failover
• Avoids all workers hammering the same candidate simultaneously

/* Priority-ordered for initial, round-robin for reconnection */
static int select_candidate(xce_cluster_config_t* conf, int* attempt,
                            int last_candidate_idx)
{
    if (*attempt < conf->ncandidates) {
        /* First pass: priority order (index 0 first) */
        return (*attempt)++;
    }

    /* Subsequent passes: round-robin starting after last connected candidate */
    int start = (last_candidate_idx + 1) % conf->ncandidates;
    return (start + (*attempt)++) % conf->ncandidates;
}

10.5 Backoff with Jitter

/* Thread-local RNG seed — each thread initializes its own seed once.
 * Using __thread avoids the data race on a shared global rng_seed when
 * multiple threads call backoff_jitter() concurrently (rand_r with the
 * same shared seed is concurrent read-modify-write, which is a race). */
static __thread unsigned int rng_seed;
static __thread int rng_init = 0;

static void rng_init_once(void) {
    if (rng_init) return;
    /* Seed from getrandom(2) for unpredictable jitter — pthread_self() XOR
     * time(NULL) is predictable (deterministic thread IDs, 1-second resolution)
     * and could allow an attacker to coordinate thundering-herd attacks.
     * getrandom(2) is available on Linux 3.17+; fall back to /dev/urandom. */
    unsigned int seed = 0;
#if defined(__linux__) && defined(SYS_getrandom)
    if (syscall(SYS_getrandom, &seed, sizeof(seed), GRND_NONBLOCK) != sizeof(seed)) {
        /* getrandom unavailable (e.g., early boot entropy depletion) — use urandom */
        int fd = open("/dev/urandom", O_RDONLY | O_CLOEXEC);
        if (fd >= 0) { read(fd, &seed, sizeof(seed)); close(fd); }
        else { seed = (unsigned int)((unsigned long)pthread_self() ^ (unsigned long)time(NULL)); }
    }
#else
    seed = (unsigned int)((unsigned long)pthread_self() ^ (unsigned long)time(NULL));
#endif
    rng_seed = seed;
    rng_init = 1;
}

/* Exponential backoff with random jitter to spread reconnect storm.
 * Uses per-thread rand_r() to avoid data races. */
static void backoff_jitter(int attempt, xce_cluster_config_t* conf)
{
    int64_t delay = (int64_t)conf->retry_base_ms * ((int64_t)1 << (attempt > 30 ? 30 : attempt));

    if (delay > conf->retry_max_ms)
        delay = conf->retry_max_ms;

    rng_init_once();
    int jitter = rand_r(&rng_seed) % ((int)delay / 2 + 1);
    delay += jitter;

    usleep((useconds_t)(delay * 1000));
}

Why jitter matters: Without jitter, all 2000 workers back off for exactly the same duration and reconnect simultaneously — a thundering herd. With 50% jitter, the reconnection window spreads across 1.5× the base backoff, allowing the new primary to accept connections gradually.

10.6 Full Connection Flow

SSL* xce_connect_cluster(xce_cluster_config_t* conf)
{
    int attempt = 0;
    int last_candidate = -1;
    int redirect_count = 0;

    for (; attempt < conf->max_retries; attempt++) {
        /* Note: there is no 'goto next_candidate' in this flow — the loop uses
         * 'continue' and 'attempt' is intentionally incremented on each redirect
         * (redirect_count provides a separate hop cap at 5). */
        /* redirect_count is NOT reset per candidate — it is a global hop counter
         * across the entire connection attempt to prevent redirect loops that cycle
         * through multiple candidates (e.g., A→B→A). */
        int idx = select_candidate(conf, &attempt, last_candidate);
        xce_candidate_t* cand = &conf->candidates[idx];

        /* Establish SSL session inline. tcp_connect_ssl() performs
         * TCP connect + TLS handshake as a single operation. On failure, returns NULL. */
        SSL* ssl = tcp_connect_ssl(cand->host, cand->port,
                                   conf->connect_timeout_ms, &conf->tls);
        if (ssl == NULL) {
            backoff_jitter(attempt, conf);
            continue;
        }

        /* Send join request to discover role */
        xce_request_t req;
        xce_reply_t   reply;
        init_join_request(&req, &conf->myinfo);
        xce_send_request(ssl, &req);
        xce_recv_reply(ssl, &reply);

        if (reply.status == 0) {
            /* Accepted by primary */
            last_candidate = idx;
            return ssl;
        }

        if (reply.status == E_XCE_REDIRECT) {
            /* Standby redirected us — validate redirect target before following.
             * Only follow to a known candidate from --peers list to prevent
             * a rogue endpoint from redirecting us to an arbitrary host. */
            int target_known = 0;
            for (int k = 0; k < conf->ncandidates; k++) {
                if (strcmp(conf->candidates[k].host, reply.data.redirect.host) == 0
                    && conf->candidates[k].port == reply.data.redirect.port) {
                    target_known = 1;
                    break;
                }
            }
            if (!target_known) {
                /* Redirect to unknown host — treat as error, try next candidate */
                WARN("redirect to unknown host %s:%u — ignoring",
                     reply.data.redirect.host, reply.data.redirect.port);
                xce_disconnect_cluster(ssl);
                backoff_jitter(attempt, conf);
                continue;
            }

            /* Follow redirect — no backoff since this is expected during discovery.
             * The redirect target must be in the candidate list (validated above).
             * Increment redirect_count BEFORE tcp_connect_ssl so the hop count is
             * enforced even if every redirect target fails to connect — without this,
             * a cycle of unreachable redirect targets would never increment the counter
             * and could loop until max_retries (not the redirect_count cap of 5). */
            if (++redirect_count >= 5) {
                xce_disconnect_cluster(ssl);
                errno = ELOOP;
                return NULL;  /* EXCE_ERR_REDIRECT_LOOP */
            }
            xce_disconnect_cluster(ssl);
            ssl = tcp_connect_ssl(reply.data.redirect.host,
                                  reply.data.redirect.port,
                                  conf->connect_timeout_ms, &conf->tls);
            if (ssl != NULL) {
                /* Re-send join request to verify this is actually the primary.
                 * TOCTOU protection: the primary may have changed between the
                 * redirect and our connection — the join reply confirms we
                 * reached the actual primary, not another standby. */
                init_join_request(&req, &conf->myinfo);
                xce_send_request(ssl, &req);
                xce_recv_reply(ssl, &reply);

                if (reply.status == 0) {
                    return ssl;  /* Confirmed primary */
                }
                /* redirect_count was already incremented before tcp_connect_ssl;
                 * if we reach here with another redirect reply, the outer loop
                 * will re-check the count on the next iteration. */
            }
            xce_disconnect_cluster(ssl);
            continue;
        }

        /* Unexpected reply — try next candidate */
        xce_disconnect_cluster(ssl);
        backoff_jitter(attempt, conf);
    }

    /* All retries exhausted */
    return NULL;
}

10.7 Reconnection on Failover

When the primary fails, workers detect it via TCP RST or heartbeat timeout. The reconnection flow:

int xce_reconnect(SSL** ssl, xce_cluster_config_t* conf)
{
    xce_disconnect_cluster(*ssl);
    *ssl = NULL;

    /* xce_connect_cluster handles candidate selection,
       redirect following, and backoff internally */
    *ssl = xce_connect_cluster(conf);
    return (*ssl != NULL) ? 0 : -1;
}

The worker's main loop integrates reconnection naturally:

/* Worker main loop — simplified */
for (;;) {
    int rc = xce_process_message(ssl);

    if (rc < 0) {
        /* Connection lost */
        TRACE("connection to controller lost, reconnecting...");

        rc = xce_reconnect(&ssl, &conf);
        if (rc < 0) {
            ERROR("failed to reconnect to cluster");
            /* Could exit, or keep retrying depending on policy */
            continue;
        }

        /* Reconnected — re-register with controller */
        on_worker_join(cluster_cls, ssl_to_endpoint(ssl));
        TRACE("reconnected to controller (primary)");
    }
}

10.8 Candidate List Distribution

Controller: reads candidate configuration from cluster.bin (binary format produced by the config tool from cluster.yaml). The controller also accepts --peers CLI flag for the candidate list:

xce_controller \
  --workdir /var/lib/xce \
  -n ctrl-1 \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000

Workers, schedulers, and API servers do NOT read any config file. They accept CLI flags only:

xce_worker \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000 \
  --log-level info

xce_scheduler \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000 \
  --log-level info

libxce's xce_cluster_config_t is populated directly from these CLI flags at startup — no YAML parsing in any component except the config tool.

Distribution: The --peers string is generated by the config tool and distributed via deployment scripts (Ansible, Salt) into systemd unit files or container environment variables. Config management tooling should ensure all nodes receive identical candidate lists atomically before any controller restarts.

Candidate list consistency: A stale --peers value on one node could cause connectivity failures — for example, if a worker's peers list includes only ctrl-1 while ctrl-1 is down, the worker cannot discover ctrl-2 or ctrl-3. Config management tooling should ensure all nodes receive identical --peers updates atomically.

Out of scope: The config version hash in heartbeat messages is deferred to P3 (see Out of Scope section). Do not implement the configVersion field, the hash embedding in OP_heartbeat, or the reload-on-mismatch logic during P2.5. The description below is retained for future reference only.

Config freshness control (deferred to P3 — not in scope for P2.5): The controller could embed a config version hash (CRC32 of the serialized candidate list, +1 to avoid 0 sentinel) in OP_heartbeat. When a worker receives a mismatched hash, it logs a warning (config changes require a restart to take effect since candidates come from CLI flags). The configVersion @23 :UInt32 Cap'n Proto field in Request is reserved for this feature. Do not add this field until P3.

10.9 libxce Changes Summary

File	Change
src/lib/include/xce/cluster.h	Add xce_cluster_config_t, xce_candidate_t types
src/lib/include/xce/cluster.h	Add xce_connect_cluster() (returns SSL*), xce_disconnect_cluster(), xce_join_cluster(), xce_reconnect() (takes SSL**) API. No xce_load_cluster_config() — callers populate xce_cluster_config_t from --peers/--ca-cert CLI flags.
src/lib/cluster.c (new)	Implement candidate selection, redirect following, backoff+jitter, reconnection. Accept pre-populated xce_cluster_config_t — no config file reading.
src/worker/main.c	Replace hardcoded controller address with xce_cluster_config_t; populate from --peers and --ca-cert CLI flags; add reconnection loop; defer worker removal (set WORKER_UNAVAILABLE instead of eager delete)
src/scheduler/	Same pattern as worker — parse --peers and --ca-cert CLI flags, populate xce_cluster_config_t, use libxce discovery.
src/frontend/apiserver/backend/clib.go	Use xce_cluster_config_t for controller discovery in RegisterToBackend()

---

Implementation Phases

P0: Foundation (Single-Controller Persistence)

Goal: Controller survives restart with job/queue/worker state intact.

P0 is broken into seven sub-phases with clear deliverables and dependency ordering. Each sub-phase produces a buildable, testable increment.

---

P0-A: Cleanup & Prerequisites

Deliverable: Clean codebase + rtti COW interface ready. No persistence behavior yet.

Step	File	Description
P0-0	src/controller/job.c + worker.c + cluster.c	Remove dead code (Part 1): unused NJOBS/jobid constants, load_jobs() #if 0 body. Do this first so P0 builds on a clean foundation
P0-0a	src/lib/zarray.c + include/zarray.h	Merge zarray_t into xzarray_t (partially completed per git history) — enables uniform COW for all data structures including endpoints. Scope: move the zarray_t definition into xzarray_t, update all zarray_* call sites to use xzarray_* equivalents, verify that existing tests still pass. This step is a prerequisite for P0-2 (header move of xzarray_t) and P0-8a (deferred endpoint cleanup). Owner: this must be completed or explicitly handed off before P0-2 begins; if the merge is not yet complete, P0-8a falls back to the immediate-cleanup path (see Part 4.9 note). IMPORTANT: P0-0a and P0-0b must be developed on the same branch or merged together before P0-2 — both steps modify zarray.c compound-literal initializers and the xzarray_type registration. P0-0a renames zarray_* functions to xzarray_*; P0-0b adds two new slots to intf_varray_t. If developed in parallel on separate branches, the initializer updates will conflict. Merge P0-0a first, then P0-0b (which builds on the renamed functions), or develop both on a single P0-A feature branch.
P0-0b	rtti/typeinfo.h + zarray.c + kvmgr.c + kv.c	rtti library enhancement (see Appendix A.9): Check the installed rtti header first — vfreeze_fn_t, vthaw_fn_t, and retain_fn_t may already be present in the installed /nix/store/...-rtti-0.1.0/include/rtti/typeinfo.h from a prior pass. Verify with grep vfreeze_fn_t $(nix eval --raw .#rtti)/include/rtti/typeinfo.h. If the types are already present, skip the header additions and go directly to the zarray.c implementation work. If not present: add retain_fn_t (bool (*)(void*, void*)), vfreeze_fn_t, and vthaw_fn_t to typeinfo.h; grow intf_varray_t from 6 to 8 slots; add vfreeze()/vthaw() global API wrappers. Remaining work in all cases (whether or not header types were already present): (1) add shadow and shadow_caps fields to xzarray_private_t in zarray.c; (2) update all compound-literal intf_varray_t initializers in zarray.c, kvmgr.c, and kv.c to include the two new slots — first convert all positional initializers to designated-initializer style ({ .vstore = ..., .vfreeze = (vfreeze_fn_t)NULL, .vthaw = (vthaw_fn_t)NULL }) before adding the new slots, since adding members to a struct with positional initializers is a compile error. Run grep -rn 'intf_varray_t' src/ to find all initializer sites before starting. Update nix derivation version from rtti-0.0.1 to rtti-0.1.0. This is a prerequisite for P0-2 which wires vfreeze/vthaw to the xzarray implementations. Requires P0-0a to be merged first (same-branch constraint above). rtti source: the rtti library source is maintained by the XCE team (not a third-party dependency). The implementer has direct access to modify typeinfo.h and bump the Nix derivation. Locate the source via nix flake show or the project's flake.nix inputs. BREAKING ABI CHANGE: both intf_varray_t (grows from 6 to 8 pointers) and xzarray_private_t (gains shadow and shadow_caps fields, changing privateSize) change size. Since no version of XCE has been released, backward binary compatibility is not required — all components are rebuilt together. Trigger a full ninja -C build clean rebuild after this step.
P0-5	src/include/xce/cluster_def.h + include/config.h	Add persist_t persist; to cluster_t. Pre-addition audit: run grep -rn 'cluster_t|sizeof.*cluster|offsetof.*cluster' src/ and verify no code assumes sizeof(cluster_t) or uses field offsets before adding the new field — adding a large struct may shift subsequent field offsets. Note c->endpoints is currently zarray_t endpoints (not xzarray_t); the deferred endpoint cleanup (P0-8b) requires a type change in P0-0a. Note: ha_state_t ha; is added in P1-5 (HA wiring) — it is not needed in P0 since no P0 codepath references c->ha. P0-5 adds only persist_t persist;. Also augment the existing include/config.h (do NOT create a new file — the file already exists with XCE_MAXNAME_LEN and job ID bit fields) with all system-level tunable constants (see Appendix A.10): add MAX_STANDBYS=3, MAX_HA_CANDIDATES=4, WAL_FORMAT_VERSION=1, WAL_ENTRY_TYPE_MAX=100, MAX_WAL_PAYLOAD_BYTES=(1<<20), MAX_SNAPSHOT_BYTES=(256*1024*1024), DEFAULT_WAL_SYNC_MS=10, DEFAULT_SNAP_INTERVAL_S=60, etc.

---

P0-B: WAL & COW Infrastructure

Deliverable: WAL writes work; COW freeze/thaw round-trip on zarray. No checkpoint or recovery yet. Requires: P0-A.

Step	File	Description
P0-1	src/controller/persist.c + include/persist.h	WAL writer, segment management, fsync batching. persist_init() receives workdir and name parameters (from CLI --workdir and -n flags). It derives wal_path = effective_dir/data/wal and snap_path = effective_dir/data/snap (where effective_dir = workdir / name). Must validate derived paths against path traversal using realpath() — reject any path that resolves outside the intended base directory or contains .. components — then create directories with mode 0700 (owner-read-only) if they do not exist, and verify permissions on existing directories at startup — refuse to start if derived wal_path or snap_path is not mode 0700 and owned by the controller UID (rather than log-and-continue). Job payloads written to WAL/snapshot files may include environment variables and command lines containing credentials; this permission gate prevents a misconfigured mount from exposing them. Runtime permission re-check: persist_checkpoint() (P0-3) must also call fstatat(dirfd(wal_dir), ".", &st, 0) and verify (st.st_mode & 0777) == 0700 && st.st_uid == getuid() before writing each new snapshot. Log ERROR and skip the snapshot write if the check fails — a post-startup permission change (e.g., misconfigured backup agent chmod) is caught within one checkpoint interval (default 60s) rather than persisting until the next restart. Add persist_t.wal_dirfd and persist_t.snap_dirfd (open directory file descriptors) so the check uses fstatat() rather than stat() to avoid TOCTOU race against path traversal. Dedicated fsync thread required: wal_sync_ms defaults to 10ms; delaymgr has 2-second granularity (sleep(precision) where precision=2) so the WAL fsync timer must be a dedicated pthread using nanosleep() or timerfd_create(CLOCK_MONOTONIC) — do not use delaymgr for this. Encryption note: WAL and snapshot files are stored unencrypted by the controller. persist_init() must log an INFO message at startup: "WAL/snapshot stored unencrypted; ensure effective_dir/data/wal and effective_dir/data/snap reside on an encrypted filesystem (e.g., LUKS/dm-crypt) if job payloads contain sensitive data". File-level encryption (e.g., AES-GCM per-record) is out of scope for this design.
P0-2	src/lib/zarray.c + include/zarray.h	COW extensions: shadow, shadow_caps, xzarray_freeze(), xzarray_thaw(); modify xzarray_iter_next() for shadow-aware merge walk (see Part 3.5); modify xzarray_freeze() to set pos = shadow_caps (see Part 3.5 vstore invariant). Header move required: xzarray_t is currently defined as a file-private struct in zarray.c; it must be moved to include/zarray.h so persist.c, repl.c, and other callers can embed or reference it. xzarray_private_t (the internal struct containing mylock_t) is also currently file-private in zarray.c and must be moved to include/zarray.h as well. _Atomic _Bool inuse audit (mandatory): changing xzarray_item_t.inuse from unsigned int :1 to _Atomic _Bool (required by Part 3.5 memory ordering) affects every read and write of this field in zarray.c. Run grep -n '\.inuse' src/lib/zarray.c before starting — expect ~15 call sites across xzarray_destroy, xzarray_store, xzarray_get, xzarray_clear, xzarray_iter_next, and the new COW paths. Every read must use atomic_load_explicit(..., memory_order_acquire) and every write must use atomic_store_explicit(..., memory_order_release). This is a checklist item — do not mark P0-2 complete until all sites are converted and verified. Requires: P0-0a — P0-0a merges zarray_t into xzarray_t; P0-2 must not begin until P0-0a is merged and tested.
P0-2a	src/lib/zarray.c	COW lock protocol: verify xzarray mylock_t mutex is taken correctly inside vget/vput during COW mode; kvmgr my_rwlock_t is the outer coordination lock (see Part 3.9)
P0-2b	src/controller/delay.c + src/controller/delay.h + all callers	delaymgr precision enhancement (prerequisite for P0-3): rename precision field to precision_ms (milliseconds); replace sleep(precision) with struct timespec ts = {0, precision_ms * 1000000L}; nanosleep(&ts, NULL);. Important: delay_task() delay parameter remains in seconds — only the sleep granularity changes. Audit all delaymgr_init() call sites (grep -rn 'delaymgr_init' src/) and update each to pass precision_ms = 200. Verify no existing delay_task() call passes seconds-based values that assume the old 2-second tick — the time_t arithmetic (ts_nextrun = now + delay) is unchanged. Update cluster.c:604 from delaymgr_init(&c->delaymgr, 2, run_delay_tasks) to delaymgr_init(&c->delaymgr, 200 /* precision_ms */, run_delay_tasks).

---

P0-C: Checkpoint & Recovery

Deliverable: Full persist/recover round-trip — controller survives restart. Requires: P0-B.

Step	File	Description
P0-3	src/controller/persist.c	Requires: P0-2b (delaymgr precision enhancement — 1-second checkpoint poll requires nanosleep granularity). Checkpoint via dual zarray freeze → serialize shadow → thaw + GC; use delaymgr for periodic checkpoint timer (see Part 3.10). checkpoint_requested inline trigger polling: The checkpoint timer thread also polls persist_t.checkpoint_requested on each timer wake. The timer callback is registered with a 1-second interval via delay_task(&c->delaymgr, 1, -1, persist_checkpoint_timer_fn, c). Each callback checks checkpoint_in_progress first (skip if set), then checkpoint_requested (run checkpoint if set), then snap_interval_s countdown (run periodic checkpoint when countdown reaches zero). The WAL fsync timer (10ms intervals) and HA heartbeat watchdog both require dedicated pthreads — they cannot use delaymgr.
P0-4	src/controller/persist.c	Recovery: load snapshot + replay WAL. Includes: (1) snapshot version check — reject snapshots where SnapshotHeader.version > WAL_FORMAT_VERSION (see Part 2.5 step 2a); (2) Cap'n Proto traversal limit — set ReaderOptions.traversalLimitInWords to the fixed upper bound MAX_SNAPSHOT_BYTES / 8 words (see Part 2.5 step 2b; do NOT use file_size / 8); (3) snapshot CRC verification over both header (headerChecksum covering lsn, term, version, counts) and body (checksum); (4) WAL entry type max guard — abort if any entry type exceeds WAL_ENTRY_TYPE_MAX (see Part 2.5).
P0-12a	job.c	Implement restore_job(cluster_t* c, uint64_t jobid, const void* capnp_payload) — inserts a job at its original jobid without allocating a new one. Used by apply_wal_entry() for WAL_JOB_SUBMIT during recovery and standby replay. Unlike job_submit(), restore_job() must accept an explicit jobid, bypass the jobid allocator, and skip BEFORE/AFTER_EVENT hooks. Duplicate jobid handling: if the jobid already exists in the kvmgr (e.g., from a duplicate WAL entry or a corrupt WAL that replays the same submit twice), restore_job() must log a WARN and return -EEXIST without overwriting the existing entry. apply_wal_entry() treats -EEXIST as a non-fatal skip. Required before P0-4 (recovery replay) and P2-2 (standby apply). Note on -EEXIST occurrence: During clean recovery (snapshot + WAL replay from a trimmed WAL), -EEXIST should never occur — the snapshot represents a consistent point-in-time state, and WAL entries after the snapshot's LSN contain only new state changes. -EEXIST is a safety guard for crash-recovery edge cases where: (1) a checkpoint failed mid-write but the WAL was not yet trimmed, causing the recovery algorithm to replay entries already present in the snapshot; or (2) a standby receives a duplicate replication entry due to reconnection. If -EEXIST occurs during what should be a clean recovery, it indicates a WAL trim bug or snapshot/WAL inconsistency and should be investigated.
P0-12	worker.c	Wire restore_worker() into recovery path. Adapter requirement: the existing restore_worker(void* cls, xce_worker_t host) takes xce_worker_t by value (struct copy), but Cap'n Proto deserialization produces a pointer-based Worker struct. Implement a wrapper worker_from_capnp() that: (1) receives the deserialized Worker Cap'n Proto object, (2) allocates and populates an xce_worker_t by deep-copying pointer fields (char* name → strdup, xce_config_t** resources → allocate and copy each config item, xce_res_threshold_t** thresholds → same), (3) calls restore_worker(cls, host) with the populated struct. The existing restore_worker() function signature is unchanged — the adapter sits between Cap'n Proto and restore_worker().

---

P0-D: Hooks & Reply Ordering

Deliverable: Persistence hooks fire on AFTER_EVENT; xce_reply() moved after hooks for sync durability. Requires: P0-C (hooks need WAL to write to).

Step	File	Description
P0-4a	src/include/xce/cluster_def.h + src/controller/job.c	Add HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME to the hook detail enum. Critical codebase fact: on_job_stop() and on_job_resume() in job.c currently call mark_job_update() which fires HOOK_DETAIL_JOB_UPDATE — they do NOT currently fire HOOK_DETAIL_JOB_EXIT. (Footnote 1 in Part 2.3 incorrectly states they fire HOOK_DETAIL_JOB_EXIT.) The audit directive must search for mark_job_update calls in the stop/resume paths, not HOOK_DETAIL_JOB_EXIT. The implementation must update on_job_stop() and on_job_resume() to call xce_run_hooks directly with HOOK_DETAIL_JOB_STOP / HOOK_DETAIL_JOB_RESUME (bypassing mark_job_update() for those events). Audit: grep -n 'mark_job_update|HOOK_DETAIL_JOB' src/controller/job.c — confirm all non-terminal paths are correctly remapped. This is a prerequisite for P0-6 WAL_JOB_STOP and WAL_JOB_RESUME hook wiring.
P0-5a	src/include/xce/cluster_def.h + types.h	Add entity terminal states: XCE_WORKER_UNAVAIL (already exists in types.h:165), XCE_WORKER_HIDDEN, QUEUE_STAT_HIDDEN; add JOB_STAT_TRANSIENT = 0x400 to job status bitmask (must not use 0x200 — JOB_STAT_WAIT already occupies 0x200, see types.h:351). This fits in the existing uint32_t state field (current highest bitmask is 0x40000 for JOB_STAT_ERR). The scheduler must skip workers that own transient jobs (no new work dispatched to them). API queries for transient jobs must NOT return the job's last-known state (e.g., PEND) as that would actively mislead users who submitted running jobs. The Go API server mapping of JOB_STAT_TRANSIENT to a user-visible TRANSIENT response is tracked separately — see P2.5-7 (or a dedicated API server story). Worker reconnection resolves the transient state by setting the job's actual terminal/running status (the worker reports job completion/exit). If the worker never reconnects, the job remains transient until operator intervention (same as existing WORKER_UNAVAILABLE behavior for workers).
P0-6	src/controller/persist.c	Hook functions: persist_job_hook_fun, persist_queue_hook_fun, persist_worker_hook_fun (requires P0-4a for HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME wiring)
P0-7	src/controller/job.c + all handler files	Move xce_reply() after AFTER_EVENT hooks in handle_jobsubmit(); verify no kvmgr lock held during hooks (see Part 8.3 lock safety analysis). Full audit required: grep for all xce_run_hooks(AFTER_EVENT + xce_reply pairs across all handler files (job.c, worker.c, join.c, cluster.c, etc.) to identify any handler that still replies before post-hooks. va_list compatibility audit: P0-6 adds persist_job_hook_fun() as an AFTER_EVENT hook for HOOK_DETAIL_JOB_NEW that reads two extra va_list arguments (int* sync_rc and persist_batch_ctx_t*). All existing registered AFTER_EVENT HOOK_DETAIL_JOB_NEW hooks (e.g., in apiserver.c, sched.c) must be audited to verify they do NOT call va_arg() for these extra arguments (they should only consume the arguments they were registered to expect). Document for each: (a) does the hook call va_arg(), (b) for which arguments, (c) whether adding the new args shifts its position. If any existing hook calls va_arg() past the standard args, it must be updated or given its own hook detail to avoid type mismatches. For async-replicated events, reply ordering is less critical, but consistency is preferred. Document each handler's reply ordering decision and rationale as part of this step.

---

P0-E: Deferred Cleanup & GC

Deliverable: Entity lifecycle management — terminal entities cleaned at checkpoint, endpoints deferred. Requires: P0-C (checkpoint GC needs thaw infrastructure).

Step	File	Description
P0-8	src/controller/worker.c	Extend existing mark_worker_unavail() to set w->host.dead_since = time(NULL) (field already exists in xce_worker_t); add XCE_WORKER_HIDDEN bitmask path; already deferred — no zarray_del removal needed (see Part 4.10)
P0-8a	src/controller/cluster.c	Fallback path (no dependency on P0-0a): Keep the existing zarray_del()/zarray_clear() calls in handle_endpoint_error() — this is the safe baseline that ships with P0-E. Mark these call sites with /* TODO P0-8b: replace with deferred cleanup after P0-0a */ comments so they are easy to find. P0-8a is complete when these comment markers are in place.
P0-8b	src/controller/cluster.c	Deferred cleanup path — requires P0-0a to be merged and green first. Replace the TODO P0-8b sites from P0-8a: close fd/SSL immediately, set ENDPOINT_DISCONNECTED state, clean up the slot at checkpoint via endpoint_should_retain(). P0-8b must not be started until P0-0a is merged and all zarray_t → xzarray_t migration tests pass. This ordering gate prevents the deferred path from shipping before the zarray infrastructure is ready.
P0-13	job.c	Implement job_is_terminal() (check `JOB_STAT_EXIT
P0-14	kvmgr.c + htab.c	Implement kvtab_gc_stale_indices() — walk hash chains, remove entries whose mapped zarray index has inuse=0. Called during thaw while kvmgr write lock is held. Internally uses ht_del() (not del_key — remains NULL per Part 4.1 invariant)
P0-14a	kvmgr.c	Implement kvmgr_capture_keys(meta_obj_t* kvmgr, slot_key_t** keys_out, size_t* n_keys_out) — walks the kvtab under caller-held write lock and deep-copies each key string into a malloc'd slot_key_t[] array. Called by persist_checkpoint() in Part 3.8 immediately after xzarray_freeze() while the write lock is still held. Depends on: P0-14 (htab walk primitives) and P0-0b (slot_key_t struct). Required before: P0-3 (checkpoint) can be tested end-to-end. The serialization pthread uses the captured key array to build Cap'n Proto key→index mappings without holding any lock. free(keys) releases the array; individual key strings are freed by the keys array destructor.

---

P0-F: Schema, Config, & Cluster Wiring

Deliverable: Full end-to-end — config parsed from YAML, schemas defined, persistence wired into cluster lifecycle. Requires: P0-D + P0-E.

Step	File	Description
P0-9	controller.capnp	Add PersistConfig, WALEntry, SnapshotHeader, Snapshot schemas; add version field (UInt32, default 1) to SnapshotHeader for format version detection; add snapKeepN @6 :UInt32 = 2 to PersistConfig (see Appendix A.5); add transientJobTimeoutS @7 :UInt32 = 300 to PersistConfig (see Appendix A.5 and persist_config_t.transient_job_timeout_s); add transientSince (UInt64, epoch seconds) to the Job schema — set when a job enters JOB_STAT_TRANSIENT at promotion; needed for the transient_job_timeout_s GC check; add deadSince to Worker schema; add bitmask flags for XCE_WORKER_HIDDEN, QUEUE_STAT_HIDDEN; add jobStatTransient = 0x400 constant comment in Job schema (status remains UInt32 bitmask; not 0x200 which is JOB_STAT_WAIT)
P0-10	cluster.c	Wire persist_init/recover/start/stop/fini into cluster_init/run/fini. Requires: P0-9 — PersistConfig and the snapshot schemas must be defined in controller.capnp before cluster.c can reference them via the generated C bindings.
P0-11	myconf.c + YAML config	Parse persistence configuration

---

P0-G: End-to-End Smoke Tests

Deliverable: Integrated persistence test suite exercising cross-sub-phase interactions. Requires: P0-F.

Gate structure note: These tests run AFTER all P0 sub-phases (P0-A through P0-F) are complete. They are end-to-end integration tests, not per-sub-phase gates. For per-sub-phase incremental verification, each sub-phase's deliverable should be tested immediately after it lands (see the "each sub-phase produces a buildable, testable increment" principle). P0-G tests validate the full assembled system.

Step	File	Description
P0-15a	Tests (WAL E2E)	WAL write/fsync/CRC round-trip: write entries, simulate crash (truncate), recover and verify correct replay up to last valid CRC
P0-15b	Tests (Checkpoint E2E)	Checkpoint + recovery: freeze → serialize → thaw cycle; snapshot-to-disk; restart from snapshot + bridging WAL; verify entity counts match
P0-15c	Tests (Hook E2E)	Hook invocation ordering: BEFORE_EVENT abort path, AFTER_EVENT sync_rc propagation, reply-after-hook sequencing; xce_reply fires after hook returns
P0-15d	Tests (GC E2E)	Retention/GC: terminal entities survive until TTL expires, cleaned at checkpoint; deferred endpoint cleanup; JOB_STAT_TRANSIENT auto-terminate path
P0-15e	Tests (Cap'n Proto E2E)	Cap'n Proto partial-merge semantics: absent optional fields are NOT zeroed; partial WAL entry applies only stated fields to in-memory struct

P1: HA Foundation (LSF-Style Election)

Goal: Standby controller can take over when primary fails.

Step	File	Description
P1-1	src/controller/ha.c + src/controller/netio.c	Verify that HA listeners (election probe, replication accept) reuse the existing iomgr/SSL pipeline without modification. No create_ssl_context() API changes needed — the trusted LAN model (Part 5.9) requires no mTLS. Peer identity validated by host:port match against --peers candidate list.
P1-2a	src/lib/hook.c + src/lib/include/xce/hook.h	Prerequisite for P1-2. Add xce_disable_hook(hooksys_t* hs, int type, hook_fn_t fn) — scans for a matching entry by (type, fn) and stores NULL atomically with atomic_store_explicit(memory_order_release). Add void xce_disable_hook(...) to hook.h. Struct change: change hook_item_t.fn from hook_fn_t fn; to _Atomic hook_fn_t fn; — required so xce_disable_hook() can store NULL with release ordering and xce_run_hooks() can load with acquire into a local before NULL-check and call. Update xce_add_hook() to use atomic_store_explicit(&it->fn, fn, memory_order_release). NULL-fn guard in xce_run_hooks(): after loading hook_fn_t local_fn = atomic_load_explicit(&it->fn, memory_order_acquire), add if (!local_fn) continue; — call local_fn(...) instead of it->fn(...). Slot reuse: xce_add_hook() scans for existing NULL-fn slots of the same type before appending — prevents unbounded list growth on promote→demote cycles. Note: xce_add_hook() currently rejects NULL fn pointers; the slot-reuse scan path must only match slots where an entry was previously registered and then disabled (not freshly zeroed).
P1-2	src/controller/ha.c + include/ha.h	Requires: P1-2a. Role management, am_i_primary(), promotion/demotion. OP_steppingDown handler: on receiving OP_steppingDown from the primary, the handler validates: (1) the sender's term matches ha->term (stale messages ignored), (2) the sender is the current primary (verified via ha->primary_host/ha->primary_port). Priority resolution mechanism: upon validation, each standby immediately runs am_i_primary() — the same election probe used at startup. This naturally resolves which standby is the highest-priority reachable candidate: each standby probes all higher-priority candidates; if none respond within election_timeout_ms, it promotes itself. This avoids the impossible requirement of knowing which other standbys were connected to the stepping-down primary at the time of the message. Non-highest-priority standbys that discover a higher-priority peer is reachable (via the probe returning "you are NOT primary") defer and wait for the new primary's heartbeat. The am_i_primary() call runs in a temporary pthread (same as startup) so the handler does not block the iomgr event loop. The handler runs in the iomgr/vtask thread for initial validation; the actual am_i_primary() probe is dispatched asynchronously.
P1-3	src/controller/ha.c	Heartbeat watchdog: OP_heartbeat exchange via iomgr/vtask pipeline; ha_check_heartbeat() on a dedicated pthread with nanosleep() or timerfd_create(CLOCK_MONOTONIC) (NOT delaymgr — see Part 5.5 heartbeat detection latency note) detects primary failure (replaces tcp_probe). Shutdown protocol (mirrors checkpoint thread pattern): add _Atomic int ha_watchdog_stop to ha_state_t. The watchdog thread checks this flag on each nanosleep() wake and exits cleanly when set. ha_fini() sets the stop flag then calls pthread_join() on the watchdog thread before returning — this prevents the watchdog from accessing freed ha_state_t (embedded in cluster_t) after cluster_fini() frees it. ha_fini() must be called from cluster_fini() before any ha_state_t memory is reclaimed. The watchdog thread must not be cancelled with pthread_cancel() — the same rationale as election probe threads (OpenSSL handlers are not cancellation-safe; see Part 5.4 startup note).
P1-3a	src/controller/ha.c	Promotion sequence: stop replay, increment term, recover, start primary; SIGUSR1 handler for manual failover (see Part 5.8). Requires: P1-2a — ha_demote_to_standby() calls xce_disable_hook() which is added in P1-2a. Do not start P1-3a until P1-2a is merged and tested.
P1-4	controller.capnp	Add term to Request, standby to HostRole, replEntry/replAck/redirect/heartbeat/steppingDown/replSnapshot/replCatchUpDone opcodes (see Part 5.10 for full opcode list), RedirectInfo struct
P1-5	cluster.c	Wire HA into startup sequence; add ha_state_t ha; to cluster_t in this step (note: persist_t persist; field was already added to cluster_t in P0-5)
P1-6	join.c	Add redirect logic in handle_join() — standby sends REDIRECT for worker/scheduler/apiserver join requests
P1-7	Tests	Election logic, promotion sequence, term fencing, redirect response

P2: Replication (Fan-Out TCP)

Goal: Primary replicates state changes to standbys in real time.

Step	File	Description
P2-1	src/controller/repl.c + include/repl.h	Replicator: fan-out send via endpoint/iomgr SSL pipeline, ACK tracking; standby_conn_t.ep replaces standby_conn_t.fd (see Part 6.5)
P2-2	src/controller/repl.c	Standby receiver: OP_replEntry dispatched via iomgr/vtask pipeline; OP_replAck sent via endpoint outq (not raw send()). apply_wal_entry() does not fire hooks — persistence hooks are registered only on the primary path (in cluster_init() after recovery completes, or in ha_promote_to_primary() step 5 on promotion). No bypass flag is needed; the hook system is empty on standbys. Standbys apply WAL entries the same way recovery does (see Part 4.2 recovery)
P2-3	src/controller/repl.c	Sync replication: repl_wait_ack() with timeout; only triggered for WAL_JOB_SUBMIT from API server connections. ACK notification mechanism: when the standby's OP_replAck arrives via iomgr/vtask, it must wake the blocked repl_wait_ack() call. Implement with a per-standby condition variable (pthread_cond_t) under repl.lock: the vtask handler signals it on receiving OP_replAck; repl_wait_ack() parks on it with pthread_cond_timedwait(). This keeps the blocking confined to the vtask thread running the handle_jobsubmit() path. Pool floor note: the default vcpu pool is 8 (myconf.c default). Under burst job submission, 8 simultaneous sync-replicating submits exhaust the pool. Clusters using sync replication should configure vcpus >= 16: the reasoning is that each blocked repl_wait_ack() occupies one vcpu for up to sync_timeout_ms (50ms default); at the typical API server connection count of 1–3, the realistic burst concurrency is low, but with multiple API servers sending simultaneous submits the pool can saturate quickly. 16 vcpus provides 2× headroom over the default and headroom for non-submit vtasks (worker joins, heartbeats) that must not be starved. Clusters with only 1 API server and low submit rates can stay at 8. Startup enforcement: persist_start() must log a WARN at startup if sync replication is enabled (sync_timeout_ms > 0) and the configured vcpus < 16: WARN("sync replication enabled with vcpus=%d (< 16); recommend vcpus >= 16 to prevent repl_wait_ack pool saturation", vcpus). This is a warning, not a hard failure, so low-concurrency deployments can still operate. Operators should monitor ctrl_vcpu_queue_depth and increase vcpus if it stays above 0 during normal operation. ACK deadlock prevention: repl_on_ack() must be callable directly from the io_worker read callback (i.e., from handle_msg() for OP_replAck before vtask dispatch) — it only does pthread_cond_signal() and an acked_lsn update under repl.ack_lock, which is safe to call from io_worker without going through vtaskmgr. If OP_replAck were dispatched via vtaskmgr like all other messages, a pool of 8 vcpus all blocked in repl_wait_ack() would deadlock: the incoming ACK sits in the vtask queue, but no vcpu is free to process it. By handling ACKs in io_worker directly, the signal wakes a sleeping vcpu regardless of pool saturation. Worker join lock constraint: handle_worker_join() holds workermgr WRITE lock during add_worker() + AFTER_EVENT hooks + xce_reply(). The persistence hook for HOOK_DETAIL_WORKER_JOIN must use persist_wal_async() only — worker join is always async-replicated and must never block on repl_wait_ack while the write lock is held
P2-4	src/controller/persist.c	Wire sync/async replication into persist_wal_sync_append() / persist_wal_async()
P2-5	src/controller/repl.c	Standby catch-up: when standby reconnects via endpoint, stream missing WAL entries. The primary trims WAL only up to min(acked_lsn) across connected standbys (standby_conn_t.acked_lsn; falls back to checkpoint_lsn when no standbys — see Part 2.4 WAL trim guard). If a reconnecting standby needs WAL entries older than the primary's oldest surviving segment, transfer the latest snapshot first, then stream WAL entries from the snapshot's LSN forward. Snapshot atomic write on standby receive: the standby writes reassembled snapshot chunks to a temp file snap/snap-<lsn>.tmp, verifies the CRC32 after the final chunk, then uses rename(2) to atomically promote to snap/snap-<lsn>.capnp. This prevents a partial snapshot file from being discovered by persist_recover() on a subsequent restart. The standby snap/ directory must have mode 0700 (same startup check as the primary — verify this is covered by the standby's persist_init() call). OP_replCatchUpDone handler (primary side): when the primary receives OP_replCatchUpDone from a standby, it must: (1) locate the matching standby_conn_t slot by endpoint pointer, (2) clear slot->in_catch_up with atomic_store_explicit(&slot->in_catch_up, 0, memory_order_release) — this unblocks repl_wait_ack() from excluding that standby, so its subsequent OP_replAck messages count toward the "at least one ACK" predicate. The handler runs in the io_worker read callback (same as OP_replAck — not dispatched via vtaskmgr) to avoid a vtask queue latency spike after catch-up. If no matching slot is found (standby disconnected between sending OP_replCatchUpDone and the primary processing it), log WARN and discard silently.
P2-6	Tests	Replication round-trip (via SSL pipeline), sync ACK timeout, standby catch-up

P2.5: libxce Smart Discovery

Note

: P2.5 carries the same implementation scope as P1 or P2 (8 steps). It is numbered 2.5 — not 3 — to reflect that the client library code (P2.5-1 through P2.5-6) depends only on P1 and can be developed in parallel with P2. However, the functional value of P2.5-7 (API server discovery, RegisterToBackend()) is only fully realized when P2 replication is also deployed: without P2, a promoted standby has no live replication from the primary, so reconnecting the API server to a promoted standby via P2.5-7 serves no purpose. Deploy P2 and P2.5 together for a complete HA experience. P2.5 alone (without P2) provides worker/scheduler auto-reconnect after failover but does not provide replication continuity.

Candidate count ceiling: MAX_STANDBYS = 3 and MAX_HA_CANDIDATES = 4 are standalone constants defined directly in include/config.h (see A6-010). The deployment ceiling is ≤3 controllers (1 primary + 2 standbys). xce_cluster_config_t must enforce the structural ceiling when populated from --peers CLI flag (reject configs with more than MAX_HA_CANDIDATES candidates). Operators are strongly advised to deploy no more than 3. Add a WARN log at startup if candidates_count > 3: "HA cluster has %d candidates; LSF-style election is validated for ≤3. Larger clusters increase split-brain risk."

Goal: Workers and schedulers can discover the primary via any candidate (P2.5 alone). API server discovery (P2.5-7) is also included here but only delivers value when deployed together with P2 replication — without P2, the promoted standby has incomplete state and reconnecting the API server to it is misleading rather than helpful. Deploy P2.5-7 only alongside P2. Design decision: libxce auto-discovery and auto-reconnect are foundational, not optional. All client-side controller connection code in the codebase will be migrated to this path — there is no "raw TCP to controller" fallback after this phase. This ensures the scheduler policy enhancement work that follows operates on a stable HA foundation.

Step	File	Description
P2.5-1	src/lib/include/xce/cluster.h	Define xce_cluster_config_t, xce_candidate_t, API functions
P2.5-2	src/lib/cluster.c (new)	Implement xce_connect_cluster(): candidate selection, redirect following, backoff+jitter. SSL: all controller connections use SSL (enforced by netio.c); xce_connect_cluster() must perform the TLS handshake and return an SSL-wrapped connection, not a raw fd. Use the same CA cert path from xce_cluster_config_t for server certificate verification — `SSL_CTX_set_verify(SSL_VERIFY_PEER
P2.5-2a	src/lib/meson.build	Add cluster.c to the libxce_src source list so the new file is compiled into libxce
P2.5-3	src/lib/cluster.c	Implement xce_reconnect(): auto-reconnect with state machine
P2.5-4	src/lib/cluster.c	REMOVED: no xce_load_cluster_config() function. Workers/schedulers/apiserver parse --peers and --ca-cert CLI flags in their main() and populate xce_cluster_config_t directly before calling libxce functions.
P2.5-5	src/worker/main.c	Replace hardcoded controller address with xce_cluster_config_t; populate from --peers and --ca-cert CLI flags in main(); add reconnection loop. Do NOT read any config file.
P2.5-6	src/scheduler/	Same pattern as worker — parse --peers and --ca-cert CLI flags, populate xce_cluster_config_t, use libxce discovery. Do NOT read any config file.
P2.5-7	src/frontend/apiserver/backend/clib.go	Use cluster discovery for controller connection in RegisterToBackend(). Decision: use CGo binding (option a). Expose xce_cluster_config_t and xce_connect_cluster() through the existing clib.go CGo shim — clib.go already imports xce headers and wraps C calls for other controller interactions, so adding cluster discovery there is consistent and avoids duplicating config parsing logic in Go. Add a #include "xce/cluster.h" import and a XceConnectCluster(conf *C.xce_cluster_config_t) int Go wrapper, then call it from RegisterToBackend() instead of the current hardcoded dial.
P2.5-7a	src/frontend/apiserver/	Go API server: JOB_STAT_TRANSIENT serialization — Update the Go jobstatus serializer to map the JOB_STAT_TRANSIENT bitmask to { "status": "TRANSIENT", "status_detail": "job state uncertain after controller failover — awaiting worker reconnect" }. Without this, users querying jobs after failover receive stale PEND state, which actively misleads them. This is a standalone story that does not depend on P2.5-7 controller discovery — it only depends on P0-5a (which defines the JOB_STAT_TRANSIENT bitmask). Should ideally ship concurrent with or immediately after P0. Deliverable: add case JOB_STAT_TRANSIENT: return "TRANSIENT" (or equivalent) to the job status switch in the Go API serializer; add a test case.
P2.5-8	Tests	Candidate selection, redirect following, backoff+jitter, reconnection

P3: Integration & Hardening

Goal: Production-ready persistence + HA.

Step	Description
P3-1	End-to-end failover test: kill primary, verify standby promotes, workers reconnect via libxce discovery
P3-2	Redirect test: worker connects to standby, receives redirect, reconnects to primary
P3-3	WAL corruption handling: test torn writes, partial segments, CRC mismatch
P3-4	Performance: benchmark 1000 job submits/sec with sync replication
P3-5	Checkpoint GC: verify expired jobs/workers/queues are cleaned; verify dependents block cleanup
P3-6	Test the manual failover state machine path by calling ha_demote_to_standby() directly from a test harness (e.g., a test-only ha_trigger_manual_failover(c) shim or a dedicated integration test binary). This validates that the HA state machine correctly handles a manual promotion trigger — role transitions, WAL flush, hook re-registration — without requiring the unimplemented xce hostcontrol --failover CLI (see Out of Scope section).
P3-7	Metrics: WAL size, replication lag, checkpoint duration, standby health, entity GC stats. Metric naming convention: All HA/persistence metrics use a ctrl_ prefix (e.g., ctrl_repl_degraded_count, ctrl_split_brain_detected, ctrl_async_events_lost) to namespace them within the controller's existing metrics perimeter. Metric names must match the pattern ctrl_[a-z_]+ and should be registered in a central metrics table with a description and type (counter/gauge/histogram). The implementation should define this table as part of persist.c or ha.c with a metrics_init() call in persist_start() / ha_init(). Split-brain alert: log a WARN-level message and expose a ctrl_split_brain_detected counter whenever a term conflict is detected (primary receives a message with a higher term than its own, or standby detects two primaries). This is the minimum operator signal that the cluster has entered a split-brain state. Async event loss counter: expose ctrl_async_events_lost — incremented for each async WAL entry that was in-flight (written locally but not yet replicated) when the primary failed. Computed by the newly promoted standby during promotion: events_lost = primary_last_lsn - standby_applied_lsn (where primary_last_lsn is recovered from the old primary's WAL at promotion time, if available; otherwise log WARN("async event loss count unavailable — primary WAL not accessible")). This gives operators visibility into the amount of async data loss after a failover. Transient job auto-termination counter: expose ctrl_transient_jobs_stuck (counter, type: counter) — incremented each time a JOB_STAT_TRANSIENT job is auto-terminated by job_should_retain() after transient_job_timeout_s elapses without worker reconnection. Accompanies a WARN-level log message with jobid and worker name. Allows operators to detect and quantify failover-induced job loss without reviewing individual log entries.

P4: Cleanup & Documentation

Step	Description
P4-1	Update src/frontend/docs/ to remove stale PostgreSQL references
P4-2	Update CLAUDE.md files with persistence/HA architecture
P4-3	Operational runbook: how to add/remove standby, manual failover, disaster recovery. Must include candidate list upgrade sequencing: (1) update --peers on all controllers first and restart them (they continue operating with the new list); (2) then update workers, schedulers, and API servers. Never update workers before controllers — a worker with a candidate list that includes a new controller before that controller knows about itself cannot connect during the transition. Add a section "Changing the candidate list" with this explicit sequence. Also include: what ctrl_split_brain_lost_submits means and how to notify affected users; how to interpret ctrl_reconciliation_stuck_workers and what manual steps to take; the JOB_STAT_TRANSIENT user-facing meaning and when to expect it.

Out of Scope

The following are explicitly not covered by this design document and are deferred to future work:

• Admin CLI for manual failover: xce hostcontrol --failover is not designed here. Manual failover is exercised in P3-6 via ha_demote_to_standby() called from a test harness. A production CLI surface for this operation is deferred to future work
• WAL compaction beyond checkpoint trimming: Checkpoint-based WAL segment deletion is specified, but online compaction (merging small segments, garbage-collecting within segments) is out of scope
• Multi-data-center replication: All replication targets are same-DC controllers connected via a dedicated management network. Cross-DC replication would require bandwidth-aware batching and is not addressed
• Monitoring/alerting integration: P3-7 lists metrics to expose, but the integration with Prometheus, Nagios, or other monitoring systems is out of scope
• Runtime configuration changes: Adding/removing standby candidates at runtime, changing persist_config_t values without restart, and online checkpoint interval adjustment are not designed here
• Partial WAL re-sync: After partition healing, the standby performs full re-sync from the primary. Incremental re-sync (sending only divergent entries) is out of scope
• Snapshot compression: Snapshots are stored as uncompressed Cap'n Proto. Compression (zstd, lz4) is a potential optimization deferred to future work
• Config version hash in heartbeat messages (Part 10.8): Deferred to P3 (operational tooling). Not needed for core discovery, failover, or connection behavior. The configVersion field in the Request Cap'n Proto schema and the CRC freshness signal are removed from P1 scope.
• Degraded-mode restricted state (Part 7.4 repl_degraded_timeout_s): Removed from scope. The "restricted mode" where the primary refuses WAL_JOB_SUBMIT after 300s without a standby ACK is a product policy decision, not a persistence or HA requirement. The existing sync+timeout+local-WAL fallback already meets the stated durability goal. Monitoring and operator escalation handle sustained degradation. repl_degraded_timeout_s is removed from persist_config_t — if HA policy needs a degradation tolerance field, it belongs in ha_config_t.
• MAX_HA_CANDIDATES = 16: Reduced to 4 in include/config.h. The design is explicitly validated for ≤3 controllers. MAX_STANDBYS reduced from 15 to 3. Setting structural limits far beyond operational needs is speculative generality.

Review Resolutions

The following issues were identified during multi-perspective review and resolved in this version:

ID	Finding	Resolution
R-1	traversalLimitInWords set to file_size / 8 provides no security (P0)	Changed to fixed upper bound MAX_SNAPSHOT_BYTES / 8 words
R-2	Job ID allocator not reset after WAL replay (P0)	Superseded by A6-014: the premise was incorrect — jobids are allocated by the Go API server's idgen, not the controller. The controller is a passive consumer that restores jobs at their original IDs during recovery. No max_replayed_jobid tracking or idgen_set_next() call is needed. See recovery step 3 note and A6-014.
R-3	acked_lsn not declared _Atomic in struct (P0)	Changed standby_conn_t.acked_lsn to _Atomic uint64_t
R-4	ha.role not atomic — role-change window incomplete (P1)	Changed ha_state_t.role to _Atomic ctrl_role_t, documented acquire/release requirements
R-5	Missing hook idempotency guard (P0)	Added hooks_registered flag to ha_state_t, required in ha_promote_to_primary()
R-6	checkpoint_mutex missing from persist_t (P1)	Added pthread_mutex_t checkpoint_mutex field
R-7	WALEntry Cap'n Proto schema confused with on-disk format (P1)	Clarified that WALEntry is replication-only; on-disk uses raw C wal_entry_hdr_t
R-8	Snapshot backward compatibility not specified (P1)	Added backward-compatibility note to recovery step 6
R-9	Split-brain detection condition unspecified (P1)	Added detection specification in Part 7.4
R-10	Heartbeat interval minimum inconsistent (P1)	Standardized to heartbeat_interval_ms >= 2000 across all sections
R-11	repl_degraded_timeout_s in wrong config struct (P1)	Removed from scope (see Out of Scope above)
R-12	WAL_CHECKPOINT entries skipped creates CRC chain gaps (P2)	Documented in Part 6.4 that skipped checkpoint entries produce expected single-entry CRC gaps
R-13	Snapshot total_size used for direct malloc without boundary check (P1)	Added explicit requirement: validate total_size <= MAX_SNAPSHOT_BYTES BEFORE malloc()
R-14	CRC32 for WAL integrity not tamper-evident (P2)	Documented threat model: CRC32 protects against accidental corruption; HMAC-SHA256 would be required for adversarial tamper resistance
R-15	WAL payload logging relies on code review only (P1)	Added recommendation for structural enforcement via opaque wal_payload_t wrapper type
R-16	vstore COW growth re-check uses OOB a->data[old_caps].inuse (P0)	Fixed to check a->caps <= old_caps (capacity comparison) instead of OOB memory access; re-read free index after growth
R-17	Standby WAL write return value unchecked before ACK (P0)	Added return value check on persist_wal_replicated_entry() — disconnect on failure, no ACK on error
R-18	Chained CRC absent from replication stream (P0)	Added hdr.prev_crc validation against ha->last_received_crc on standby; last_received_crc field added to ha_state_t
R-19	Replicated entry LSN handling unspecified (P1)	Added persist_wal_replicated_entry() function — writes primary's LSN/CRC directly, updates current_lsn and last_crc to match
R-20	seg_offset in-memory and lost on crash — recovery trust boundary undefined (P1)	Clarified: recovery scans full segment file using CRC validity, not seg_offset — seg_offset is runtime-only for fsync tracking
R-21	WAL segment rotation prev_crc == WAL_CRC_NONE handling during recovery unspecified (P1)	Documented: WAL_CRC_NONE at segment-aligned position is valid chain break, not corruption
R-22	COW vstore find-slot-and-write not atomic — concurrent slot overwrite (P0)	Wrapped entire COW-mode find-slot-and-write in mylock_lock(&priv->lock); simplified growth path
R-23	vput grow path calls xzarray_ensure_items() without mutex during COW (P0)	Resolved by R-22 fix — entire COW write path (including growth) is now under the inner mutex
R-24	Post-growth slot assignment a->caps - 1 double-claimed by concurrent threads (P0)	Resolved by R-22 fix — mutex held around growth + assignment prevents double-claim
R-25	Shadow-only iteration is prose discipline, not enforced API (P0)	xzarray_freeze() builds per-kvmgr slot_key_t[] under write lock — serialization reads this array, not kvtab (Part 3.8)
R-26	malloc(0) for WAL_CHECKPOINT entries → NULL dereference (P0)	Added payload_len > 0 guard before malloc; apply_wal_entry() receives NULL for zero-payload entries
R-27	Fallback to older snapshot with trimmed WAL → silent stale boot (P0)	Added WAL-existence verification before accepting older snapshot fallback
R-28	hdr.term > ha->term + 1 permanently disconnects multi-promotion standby (P0)	Replaced with term-accept-and-update: primary's term is authoritative, standby updates ha->term to match
R-29	_Atomic ctrl_role_t on enum type — C11 semantics implementation-defined (P0)	Changed to _Atomic int32_t with cast to ctrl_role_t at access point
R-30	handle_jobsubmit() ignores xce_run_hooks() return value — false ACK on WAL failure (P0)	Added return value check: -EIO aborts submit, -ETIMEDOUT accepts with degraded durability warning
R-31	malloc(hdr.payload_len) NULL check missing in standby receiver (P0)	Added OOM check; break connection on failure
R-32	No heartbeat during persist_recover() on new primary → split-brain (P1)	Added step 5.5: start heartbeat before persist_recover() with new term, defer cluster_start()
R-33	Catch-up from dead primary specified in promotion step 3 (P1)	Removed catch-up-from-dead-primary — async gaps acknowledged as lost, self-heal via worker reconnect
R-34	No total_wal_bytes field for inline checkpoint trigger (P1)	Added total_wal_bytes counter to persist_t struct
F-1	_Atomic uint32_t cast of unsigned int :1 bitfield — C11 prohibits atomic operations on bitfields (P0)	Changed _Atomic uint32_t to _Atomic _Bool for inuse; added xzarray_private_t.free_idx cache for O(1) slot allocation
F-2/F-5	ha->term missing from ha_state_t — no authoritative term field in struct (P0)	Added _Atomic uint64_t term to ha_state_t; all accesses use atomic_load/store_explicit with acquire/release
F-3	Plain c->ha.role == reads against _Atomic int32_t in pseudocode (P0)	All pseudocode role comparisons updated to use (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire)
F-4	Unbounded term-accept — standby with corrupted term could overflow (P1)	Added MAX_TERM_JUMP (10000) ceiling; terms exceeding ceiling cause disconnect
F-6	Heartbeat starts before cluster_start() — subsystems not initialized (P1)	Step 5.5 clarified: heartbeat starts before cluster_start() with new term; messages contain only term and role until step 8
F-7	persist_t.key_snapshot lifecycle broken — xzarray_freeze() has no access to persist_t, wrong type (P0)	Redesigned as per-kvmgr slot_key_t[] built during xzarray_freeze() under write lock, freed after serialization
F-8	total_wal_bytes decrement on WAL trim not specified (P1)	Added trim accounting: total_wal_bytes decremented by fstat(fd).st_size on segment deletion
F-9	Shadow use-after-free during abnormal shutdown after 5-second deadline (P1)	Added pthread_join() of checkpoint thread before freeing cluster state on shutdown
F-10	promotion_retry_count data race between watchdog and vcpu threads (P1)	Changed to _Atomic int with relaxed atomic_load/store
F-11	a->pos = priv->shadow_caps truncates size_t to int (P1)	Changed loop variable from int idx to size_t idx in vstore COW scan
F-12	Async WAL write failure silently swallowed — no logging (P1)	Added WARN() log on persist_wal_replicated_entry() failure
F-13	Step 5.5 references persist_recover() but promotion doesn't call it (P1)	Changed wording to reference cluster_start() instead
F-14	apply_wal_entry() return value ignored — corrupted entry silently accepted (P0)	Added return value check; break replication stream on failure
F-15	replicator_t.ack_lock is mylock_t but pthread_cond_timedwait requires pthread_mutex_t* (P0)	Changed to pthread_mutex_t — same rationale as checkpoint_mutex (R-6)
F-16	ha->in_catch_up referenced in ACK gate but absent from ha_state_t (P0)	Added in_catch_up field to ha_state_t in Part 5.3 and Appendix A.4 with lifecycle spec
F-17	SIGUSR1 handler calls non-async-signal-safe ha_demote_to_standby() (P0)	SIGUSR1 sets atomic_int failover_requested only; watchdog thread executes actual demotion
F-18	vfreeze_fn_t signature incompatible with slot_key_t capture (P1)	Moved key capture to caller-side kvmgr_capture_keys() after vfreeze() returns, under write lock
F-19	P0-4 step says traversalLimitInWords proportional to file_size / 8, contradicting R-1 (P1)	Updated P0-4 to fixed upper bound MAX_SNAPSHOT_BYTES / 8 words
F-20	1 << attempt signed overflow + shared rng_seed data race in backoff_jitter (P1)	Saturating shift at 30; __thread per-thread seed with pthread_self init
A5-001	Checkpoint timer described as "1-second delaymgr timer" but WAL fsync and heartbeat require sub-second granularity (P1)	Clarified P0-3: checkpoint timer uses delaymgr (1s poll is adequate); WAL fsync and heartbeat use dedicated pthreads for sub-second precision
A5-002	Capnp schema missing opcodes added in P1 milestones (P1)	Added heartbeat @26, steppingDown @27, replSnapshot @28, replCatchUpDone @29 to Part 5.10 schema
A5-003	WAL CRC chain WAL_CRC_NONE at segment start is permissive — could mask corruption (P1)	Clarified: WAL_CRC_NONE at segment boundaries (writer-side rotation, after fsync succeeded) is valid and required; WAL_CRC_NONE encountered mid-segment during catch-up replay is a protocol error
A5-004	Hook list mutation during demotion — xce_remove_hook() could race with concurrent iteration (P1)	Added lock-free/RCU-style removal spec: xce_remove_hook() sets disabled flag, defers memory reclamation
A5-005	xce_connect_cluster() returns int fd but TLS-first architecture requires SSL* (P1)	Changed return type to SSL*; added xce_disconnect_cluster(SSL*); updated xce_reconnect to SSL**
A5-006	GC use-after-free: checkpoint traverses deleted entities without protection (P1)	Added GC use-after-free mitigation section: per-operation READ lock hold for P0; epoch-based reclamation as P3 optimization
A5-007	xzarray_freeze() OOM rollback of a->pos is racy if readers observed new value (P2)	Noted as P2 — freeze under write lock makes rollback window single-threaded
A5-008	ha->in_catch_up plain int — data race between iomgr and watchdog threads (P1)	Changed to _Atomic int in_catch_up with documented access patterns and memory ordering
A5-009	promotion_failed should be _Atomic int for watchdog/vcpu access (P2)	Already _Atomic; existing promotion_retry_count atomic is sufficient
A5-010	Missing OP_steppingDown handler in P1-2 milestone (P1)	Added P1-2 handler spec: term validation, sender verification, priority-based promotion
A5-011	P0/P1 milestone dependency ordering unspecified — implementer could build P1 before P0 (P2)	Dependencies are implied by sequential section numbering; explicit DAG not in scope
A5-012	primary_ep pointer lifecycle on standby — no documented safe access pattern (P1)	Added lifecycle spec: iomgr-owned borrowed reference, NULL-before-free under ha->lock, all access must check NULL
A5-013	repl_remove_standby() leaves stale slot data before nstandbys decrement (P1)	Added cleanup requirement: zero slot fields before decrementing nstandbys
A5-014	Async hook error handling unspecified — what happens when AFTER_EVENT hook returns error for async-replicated events (P1)	Added error convention spec: same as sync hooks (-EIO aborts, -ETIMEDOUT degraded durability)
A5-015	Concurrent snapshot catch-up memory — MAX_STANDBYS × MAX_SNAPSHOT_BYTES aggregate (P2)	Added aggregate memory budget analysis: primary holds negligible per-chunk memory; standby reassembly buffer bounded by single MAX_SNAPSHOT_BYTES
A5-016	ha_check_heartbeat() uses plain ha->role read without atomic_load_explicit (P1)	Fixed to use (ctrl_role_t)atomic_load_explicit(&ha->role, memory_order_acquire) — consistent with F-3 fix
A5-017	Appendix A.4 in_catch_up field typed as plain int instead of _Atomic int (P1)	Fixed to _Atomic int to match Part 5.3 definition and documented access pattern
A5-018	Stale "COW mode growth" snippet (pre-R-22 intermediate design) present alongside final xzarray_store() listing — contradicts the mutex-first approach	Removed the superseded snippet; added a forward-reference note pointing to the definitive R-22 implementation below
A5-019	Plain reads/writes to _Atomic uint64_t ha->applied_lsn in standby receiver pseudocode (Part 6.4) — contradicts documented require of acquire/release ordering	Replaced plain accesses with atomic_load_explicit(acquire) into local applied variable; replaced write with atomic_store_explicit(release)
A5-020	Plain !ha->in_catch_up in standby receiver pseudocode (Part 6.4) — contradicts documented _Atomic int field with required acquire ordering	Changed to !atomic_load_explicit(&ha->in_catch_up, memory_order_acquire)
A5-021	ha_check_heartbeat() reads ha->primary_ep and ha->last_heartbeat without ha->lock — contradicts lifecycle comment requiring mutex; ha_state_t missing lock field entirely	Added pthread_mutex_t lock field to ha_state_t (Part 5.3 and Appendix A.4); updated ha_check_heartbeat() to snapshot primary_ep and last_heartbeat under ha->lock before use
A5-022	Thaw function reads _Atomic _Bool inuse via plain !x.inuse and x.inuse — C11 plain access gives seq_cst ordering inconsistently with relaxed ordering used by surrounding explicit stores	Changed all three plain inuse reads in xzarray_thaw() to atomic_load_explicit(memory_order_relaxed) matching the explicit stores in the same function
A6-001	Part 5.5 code sketch registers heartbeat check via delay_task(&c->delaymgr, ...) — directly contradicts the same section's requirement for a dedicated pthread	Replaced delaymgr sketch with a reference to the dedicated-pthread requirement; noted delaymgr's 2-second tick granularity (precision=2 in cluster.c) as the reason
A6-002	Remaining plain ha->applied_lsn read in Part 6.4 CRC check block (if (ha->applied_lsn == 0)) missed in A5-019 pass	Changed to atomic_load_explicit(&ha->applied_lsn, memory_order_acquire) == 0
A6-003	Part 9.4 redirect sends primary_host/primary_port without nil-check — on cold start before any heartbeat, standby sends garbage redirect (empty host, port 0)	Added guard: if primary_host[0] == '\0' || primary_port == 0, reply with E_XCE_NOMASTER instead of redirect
A6-004	Part 5.4 election probe peer-cert validation describes only application-layer CN check — missing SSL_VERIFY_PEER TLS-layer requirement that Part P2.5-2 already mandates for xce_connect_cluster()	Added explicit SSL_CTX_set_verify(SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT) requirement to Part 5.4; cross-referenced P2.5-2
A6-005	persist_t has last_crc for local WAL chain but no last_replicated_crc for replication chain — WAL_CHECKPOINT entries break the standby's chained-CRC validation unless the primary skips them from the replicated chain	Added last_replicated_crc field to persist_t (Appendix A.1) with documentation of when it is updated vs last_crc
A6-006	slot_key_t.key is const char* (borrowed pointer into kvtab) — kvtab can rehash and move strings after the write lock is released during lock-free serialization, causing use-after-free	Changed slot_key_t.key to char key[XCE_MAXNAME_LEN] (owned deep copy populated by strndup in kvmgr_capture_keys); updated the serialization snapshot code and comment
A6-007	P0-10 description has no prerequisite dependency on P0-9 — implementer can attempt to wire persist_init into cluster.c before Cap'n Proto schemas are generated	Added Requires: P0-9 to P0-10 description
A6-008	P0-0b description did not call out that intf_varray_t and xzarray_private_t struct changes are breaking ABI — partial builds corrupt makeInstance() private section allocations	Added explicit breaking-ABI note to P0-0b; required converting positional initializers to designated style before adding new slots; required full ninja -C build clean rebuild
A6-009	Part 3.9 lock hierarchy described xzarray mylock_t as "not taken in vput" without distinguishing normal/COW/thaw modes — contradicted Part 3.5's COW-mode inner-mutex requirement (R-22)	Replaced single-sentence description with three-mode table: (a) normal — inner mutex skipped; (b) COW vstore — inner mutex required; (c) thaw — write lock sufficient
A6-010	Part 6.2 "constant reconciliation" note still referenced old MAX_HA_CANDIDATES = 16 and MAX_STANDBYS = MAX_HA_CANDIDATES - 1 derivation, contradicting Out of Scope reduction to 4/3	Updated to state both constants defined directly in config.h (MAX_HA_CANDIDATES = 4, MAX_STANDBYS = 3); removed derived formula
A6-012	kvmgr_capture_keys() called in Part 3.8 checkpoint flow but never defined anywhere in the document	Added full function signature and documentation to Appendix A.1
A6-013	Recovery step 3 required idgen_set_next() — a function that does not exist; idgen lives in Go API server (clib.go), not the controller	Removed idgen_set_next() requirement; documented that the controller never allocates jobids (assigned by API server before submission) and recovery only restores jobs at their original IDs from the WAL
A6-014	R-2 "Job ID allocator reset" based on incorrect assumption that the controller allocates jobids	Corrected: jobids are allocated by the Go API server's idgen; the controller is a passive consumer. Recovery single-threaded note added to step 3
A6-015	xce_remove_hook() does not exist in hook.c — required for hook unregistration on demotion	Replaced with xce_disable_hook() approach: set entry->fn = NULL in place; xce_run_hooks() skips NULL entries. No RCU/deferred-reclamation complexity; bounded memory (3 persist hooks max). Requires adding xce_disable_hook() to src/lib/hook.c
A6-016	Part 6.4 did not clarify that apply_wal_entry() on standby is effectively single-threaded	Added serialization note: all OP_replEntry messages arrive over a single endpoint; iomgr/vtask endpoint_block_tasks() serializes dispatch — xkvmgr_add() calls in apply_wal_entry() are never concurrent
A6-017	delaymgr uses sleep(precision) with precision=2 (integer seconds) — makes 1-second checkpoint poll (P0-3) impossible, and adds up to 2s jitter to heartbeat detection	Specified delaymgr enhancement: replace sleep() with usleep(), change precision field to precision_ms (default 200ms). Allows 1-second checkpoint poll (5 ticks × 200ms) without a dedicated pthread. P0-3 notes this as a prerequisite
A6-018	Part 10.8 listed NFS as a valid distribution mechanism for cluster.yaml, contradicting the no-shared-filesystem architecture	Removed NFS from distribution options; documented that XCE assumes no shared filesystem — cluster.yaml must be distributed via config management tooling (Ansible, Salt, deployment pipeline) on each node's local storage
A6-019	Product-lens review flagged "split-brain data loss not surfaced at occurrence" as a P1 omission	Reframed as an intentional CAP trade-off: XCE chooses AP (availability + partition tolerance) over CP (consistency). Real-time split-brain detection is impossible without quorum — implementing it is equivalent to Raft. The split_brain_risk_accepted: true key is the operator's explicit CAP position acknowledgment. Post-partition visibility is provided by ctrl_split_brain_detected counter and ERROR-level logs
A6-020	AFTER_EVENT hook return value used to detect persistence failures — breaks if any subsequent hook overwrites rc; requires transaction rollback to undo in-memory state on failure	Replaced with side-channel pattern: persistence hook always returns 0 (AFTER_EVENT failures tolerated per XCE hook design; pre-hook BEFORE_EVENT failures abort, post-hook AFTER_EVENT failures are logged). WAL result stored in persist_t.last_sync_rc; handle_jobsubmit() reads it directly after xce_run_hooks(AFTER_EVENT) returns. Field added to persist_t struct.
A6-021	sizeof(xzarray_item_t) change from unsigned int :1 to _Atomic _Bool flagged as backward compatibility concern	Dismissed: no released version of XCE exists; all components are rebuilt together. The struct layout can change freely. calloc() sizing in freeze and xzarray_ensure_items() automatically uses the new size after recompile.
A6-022	No recovery path for corrupt-all-snapshots fatal abort scenario — "restore from backup" is undefined	Added snap_keep_n config (default 2) to retain previous snapshots as last-known-good fallback. Recovery walks snapshots newest-to-oldest, falling back if checksum fails and WAL can bridge the gap. snap_keep_n added to persist_config_t and Cap'n Proto PersistConfig schema.
A6-023	rtti source location flagged as unknown — P0-0b appeared to lack an implementation path	Clarified: rtti is maintained by the XCE team (not a third-party dependency). Source is accessible via the project's flake.nix inputs. No external coordination required.
A6-024	Part 1.2 said restore_worker() "signature will need to be updated" — contradicted P0-12 which says "signature is unchanged"	Fixed Part 1.2: signature is unchanged; worker_from_capnp() adapter in P0-12 bridges Cap'n Proto to restore_worker(). Recovery calls the adapter, not restore_worker() directly
A6-025	P2.5 "parallel with P2" claim misleading — P2.5-7 (API server discovery) only has value when P2 replication is live	Clarified: P2.5 code can be developed in parallel with P2; P2.5-7 should be deployed together with P2 for full functionality. P2.5 alone provides worker/scheduler auto-reconnect but not replication continuity
A6-026	COW active array allocated at exact shadow_caps — any vstore() call immediately triggers realloc() which moves a->data, causing dangling pointers in concurrent vget() callers holding READ lock	Pre-grow active array to 1.5 × shadow_caps + delta at freeze time. [shadow_caps, cow_caps) range is immediately available for vstore() — no realloc needed in the common case. If full, vstore() returns -ENOMEM (WARN + client retry after COW closes)
A6-027	P0-8a conditional fallback had no merge-order gate — deferred cleanup path could be permanently skipped	Split into P0-8a (fallback with TODO markers, no P0-0a dependency) and P0-8b (deferred path, explicitly requires P0-0a merged and green before starting)
A6-028	P0-7 reply-after-hooks audit scoped only to handle_jobsubmit() — other handlers with AFTER_EVENT + xce_reply have the same ordering gap	Expanded P0-7 scope to full codebase audit: grep all xce_run_hooks(AFTER_EVENT + xce_reply pairs across all handler files; document each handler's reply ordering decision
A6-029	SIGUSR1 manual failover has no access control — any same-UID process can trigger failover	Documented: acceptable for initial implementation on administratively restricted management nodes. Future xce hostcontrol --failover CLI (SSH-authenticated) is the preferred interface. Added note about future --disable-sigusr1-failover startup flag
A7-001	Part 3.10 "ticks" framing for delay_task() was wrong — delay.c uses time_t arithmetic (seconds), not tick counts; tick description contradicted P0-3 which correctly says delay=1 second	Removed ticks framing; clarified delaymgr enhancement as nanosleep precision change only (delay_task parameters remain seconds throughout); corrected delay_task call to delay=1 second
A7-002	snap_keep_n added to persist_config_t C struct but absent from PersistConfig Cap'n Proto schema in Appendix A.5 — field could never be read from config	Added snapKeepN @6 :UInt32 = 2 to PersistConfig schema; updated P0-9 step to include the field
A7-003	COW vstore() growth path called xzarray_ensure_items() (realloc) despite pre-grow design intent — realloc moves a->data, dangling vget() pointers under READ lock	Changed growth path to return -ENOMEM instead of realloc; documented rationale; -ENOMEM is transient (COW < 100ms) — client retries
A7-004	xce_disable_hook() and xce_run_hooks() NULL guard had no implementation phase step — ha_demote_to_standby() calls a function not yet in hook.c	Added P1-2a step: add xce_disable_hook() to src/lib/hook.c and add if (!it->fn) continue; guard to xce_run_hooks(); marked as prerequisite for P1-3a
A7-005	Part 3.8 said checkpoint_lsn = max LSN across kvmgrs — should be minimum; using max trims WAL entries needed to replay the earliest-frozen kvmgr	Changed to minimum LSN; documented why minimum is correct for cross-kvmgr WAL bridging
A7-006	ha_state_t.promotion_failed was plain int while promotion_retry_count was _Atomic int — mismatch, data race on watchdog vs vcpu reads	Changed to _Atomic int promotion_failed in both Part 5.3 and Appendix A.4 struct definitions
A7-007	P2.5 intro still said MAX_STANDBYS = MAX_HA_CANDIDATES - 1 (defined in repl.h, currently 15) — stale text contradicting A6-010	Fixed to MAX_STANDBYS = 3 standalone in include/config.h; removed derived-formula reference
A7-008	Part 7.4 Degraded Mode contained full implementation spec for repl_degraded_timeout_s despite Out of Scope removing it — contradiction caused scope confusion	Added explicit callout box at top of degraded-mode timeout paragraph marking it out of scope; replaced full spec with concept summary
A7-009	Part 7.4 step 4 said self-healing was via "WAL replay" — misleading after a checkpoint trims the degraded WAL, where full snapshot transfer is required	Added "or full snapshot transfer" to the self-healing description
A7-010	Part 3.11 snapshot format diagram missing version field and headerChecksum field — reviewers and implementers could miss these required fields	Added version: UInt32 (=1) and headerChecksum: CRC32(header fields) to the SnapshotHeader diagram
A7-012	Part 10.8 contained full configVersion CRC hash implementation spec despite Out of Scope deferring it to P3	Added deferred callout; replaced full spec with concept summary; noted configVersion @23 is reserved
A7-013	snap_keep_n fallback description implied protection against completed-checkpoint corruption — actually only protects against crash-during-checkpoint	Clarified: fallback works only when WAL bridging the old→new snapshot gap still exists (crash-during-checkpoint scenario); aborts with fatal error when WAL is already trimmed
A7-014	P0-9 phase step description did not mention snap_keep_n/snapKeepN schema addition	Added snapKeepN @6 :UInt32 = 2 explicitly to P0-9 step description
A7-015	last_sync_rc was a shared int on persist_t — concurrent vcpu threads race on reads/writes, causing false ACKs on WAL failures	Removed from persist_t; replaced with stack-allocated int sync_rc in handle_jobsubmit() passed as extra va_list argument to xce_run_hooks() and read by persist_job_hook_fun() — per-request, no sharing across threads
A7-016	xce_disable_hook() could grow hook list indefinitely (new NULL slot per promote/demote cycle) and fn pointer reads were non-atomic — use-after-NULL-write race	Made hook_item_t.fn _Atomic hook_fn_t; xce_disable_hook() stores NULL with release; xce_run_hooks() loads into local with acquire then checks+calls local; xce_add_hook() reuses existing NULL slots before appending — bounds list to max-ever-registered slots
A7-017	JOB_STAT_TRANSIENT had no TTL or auto-resolution path — permanently stuck scheduling slots if worker never reconnects	Added transient_job_timeout_s config (default 300s) to persist_config_t and PersistConfig schema (@7); after timeout, job → JOB_STAT_EXIT with reason "worker did not reconnect"; worker → WORKER_UNAVAILABLE; increments ctrl_transient_jobs_stuck
A7-018	Array job batch sync mechanism said "accumulate in a single hook call" but AFTER_EVENT fires once per sub-job — no mechanism for cross-call accumulation	Specified heap-allocated persist_batch_ctx_t* passed via same va_list slot as sync_rc for array submits; accumulates WAL entries across calls; flushes every 100 sub-jobs; bounded by per-request allocation (no shared state)
A7-019	ha_init() mutex initialization timing relative to heartbeat thread and promotion path was unspecified	Added explicit ordering note: ha_init() must run before am_i_primary(); mutex is initialized once at startup and never re-initialized on re-promote cycles (re-init with waiters = UB)
A7-020	reconciliation_progress denominator (workers_at_old_primary) was undefined — implementers could not compute it	Specified as: count of workers in promoted standby's kvmgr with status NOT including XCE_WORKER_UNAVAIL or XCE_WORKER_HIDDEN at the moment of promotion — derivable from replicated state, no data from dead primary needed
A7-021	delaymgr precision enhancement had no dedicated phase step — was only noted as a prerequisite inline in P0-3	Added P0-2b step: rename precision → precision_ms, replace sleep() with nanosleep(), audit all delaymgr_init() callers, update cluster.c:604 to precision_ms=200
A8-001	"Why this works" paragraph in Part 8.3 still referenced removed c->persist.last_sync_rc field	Updated to reference per-request sync_rc stack variable and document thread-private design
A8-002	persist_batch_ctx_t was referenced in Part 8.3 and A7-018 but never defined — no struct or Appendix entry	Added full typedef struct persist_batch_ctx_t to Appendix A.1 with all fields documented
A8-003	va_list slot ambiguity: &sync_rc (int*) and persist_batch_ctx_t* described as occupying "same slot" — C UB when va_arg reads different types at same position	Replaced with two distinct va_list arguments (always present): &sync_rc first, batch_ctx (NULL for scalar, non-NULL for array) second; hook always reads both with typed va_arg calls; call site updated with persist_batch_ctx_alloc() example
A8-004	P1-2a listed after P1-2 in phase table but is a prerequisite of P1-2 — table order inverts the dependency	Moved P1-2a before P1-2; added Requires: P1-2a note to P1-2; added _Atomic hook_fn_t and atomic_store/load_explicit requirements to P1-2a
A8-005	P0-9 schema step missing transientJobTimeoutS @7 and transientSince field for Job schema	Added both to P0-9; transientSince is required for the transient_job_timeout_s GC check in job_should_retain()
A8-006	transient_job_timeout_s auto-termination had no implementation path — job_should_retain() returns true for JOB_STAT_TRANSIENT (not terminal per job_is_terminal())	Added transient-job TTL check to job_should_retain() before the job_is_terminal() guard; auto-terminates to JOB_STAT_EXIT with WARN log + ctrl_transient_jobs_stuck increment
A8-007	ctrl_transient_jobs_stuck metric named in config comment and A7-017 but absent from P3-7 metric inventory	Added counter definition to P3-7 with description and WARN log requirement
A8-008	P0-3 had no explicit Requires: P0-2b — the dependency was only in P0-2b's description and prose	Added Requires: P0-2b to P0-3 step row

---

Appendix: Data Structures & Signatures

A.1 Persistence Module

/* persist.h — see Appendix A.10 for system-level tunable constants in include/config.h */

#include "config.h"   /* WAL_FORMAT_VERSION, MAX_SNAPSHOT_BYTES, MAX_WAL_PAYLOAD_BYTES,
                         WAL_ENTRY_TYPE_MAX, WAL_HDR_CRC_SIZE, DEFAULT_WAL_SYNC_MS, etc. */

typedef struct {
    int             wal_fd;         /* Current WAL segment fd */
    uint64_t        current_lsn;    /* Next LSN to assign */
    uint64_t        last_crc;       /* Previous record's CRC (chained) */
    uint64_t        term;           /* Current election term */
    char            seg_path[XCE_MAXPATH_LEN];  /* Current segment path */
    uint32_t        seg_offset;     /* Write offset in current segment */
    uint64_t        checkpoint_lsn; /* LSN of last completed checkpoint (for WAL trim guard) */
    persist_config_t config;
    cluster_t*      cluster;        /* Back-reference */
    mylock_t        lock;           /* Protects WAL write + LSN */
    atomic_int      checkpoint_in_progress; /* Non-zero while checkpoint is active */
    atomic_int      checkpoint_requested;   /* Set by inline WAL size trigger (Part 3.11);
                                               cleared by checkpoint timer after completing */
    pthread_cond_t  checkpoint_done;        /* Signaled when checkpoint completes or is canceled */
    pthread_mutex_t checkpoint_mutex;       /* Dedicated mutex paired with checkpoint_done.
                                               MUST NOT be persist.lock (which is mylock_t) —
                                               pthread_cond_timedwait requires pthread_mutex_t*.
                                               See Part 3.8 for rationale. */
    pthread_t       snap_thread;            /* ID of the active serialization pthread, or 0 if
                                               none is running. Set by persist_checkpoint_timer_fn()
                                               when it spawns the serialization pthread; read by
                                               cluster_fini() for pthread_join() on the deadline-
                                               exceeded shutdown path. Zero-initialized (memset
                                               in persist_init()). cluster_fini() MUST check
                                               snap_thread != 0 before calling pthread_join() —
                                               pthread_join(0, NULL) is undefined behavior. */
    uint64_t        total_wal_bytes;        /* Cumulative WAL size across all segments.
                                               Updated on each write, adjusted on segment
                                               rotation and WAL trimming. Used by the inline
                                               checkpoint trigger (Part 3.11) to decide when
                                               to checkpoint based on snap_wal_size_mb. */
    uint32_t        last_replicated_crc;   /* CRC of the last WAL entry that was sent to standbys.
                                               Distinct from last_crc (which includes WAL_CHECKPOINT
                                               entries that are local-only and never replicated).
                                               Used as prev_crc for the next replicated entry so
                                               that standbys' chained-CRC validation works correctly
                                               across checkpoint boundaries. Updated only when a
                                               non-WAL_CHECKPOINT entry is broadcast to standbys.
                                               See Part 6.4 WAL_CHECKPOINT chain gap handling.
                                               LOCK: protected by persist.lock. All updates to
                                               last_replicated_crc must be made while holding
                                               persist.lock (same lock as last_crc). The WAL fsync
                                               thread (which writes WAL_CHECKPOINT and updates
                                               last_crc) does NOT update last_replicated_crc —
                                               the two fields are always updated on different code
                                               paths, preventing the race where a checkpoint write
                                               interleaves with a replication CRC read. */
    /* NOTE: last_sync_rc was removed (A7-015). WAL write results are now communicated
     * via a per-request stack variable (&sync_rc) passed as an extra va_list argument
     * to xce_run_hooks(AFTER_EVENT). A shared persist_t field races across concurrent
     * vcpu threads processing simultaneous job submits from different API servers. */
} persist_t;

int  persist_init(persist_t* p, persist_config_t* config, cluster_t* c);
int  persist_start(persist_t* p);
void persist_stop(persist_t* p);
void persist_fini(persist_t* p);

/* Flush the current WAL segment to disk (fsync).
 * Called during promotion (Part 5.6 step 3) to ensure all replicated
 * entries are durable before the standby takes over as primary. */
int  persist_flush(persist_t* p);

/* Async WAL write — returns immediately after writing + sending to standbys */
int  persist_wal_async(persist_t* p, uint32_t type, void* payload, uint32_t len);

/* Sync WAL write — writes+fsyncs locally, then waits for at least one standby ACK.
 * Return values distinguish the two failure modes:
 *   0          — success (written + fsynced + ACKed)
 *  -EIO        — local write or fsync failed (permanent; do not send to client)
 *  -ETIMEDOUT  — wrote locally but no standby ACKed within sync_timeout_ms
 *                (transient; client may retry; entry is locally durable)
 * Callers must treat -EIO and -ETIMEDOUT differently: -EIO aborts the operation,
 * -ETIMEDOUT can be surfaced to the client as a "degraded durability" warning.
 *
 * IMPORTANT: -ETIMEDOUT is a durability warning, not a consistency error. When
 * -ETIMEDOUT is returned, the in-memory mutation (e.g., job_submit()) has already
 * completed — the job ID is assigned, the kvmgr entry exists, and the local WAL
 * entry is fsynced to disk. There is NO rollback of the in-memory state. The
 * persistence hook call from AFTER_EVENT cannot undo the business logic that
 * preceded it. Callers that receive -ETIMEDOUT should still reply to the client
 * with the result (e.g., the jobid), and log the degraded durability event.
 * The entry will be replicated to standbys when they reconnect (see Part 7.4
 * degraded mode self-healing). */
int  persist_wal_sync_append(persist_t* p, uint32_t type, void* payload, uint32_t len);

/* Write a replicated WAL entry — uses the primary's LSN and CRC.
 * Called by the standby replication receiver (Part 6.4) to write entries
 * received from the primary. Unlike persist_wal_sync_append/async, this
 * function does NOT assign a new local LSN. Instead it:
 *   1. Writes hdr directly (primary-assigned lsn, crc, prev_crc) to the WAL
 *   2. Updates p->current_lsn = hdr.lsn so the next local entry gets hdr.lsn + 1
 *   3. Updates p->last_crc = hdr.crc so the chained CRC chain remains valid
 *   4. Updates p->term = hdr.term to stay in sync with the primary's term
 *   5. fsyncs only if hdr.type == WAL_JOB_SUBMIT (sync replication)
 * Must NOT be called on the primary — use persist_wal_async or persist_wal_sync_append.
 * Returns 0 on success, -errno on I/O failure.
 * IMPORTANT: After calling this, the caller must NOT advance its own LSN
 * counter — p->current_lsn is already set to hdr.lsn. */
int  persist_wal_replicated_entry(persist_t* p, const wal_entry_hdr_t* hdr, const void* payload);

/* Checkpoint — dual zarray freeze + serialize + thaw + GC */
int  persist_checkpoint(persist_t* p, cluster_t* c);

/* Recovery — load latest snapshot + replay WAL */
int  persist_recover(persist_t* p, cluster_t* c);

/* Hook functions — registered in cluster_init() */
int  persist_job_hook_fun(int type, int situation, int detail, void* data, va_list ap);
int  persist_queue_hook_fun(int type, int situation, int detail, void* data, va_list ap);
int  persist_worker_hook_fun(int type, int situation, int detail, void* data, va_list ap);

/* Apply a single WAL entry to in-memory state — used by both persist_recover()
 * and the standby replication receiver. Persistence hooks are registered only
 * on the primary path (after persist_recover() completes in cluster_init(), or
 * in ha_promote_to_primary() step 5 on promotion), so apply_wal_entry() does
 * not fire hooks — the hook system is empty during recovery and on standbys.
 *
 * Semantics:
 *   WAL_JOB_SUBMIT  — calls restore_job(c, jobid, payload) to insert the job at its
 *                     original jobid without allocating a new one. Do NOT call
 *                     job_submit() here — job_submit() assigns a new jobid, which
 *                     breaks replay by creating duplicate or wrong-ID jobs.
 *                     If the jobid already exists, restore_job() returns -EEXIST;
 *                     apply_wal_entry() logs a WARN and skips the entry (non-fatal).
 *   WAL_JOB_START   — updates job stat, start_time, exec_host in-place
 *   WAL_JOB_FINISH  — updates job stat, end_time, exit_code in-place
 *   WAL_JOB_FAIL    — updates job stat, end_time, fail_reason in-place
 *   WAL_JOB_STOP    — updates job stat to JOB_STAT_USUSP in-place
 *   WAL_JOB_RESUME  — updates job stat from USUSP back to PEND in-place
 *   WAL_JOB_UPDATE  — merges partial Job payload into existing job object
 *   WAL_JOB_CKPT    — updates job ckpt_time in-place
 *   WAL_JOB_RUSAGE  — updates job rusage fields in-place
 *   WAL_JOB_PREASON — updates job preason field in-place
 *   WAL_QUEUE_CTRL  — creates or updates queue from QueueConf payload.
 *                     Idempotent: if the queue already exists (matched by name
 *                     or key), update its config in-place; otherwise create it.
 *                     During WAL replay or standby replication, duplicate
 *                     WAL_QUEUE_CTRL entries for the same queue must not create
 *                     duplicates — the create path checks for existing queues first.
 *   WAL_WORKER_CTRL — creates or updates worker from Worker payload.
 *                     Idempotent: if the worker already exists (matched by hostname
 *                     or endpoint), update its state in-place; otherwise create it.
 *                     Same duplicate-safety guarantee as WAL_QUEUE_CTRL.
 *   WAL_CHECKPOINT  — no-op (marker only, no payload)
 *
 * Returns 0 on success, -errno on deserialization or mutation failure.
 * Unknown entry types are WARN-logged and skipped (return 0) to allow forward
 * compatibility with future WAL types added by newer controller versions.
 * WARNING: Skipping unknown entries during recovery silently loses the
 * state changes those entries represent — see recovery forward-compat note. */
int apply_wal_entry(cluster_t* c, const wal_entry_hdr_t* hdr, const void* payload);

/* Array job batch accumulation context — allocated on handle_jobsubmit() stack for
 * array submits (submit->array_size > 1), freed on return from handle_jobsubmit().
 * Passed via va_list to persist_job_hook_fun() instead of int* sync_rc:
 *   - single job: va_list carries (jlist_ptr, &sync_rc)
 *   - array job:  va_list carries (jlist_ptr, batch_ctx_ptr)
 * The hook distinguishes the two cases by checking detail == HOOK_DETAIL_JOB_NEW
 * and submit->array_size — or by inspecting the cluster_t's in-flight array state.
 *
 * The batch context accumulates WAL entries for N sub-jobs (default batch_size=100),
 * then calls repl_broadcast() once and waits for one ACK at the batch's highest LSN.
 * sync_rc is embedded in the context so the hook can report the final WAL result. */
typedef struct {
    int      sync_rc;         /* Final WAL result — 0, -EIO, -ETIMEDOUT, or -EAGAIN.
                                * -EAGAIN: transient COW-full condition (xzarray_store returned
                                * -ENOMEM during COW window). Unlike -EIO, the job must NOT be
                                * marked terminal via job_mark_wal_error_exit() — the in-memory
                                * state may or may not have been inserted depending on where in
                                * the submit path the -ENOMEM occurred. handle_jobsubmit() must
                                * reply with a retriable error code (EAGAIN/EBUSY) so the client
                                * retries after the COW window closes (< 100ms). */
    uint64_t batch_high_lsn;  /* Highest LSN in the current batch (for repl_wait_ack) */
    uint32_t sub_jobs_in_batch; /* Sub-jobs accumulated in the current batch */
    uint32_t batch_size;      /* Flush threshold (default 100) */
    uint32_t total_sub_jobs;  /* Total sub-jobs in the array (submit->array_size) */
    uint32_t sub_jobs_done;   /* Sub-jobs processed so far (hook call count) */
} persist_batch_ctx_t;

/* Key→index mapping used during checkpoint serialization (Part 3.8).
 * Lives in kvmgr.h — used by both kvmgr.c (populate) and persist.c (consume).
 * key[] holds an owned deep copy populated by kvmgr_capture_keys() via strndup,
 * safe to read after the kvmgr write lock is released. */
typedef struct {
    int    index;                   /* shadow slot index */
    char   key[XCE_MAXNAME_LEN];    /* owned copy — not a borrowed pointer into kvtab */
    size_t key_len;                 /* length including null terminator */
} slot_key_t;

/* Capture key→index mappings from a kvmgr's kvtab into a flat slot_key_t[] array.
 * Called during checkpoint freeze (Part 3.8), AFTER vfreeze() returns, while the
 * kvmgr write lock is still held.  Allocates and populates keys_out[0..n_keys_out-1]
 * with one entry per live entity.  Each entry's key field is a deep-copied string
 * (strndup into slot_key_t.key[XCE_MAXNAME_LEN]) — safe to read after the write
 * lock is released during the lock-free serialization phase.
 * Caller must free(keys) after serialization completes.
 *
 * Returns 0 on success, -ENOMEM if allocation fails (in which case *keys_out is NULL
 * and *n_keys_out is 0 — caller must abort the checkpoint and call persist_checkpoint_abort()).
 *
 * NOTE: Supports both KVMGR_KEY_INT and KVMGR_KEY_STRING kvmgrs.  For integer-keyed
 * kvmgrs, the key is formatted as a decimal string before copying.
 * Caller must hold the kvmgr write lock for the entire duration of this call. */
int kvmgr_capture_keys(meta_obj_t* kvmgr, slot_key_t** keys_out, size_t* n_keys_out);

A.2 zarray COW Extensions

/* zarray.h — additions for COW snapshot support */

/* NOTE on struct layout: Part 3.4 defines xzarray_private_t (the RTTI private
 * section) as the home for shadow and shadow_caps fields, accessed via
 * RTTI_OBJ_TO_PRIV(). The struct definition below shows these fields as members
 * of xzarray_t for readability, but in the actual implementation they live in
 * the private section and are accessed as priv->shadow and priv->shadow_caps.
 * The RTTI type system allocates the private section alongside each instance.
 * This is the same pattern used by xzarray_private_t.lock (mylock_t). */

typedef struct {
    int             count;
    size_t          caps;
    size_t          delta;
    size_t          pos;     /* next scan position for vstore (was int — size_t to match shadow_caps) */
    xzarray_item_t* data;        /* Active array */
    destroy_fn_t    killer;
} xzarray_t;

/* Private section — allocated by makeInstance() alongside each xzarray_t instance.
 * Accessed via RTTI_OBJ_TO_PRIV(obj). See Part 3.4 for the design rationale. */
typedef struct {
    mylock_t          lock;
    xzarray_item_t*   shadow;      /* Frozen snapshot (NULL when not in COW) */
    size_t            shadow_caps; /* Shadow array capacity */
} xzarray_private_t;

/* Freeze: swap data → shadow, create new empty active.
   Caller must hold write lock. O(1) — pointer swap only.
   Returns 0 on success, -ENOMEM on allocation failure.
   Public API: vfreeze(var) — RTTI dispatches to xzarray_freeze(meta_obj_t*).
   Do not call xzarray_freeze(xzarray_t*) directly; it is a file-private
   static function that takes meta_obj_t*, not xzarray_t*. */
extern int vfreeze(var);

/* Thaw: merge shadow back into active, apply GC.
   should_retain(arg, item) returns true if entity should survive.
   Caller must hold write lock.
   Frees shadow when done. Returns 0 on success.
   Public API: vthaw(var, should_retain, arg) — RTTI dispatches to xzarray_thaw(meta_obj_t*, retain_fn_t, void*).
   Do not call xzarray_thaw(xzarray_t*, ...) directly; it is a file-private
   static function that takes meta_obj_t*, not xzarray_t*. */
/* retain_fn_t is defined in rtti/typeinfo.h (see Appendix A.9) */
extern int vthaw(var, retain_fn_t should_retain, void* arg);

A.3 Replication Module

/* repl.h — see Appendix A.10 for MAX_STANDBYS and other tunables in include/config.h */

#include "config.h"   /* MAX_STANDBYS, DEFAULT_SYNC_TIMEOUT_MS */

typedef struct {
    endpoint_t*  ep;           /* Full endpoint (SSL, FSM, outq) — uses iomgr/vtask pipeline */
    _Atomic uint64_t acked_lsn; /* C11 atomic — accessed locklessly by WAL trim (min across standbys)
                                 * and written by repl_on_ack() under ack_lock. Use
                                 * atomic_store_explicit(acquire)/atomic_load_explicit(release).
                                 * On LP64 platforms a plain uint64_t is atomically readable,
                                 * but using _Atomic documents the intent and ensures correctness
                                 * if compiled for ILP32. See Part 6.2 for full analysis. */
    int          lagging;
    char         host[XCE_MAXNAME_LEN];
    uint32_t     port;
    _Atomic int  in_catch_up;  /* Primary-side mirror of the standby's ha_state_t.in_catch_up.
                                 * Set to 1 by repl_add_standby() when applied_lsn < primary's
                                 * committed_lsn at join time (standby needs catch-up).
                                 * Cleared to 0 by repl_on_catchup_done() when OP_replCatchUpDone
                                 * is received from this standby.
                                 * Read by repl_wait_ack() to exclude catching-up standbys from
                                 * the "at least one ACK" predicate — standbys in catch-up never
                                 * send OP_replAck for live entries (per Part 6.4 ACK gate).
                                 * Use atomic_load/store_explicit with acquire/release. */
} standby_conn_t;

typedef struct {
    standby_conn_t  standbys[MAX_STANDBYS];
    int             nstandbys;
    mylock_t        lock;
    uint64_t        pending_lsn;
    pthread_cond_t  ack_cond;
    pthread_mutex_t ack_lock;     /* Must be pthread_mutex_t for pthread_cond_timedwait —
                                   * same rationale as checkpoint_mutex (Part 3.8). */
    uint64_t        last_disconnected_lsn[MAX_STANDBYS]; /* Per-slot: acked_lsn at disconnect
                                                            time, for one-interval trim delay */
    struct timespec last_disconnect_time[MAX_STANDBYS];   /* Per-slot: when the standby
                                                            disconnected, for trim delay */
    /* DISCONNECT CLEANUP REQUIREMENT: repl_remove_standby() MUST clear the
     * disconnected slot's fields (set ep=NULL, acked_lsn=0, lagging=0,
     * host[0]='\0', port=0) BEFORE decrementing nstandbys. This ensures
     * the WAL trim guard's min(acked_lsn) loop — which iterates
     * standbys[0..nstandbys-1] — never includes stale data from a
     * previously-disconnected slot. Similarly, repl_add_standby() MUST
     * fully initialize the new slot before incrementing nstandbys.
     * Both operations hold repl.lock, so the loop sees a consistent
     * snapshot. The last_disconnected_lsn[] and last_disconnect_time[]
     * arrays are NOT cleared on disconnect — they persist across
     * reconnections to support the one-interval trim delay.
     *
     * ACKED_LSN INITIALIZATION (CRITICAL): repl_add_standby() MUST initialize
     * the new slot's acked_lsn to persist->checkpoint_lsn (the last completed
     * checkpoint LSN), NOT to 0. Initializing to 0 would cause min(acked_lsn)
     * to equal 0, blocking ALL WAL trimming until the new standby fully catches
     * up from LSN 0 — potentially accumulating gigabytes of un-trimmable WAL.
     * checkpoint_lsn is the safe floor: WAL before checkpoint_lsn is covered
     * by the snapshot the standby will receive during catch-up, so it can be
     * trimmed immediately. WAL from checkpoint_lsn forward is what the standby
     * needs; min(acked_lsn) naturally preserves it until the standby ACKs those
     * entries. repl_add_standby() signature: accepts persist_t* for checkpoint_lsn
     * lookup, or the caller passes checkpoint_lsn directly as a uint64_t. */
} replicator_t;

/* Fan-out send to all standbys */
int  repl_broadcast(replicator_t* r, wal_entry_hdr_t* hdr, void* payload);

/* Wait for at least one standby to ACK the given LSN (sync mode).
 * Implementation MUST check the predicate before waiting to avoid a
 * signal-before-wait race: if repl_on_ack() signals ack_cond before
 * repl_wait_ack() parks on it, the signal is lost. Correct pattern:
 *   lock(ack_lock);
 *   while (r->acked_lsn < target_lsn && !timed_out)
 *       pthread_cond_timedwait(&r->ack_cond, &r->ack_lock, &deadline);
 *   unlock(ack_lock);
 * The while-loop predicate check must be inside the lock, before the first wait.
 *
 * CATCH-UP EXCLUSION: Standbys with in_catch_up != 0 (ha->in_catch_up set in the
 * ha_state_t of the standby's slot) do NOT send OP_replAck for live entries —
 * they gate ACKs until catch-up completes (Part 6.4). Therefore repl_wait_ack()
 * MUST exclude catching-up standbys from its "at least one ACK" predicate.
 * If the only connected standby is in catch-up, repl_wait_ack() falls back to
 * degraded mode immediately (does not wait for the full timeout_ms) and returns
 * REPL_DEGRADED. Callers treat REPL_DEGRADED the same as a timeout — the WAL
 * entry is durable on the primary's local WAL but not yet confirmed on any standby.
 * The per-standby in_catch_up flag is read from standby_conn_t.in_catch_up
 * (a copy of the remote standby's ha->in_catch_up, updated on each OP_replAck
 * that carries the standby's applied_lsn — if applied_lsn < primary's
 * committed_lsn at join time, in_catch_up is set; cleared on OP_replCatchUpDone). */
int  repl_wait_ack(replicator_t* r, uint64_t target_lsn, uint32_t timeout_ms);

/* Called when ACK received from a standby */
void repl_on_ack(replicator_t* r, uint64_t acked_lsn);

/* Add/remove standby connections (via endpoint/iomgr SSL pipeline) */
/* checkpoint_lsn: pass c->persist.checkpoint_lsn — the new slot's acked_lsn is
 * initialized to this value (not 0) to avoid blocking WAL trimming while the
 * standby catches up. See ACKED_LSN INITIALIZATION comment in replicator_t. */
int  repl_add_standby(replicator_t* r, endpoint_t* ep, const char* host, uint32_t port,
                      uint64_t checkpoint_lsn);
int  repl_remove_standby(replicator_t* r, endpoint_t* ep);

/* Check standby health and lag */
int  repl_health_check(replicator_t* r);

A.4 HA Module

/* ha.h */

typedef enum { CTRL_PRIMARY = 0, CTRL_STANDBY = 1 } ctrl_role_t;

typedef struct {
    _Atomic uint64_t term;   /* C11 atomic election term (fencing token). ALL accesses
                              * must use atomic_load/store_explicit with acquire/release.
                              * The term is the sole authoritative election term —
                              * persist_t.term is an alias populated from this field
                              * during promotion (Part 5.6 step 2) for convenience in
                              * the WAL write path. Standby receiver updates this field
                              * via atomic_store_explicit(release) — see Part 6.4. */
    _Atomic int32_t role;    /* Atomic — ALL reads must use atomic_load_explicit(acquire).
                              * Uses _Atomic int32_t (not _Atomic ctrl_role_t) because
                              * C11 atomic semantics on enum types are implementation-defined;
                              * some compilers silently ignore _Atomic on enums, falling back
                              * to non-atomic loads. The role-check window mitigation (Part 5.7)
                              * requires a proven lock-free atomic — _Atomic int32_t guarantees
                              * this on all supported platforms. Cast to ctrl_role_t at access:
                              *   (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire)
                              * Writes use atomic_store_explicit with release:
                              *   atomic_store_explicit(&c->ha.role, (int32_t)CTRL_PRIMARY, memory_order_release) */
    int             my_priority;      /* My index in candidates[] */
    replicator_t    repl;             /* Replicate to standbys (primary) */
    _Atomic uint64_t applied_lsn;     /* Last applied WAL entry (standby).
                                       * Must be _Atomic — written by iomgr thread, read by watchdog.
                                       * See Part 5.3 for full rationale. */
    endpoint_t*     primary_ep;       /* Connection to current primary via iomgr SSL pipeline (standby).
                                       * See Part 5.2 comment for full lifecycle spec. */
    char            primary_host[XCE_MAXNAME_LEN];
    uint32_t        primary_port;
    struct timespec last_heartbeat;   /* Last heartbeat from primary (CLOCK_MONOTONIC, standby) */
    _Atomic int     promotion_failed;      /* Non-zero if last promotion attempt failed — retry on next watchdog tick.
                                            * Must be _Atomic: written by ha_promote_to_primary() on the watchdog
                                            * thread, read by ha_check_heartbeat() on the same thread but also
                                            * visible to vcpu threads checking promotion status. Relaxed ordering. */
    _Atomic int     promotion_retry_count; /* Number of consecutive failed promotion attempts.
                                            * Must be _Atomic because it is incremented by the
                                            * watchdog thread and read by vcpu threads during
                                            * promotion status checks. Use atomic_load/store
                                            * with relaxed ordering. */
    _Atomic int     in_catch_up;           /* C11 atomic — non-zero while standby is processing historical
                                            * catch-up entries. See Part 5.3 for full lifecycle and
                                            * access pattern documentation. */
    int             hooks_registered;      /* Idempotency guard for persist hooks — prevents
                                            * double-registration on promote→demote→re-promote cycles.
                                            * ha_promote_to_primary() must check this flag before
                                            * calling xce_add_hook(). Reset to 0 on demote. */
    atomic_int      failover_requested;    /* Async-signal-safe flag set by SIGUSR1 handler.
                                            * Polled by HA watchdog thread which executes the
                                            * actual failover (Part 5.8). Never read/written
                                            * outside of SIGUSR1 handler and watchdog thread. */
    uint32_t        last_received_crc;     /* CRC of the last received WAL entry on the replication
                                            * stream — used for chained CRC validation (P0-C fix).
                                            * Mirrors persist_t.last_crc but on the standby side. */
    ha_config_t     config;           /* HA configuration (timeouts, intervals) */
    pthread_mutex_t lock;             /* Protects primary_ep and last_heartbeat against concurrent
                                       * iomgr NULL-then-free and watchdog reads. Initialized in
                                       * ha_init(). See Part 5.3 for full access protocol. */
} ha_state_t;

/* Check if this controller should be primary */
static int am_i_primary(cluster_t* c);

/* Heartbeat watchdog — called from the dedicated HA watchdog pthread (NOT delaymgr). See P1-3 */
static void ha_check_heartbeat(cluster_t* c);

/* Promote self from standby to primary */
int ha_promote_to_primary(cluster_t* c);

/* Demote self from primary to standby (admin-triggered failover) */
int ha_demote_to_standby(cluster_t* c);

A.5 Cap'n Proto Schema Additions

# Persistence configuration — embedded in ClusterConfig
# NOTE: Path fields (wal_path, snap_path) are REMOVED from this schema.
# Paths are derived from --workdir and -n CLI flags at controller startup:
#   effective_dir = workdir + "/" + name
#   wal_path      = effective_dir + "/data/wal"
#   snap_path     = effective_dir + "/data/snap"
struct PersistConfig $C.mapname("persist_config_t") {
    walSyncMs    @0 :UInt32 $C.mapname("wal_sync_ms") = 10;
    snapIntervalS@1 :UInt32 $C.mapname("snap_interval_s") = 60;
    snapWalSizeMb@2 :UInt32 $C.mapname("snap_wal_size_mb") = 64;
    finishedRetainS @3 :UInt32 $C.mapname("finished_retain_s") = 3600;
    workerRetainS  @4 :UInt32 $C.mapname("worker_retain_s") = 86400;
    syncTimeoutMs@5 :UInt32 $C.mapname("sync_timeout_ms") = 50;
    snapKeepN    @6 :UInt32 $C.mapname("snap_keep_n") = 2;   # last-known-good snapshot retention
    transientJobTimeoutS @7 :UInt32 $C.mapname("transient_job_timeout_s") = 300;  # 0 = no auto-termination
}

# WAL entry — used in Cap'n Proto schema for replication messages (OP_replEntry).
# On-disk WAL format uses a raw C struct (wal_entry_hdr_t, see Part 2.2), NOT Cap'n Proto.
# The WALEntry schema duplicates the C header fields for network transport only.
# Fields: term, lsn, timestamp, type, crc, prevCrc mirror wal_entry_hdr_t.
# Payload is the Cap'n Proto-serialized entity data (Job, QueueConf, Worker, etc.).
struct WALEntry $C.mapname("wal_entry_t") {
    term        @0 :UInt64;
    lsn         @1 :UInt64;
    timestamp   @2 :UInt64;
    type        @3 :UInt32;
    payload     @4 :Data;
    crc         @5 :UInt32;
    prevCrc     @6 :UInt32;
}

# Snapshot header — first record in a checkpoint file
struct SnapshotHeader $C.mapname("snapshot_hdr_t") {
    lsn             @0 :UInt64;
    timestamp       @1 :UInt64;
    term            @2 :UInt64;
    version         @7 :UInt32 = 1;   # WAL format version — bumped on incompatible schema changes
    nJobs           @3 :UInt32;
    nQueues         @4 :UInt32;
    nWorkers        @5 :UInt32;
    checksum        @6 :UInt32;  # CRC32 over entire serialized Snapshot body (excluding this header)
    headerChecksum  @8 :UInt32;  # CRC32 over header fields (lsn, timestamp, term, version, nJobs, nQueues, nWorkers) — protects security-critical term and lsn from tampering
}

# Full snapshot — checkpoint file contents
struct Snapshot $C.mapname("snapshot_t") {
    header      @0 :SnapshotHeader;
    jobs        @1 :List(Job);
    queues      @2 :List(QueueConf);
    workers     @3 :List(Worker);
    resources   @4 :List(Resource);
    configs     @5 :List(ConfigVar);
}

# Redirect response — standby tells client where the primary is
struct RedirectInfo $C.mapname("xce_redirect_t") {
    host @0 :Text $C.mapname("host");
    port @1 :UInt32 $C.mapname("port");
}

# In RequestReply union — add redirect variant:
# redirect @8 :RedirectInfo $C.mapname("redirect");

# New error code in common/
# #define E_XCE_REDIRECT (-100)  /* Not an error — redirect instruction */
# #define EXCE_ERR_REDIRECT_LOOP (-101)  /* Too many redirect hops (>= 5) */

# Entity terminal states for deferred cleanup
# Worker status uses bitmask flags on UInt32, not enum (see Part 4.4):
#   #define XCE_WORKER_UNAVAIL   (0x00010000)  — Worker daemon died / disconnected
#   #define XCE_WORKER_HIDDEN    (0x00200000)  — Admin explicitly removed entity
# Job status — add transient bitmask (NOTE: 0x200 is JOB_STAT_WAIT, use 0x400):
#   #define JOB_STAT_TRANSIENT   (0x400)       — Job state uncertain after failover
# Queue status — add hidden bitmask:
#   #define QUEUE_STAT_HIDDEN    (0x0100)      — Admin deleted queue

A.6 libxce Discovery Module

/* xce_cluster.h — libxce cluster discovery API */

typedef struct {
    char     host[XCE_MAXNAME_LEN];
    uint32_t port;
} xce_candidate_t;

typedef struct {
    char     ca_cert_path[PATH_MAX];  /* CA cert path for SSL_VERIFY_PEER (empty = skip) */
} xce_tls_config_t;

typedef struct {
    xce_candidate_t  candidates[MAX_HA_CANDIDATES];
    int              ncandidates;           /* Number of candidates configured */
    uint32_t         connect_timeout_ms;    /* Per-candidate connect timeout (default 2000) */
    uint32_t         retry_base_ms;         /* Backoff base (default 100) */
    uint32_t         retry_max_ms;          /* Backoff cap (default 5000) */
    int              max_retries;           /* Max reconnection attempts (default 10) */
    xce_tls_config_t tls;                  /* TLS options — CA cert path for SSL_VERIFY_PEER.
                                            * Populated from --ca-cert CLI flag (P2.5-5/P2.5-6). */
} xce_cluster_config_t;

/* Connect to the cluster — returns SSL* connected to primary, or NULL on failure.
 * The returned SSL* is owned by the caller and must be freed with
 * xce_disconnect_cluster() when no longer needed. */
SSL* xce_connect_cluster(xce_cluster_config_t* conf);

/* Disconnect and free an SSL connection returned by xce_connect_cluster */
void xce_disconnect_cluster(SSL* ssl);

/* Connect + join as specific role */
SSL* xce_join_cluster(xce_cluster_config_t* conf, xce_peerinfo_t* myinfo);

/* Handle disconnection — auto-reconnect with backoff + redirect following.
 * On success, *ssl is updated to the new connection. On failure, *ssl is NULL. */
int  xce_reconnect(SSL** ssl, xce_cluster_config_t* conf);

/* REMOVED: xce_load_cluster_config() — no YAML parsing in libxce.
 * xce_cluster_config_t is populated by caller from --peers CLI flag. */

A.7 New Files Summary

File	Purpose	Approximate Size
src/controller/persist_wal.c	WAL writer, segment management, fsync batching	~400 LOC
src/controller/persist_snap.c	Checkpoint, recovery, hook functions	~400 LOC
src/controller/include/persist.h	Persistence API and types	~80 LOC
src/controller/repl.c	Fan-out replication via endpoint/iomgr SSL pipeline	~500 LOC
src/controller/include/repl.h	Replication API and types	~60 LOC
src/controller/ha.c	LSF-style priority election, monitor thread	~300 LOC
src/controller/include/ha.h	HA API and types	~40 LOC
src/lib/cluster.c	Candidate selection, redirect following, backoff+jitter, reconnection	~400 LOC
src/lib/include/xce/cluster.h	Discovery API and types	~50 LOC
Total new code		~2230 LOC

A.8 Modified Files Summary

File	Change
src/lib/zarray.c + include/zarray.h	Merge zarray_t into xzarray_t (P0-0a); add shadow, shadow_caps fields; add xzarray_freeze(), xzarray_thaw(); modify vget, vput, iter_next for COW mode
src/controller/job.c	Remove dead NJOBS/jobid/load_jobs body (P0-0); move xce_reply() after AFTER_EVENT in handle_jobsubmit()
src/lib/kvmgr.c	Register persist_nop_freeze/persist_nop_thaw for the kvmgr hash table's own varray interface (kvmgr_type in kvmgr.c). The kvmgr has TWO varray interfaces: (1) the hash table's varray (kvmgr itself) — which does NOT support COW and uses no-op freeze/thaw; and (2) the data zarray (xzarray_type in zarray.c) — which IS the COW target and uses the real xzarray_freeze/xzarray_thaw implementations. An implementer must NOT accidentally register no-ops for the data zarray, or checkpoint COW will silently skip freezing the data array. The xzarray_type registration in zarray.c already includes xzarray_freeze/xzarray_thaw slots — it is NOT modified by P0-0b. P0-0b only adds no-op slots to the kvmgr hash table's own varray compound literal in kvmgr.c.
src/controller/cluster.c	Wire persist_init/recover/start/stop/fini and HA into lifecycle; wire cluster_load_jobs() to persistence loader; deferred endpoint cleanup (after zarray merge); use delaymgr for checkpoint timer; heartbeat watchdog uses dedicated pthread (see P1-3)
src/controller/include/persist.h	Add persist_nop_freeze/persist_nop_thaw static inline no-ops for varray types that don't support COW
src/controller/netio.c + src/include/xce/netio.h	No changes needed for HA — existing SSL pipeline reused as-is for controller-to-controller connections (trusted LAN model, Part 5.9)
src/controller/join.c	Add redirect logic in handle_join() — standby sends E_XCE_REDIRECT + RedirectInfo for worker/scheduler/apiserver join requests; worker join WRITE lock is acceptable — async replication only (never blocks on repl_wait_ack inside write lock)
src/controller/worker.c	Extend existing mark_worker_unavail() with w->host.dead_since = time(NULL); add WORKER_HIDDEN state path; handle_worker_leave() already calls mark_worker_unavail() and sets w->ep = NULL — no structural change needed; wire restore_worker() into recovery path
src/include/xce/cluster_def.h	Add persist_t persist; (P0-5) and ha_state_t ha; (P1-5) to cluster_t; add WORKER_UNAVAILABLE, WORKER_HIDDEN entity states; worker status uses bitmask flags (e.g., XCE_WORKER_HIDDEN = 0x00200000), not state enum
src/schema/controller.capnp	Add PersistConfig, WALEntry, SnapshotHeader, Snapshot, RedirectInfo schemas; add term to Request; add standby to HostRole; add replEntry/replAck/redirect/heartbeat opcodes; add redirect variant to RequestReply; add deadSince to Worker; worker status remains UInt32 (bitmask, not enum)
src/controller/myconf.c	Parse persistence and HA configuration from YAML
src/worker/main.c	Replace hardcoded controller address with xce_cluster_config_t; add reconnection loop
src/scheduler/	Same pattern as worker — use xce_cluster_config_t
src/frontend/apiserver/backend/clib.go	Use xce_cluster_config_t for controller discovery in RegisterToBackend()
rtti/typeinfo.h	Add vfreeze_fn_t/vthaw_fn_t types, grow intf_varray_t to 8 slots, add vfreeze()/vthaw() global API wrappers (see Appendix A.9)
rtti-0.1.0 (nix)	Bump version from 0.0.1, update nix derivation hash and XCE import path

A.9 rtti Library Enhancements

The rtti library (rtti-0.0.1) is maintained alongside XCE and requires the following additions to support the COW snapshot mechanism. Since there are no released versions of rtti, backward compatibility is not a constraint — these changes are made in-place and version bumped to rtti-0.1.0.

A.9.1 New Function Pointer Types

Add to rtti/typeinfo.h before the intf_varray_t struct definition, after the existing vclear_fn_t typedef (around line 172):

/* COW snapshot support — freeze/thaw for application-level copy-on-write */
typedef bool (*retain_fn_t)(void* arg, void* item);
typedef int  (*vfreeze_fn_t)(meta_obj_t* obj);
typedef int  (*vthaw_fn_t)(meta_obj_t* obj, retain_fn_t should_retain, void* arg);

retain_fn_t is the callback used by vthaw to decide whether an entity survives checkpoint GC. It returns true to retain, false to garbage-collect. Passing NULL for should_retain means "keep all shadow entities, no GC" — used when aborting a checkpoint (merge shadow back without discarding anything). This type must live in rtti (not in persist.h or zarray.h) because vthaw_fn_t references it in the intf_varray_t interface — callers that invoke vthaw(var, my_retain, arg) need the type visible through the rtti header.

A.9.2 Updated intf_varray_t

Add two new slots after the existing six:

typedef struct {
    vstore_fn_t   vstore;
    vget_fn_t     vget;
    vget_vec_fn_t vget_vec;
    vput_fn_t     vput;
    vput_vec_fn_t vput_vec;
    vclear_fn_t   vclear;
    vfreeze_fn_t  vfreeze;  /* NEW — freeze (O(1) ptr swap) */
    vthaw_fn_t    vthaw;    /* NEW — thaw + merge + GC */
} intf_varray_t;

Layout: The struct grows from 24 bytes (6 × 8 byte pointers on LP64) to 32 bytes (8 × 8 byte pointers). The interface_table_t.varray field is already a pointer to intf_varray_t, so the struct size change is isolated to the varray interface — no other interface structs are affected. All existing compound-literal initializers (like the one in kvmgr.c:288 and zarray.c:666) must be updated to supply the two new NULL initializers. The C99 designated-initializer pattern { .vstore = ..., .vget = ..., } is recommended to make this robust against future reordering.

A.9.3 New Global API Functions

Add extern declarations after the existing varray API block (around line 259 of typeinfo.h):

extern int vfreeze(var);                                          /* O(1) freeze — pointer swap */
extern int vthaw(var, retain_fn_t should_retain, void* arg);      /* merge shadow + optional GC; should_retain=NULL keeps all */

These dispatch through RTTI_OBJ_TO_TYPE(obj)->interfaces.varray->vfreeze / ->vthaw, following the same pattern as the existing vstore, vget, etc. wrappers.

A.9.4 No-op Implementations in XCE

librtti provides only the abstract interface — the no-op implementations belong in XCE, not in librtti. Registration sites for varray types that don't support COW supply their own no-ops:

/* In persist.h — shared no-ops for XCE types that don't support COW */
static inline int persist_nop_freeze(meta_obj_t* obj) { return 0; }
static inline int persist_nop_thaw(meta_obj_t* obj, retain_fn_t f, void* a) { return 0; }

Types like kvmgr that don't support COW register these no-ops in their compound literal:

/* kvmgr.c */
.vfreeze = (vfreeze_fn_t)persist_nop_freeze,
.vthaw   = (vthaw_fn_t)persist_nop_thaw,

The library dispatch wrappers (vfreeze(), vthaw()) return -EINVAL when the method pointer is NULL — this is a bug catcher, not a feature gate. Every registration site must supply all 8 function pointers; a NULL slot is a programming error.

A.9.5 xzarray Registration

In zarray.c, the typeinfo_t xzarray_type registration must update the varray interface compound literal to include the new slots:

.varray = &(intf_varray_t) {
    .vstore   = (vstore_fn_t)xzarray_store,
    .vget     = (vget_fn_t)xzarray_get_vec,    /* xzarray_get_vec handles single-get too */
    .vget_vec = (vget_vec_fn_t)xzarray_get_vec,
    .vput     = (vput_fn_t)xzarray_put,
    .vput_vec = (vput_vec_fn_t)xzarray_put_vec,
    .vclear   = (vclear_fn_t)xzarray_clear,
    .vfreeze  = (vfreeze_fn_t)xzarray_freeze,  /* NEW */
    .vthaw    = (vthaw_fn_t)xzarray_thaw,       /* NEW */
},

A.9.6 Build Integration

• Version: Bump rtti-0.0.1 → rtti-0.1.0 in the nix derivation (or equivalent build system metadata)
• Nix store path: The updated library gets a new hash; update the import path in XCE's meson.build or nix flake accordingly
• Compiler flag: No new dependencies; <stdatomic.h> (for the atomic_int election_done flag in Part 5.4) is C11 and should already be in the toolchain
• Test coverage: Add tests for xzarray_freeze/xzarray_thaw round-trip, COW-mode vstore linear scan, and shadow merge correctness with overlapping indices

A.10 System-Level Tunable Constants — include/config.h

All system-level tunable constants are consolidated into include/config.h so that operators and developers can find and adjust limits in one place. Module-specific headers (persist.h, repl.h, cluster_def.h) #include "config.h" and use these definitions — they do not define their own local copies.

/* include/config.h — System-level tunable constants for XCE controller
 * All #define values below are the system-wide defaults.
 * Runtime-overridable values (persist_config_t, ha_config_t) are NOT here —
 * they live in their respective config structs and are set from cluster.bin (controller)
 * or from --peers/--ca-cert CLI flags (workers/schedulers). */

/* --- HA limits --- */
#define MAX_HA_CANDIDATES       4       /* Structural max candidates from --peers flag.
                                          * Reduced from 16: design validated for ≤3. */
#define DEFAULT_HB_INTERVAL_MS  3000    /* Primary heartbeat interval (ms) */
#define DEFAULT_HB_TIMEOUT_MS   10000   /* Standby heartbeat timeout (ms) */
#define DEFAULT_ELECTION_TIMEOUT_MS 2000  /* Election probe deadline (ms) */
#define DEFAULT_SYNC_TIMEOUT_MS 50      /* Sync replication ACK timeout (ms) */
#define DEFAULT_MAX_PROMOTION_RETRIES 5 /* Max failed promotion attempts */
#define DEFAULT_HA_CERT_WARN_DAYS 30   /* Warn if cert expires within N days */
#define MAX_TERM_JUMP           10000   /* Maximum term increment accepted in a single replication
                                          * or OP_replEntry message. Prevents term overflow
                                          * from a corrupted or malicious packet forcing the
                                          * controller into permanent step-down mode.
                                          * Defined in config.h (not inline in Part 6.4) so
                                          * all callers share a single source of truth. */

/* --- Persistence limits --- */
#define WAL_FORMAT_VERSION      1       /* Bumped on incompatible schema changes.
                                          * When bumping, write a migration function
                                          * (e.g., migrate_snapshot_v1_to_v2) before
                                          * changing this value. Older snapshots
                                          * load only if all new fields are optional. */
#define WAL_SEG_SIZE            (64 * 1024 * 1024)  /* 64 MB pre-allocated WAL segment */
#define WAL_HDR_CRC_SIZE        32      /* sizeof(wal_entry_hdr_t) - 2*sizeof(uint32_t) */
#define MAX_WAL_PAYLOAD_BYTES   (1u << 20)  /* 1 MB per WAL entry */
#define WAL_ENTRY_TYPE_MAX     100     /* Highest valid entry type — bump when new types added */
#define MAX_SNAPSHOT_BYTES      (256 * 1024 * 1024)  /* 256 MB — reject larger snapshots */
#define MAX_AUTO_TERM_PER_CHECKPOINT 10240  /* Max JOB_STAT_TRANSIENT jobs auto-terminated in one
                                              * checkpoint cycle. Bounds the heap-allocated
                                              * auto_term[] array in persist_checkpoint().
                                              * Jobs beyond this cap survive to the next
                                              * checkpoint (bounded delay, not data loss). */

/* --- Persistence defaults (runtime-overridable via persist_config_t) --- */
#define DEFAULT_WAL_SYNC_MS     10      /* fsync interval (ms) */
#define DEFAULT_SNAP_INTERVAL_S 60      /* checkpoint interval (s) */
#define DEFAULT_SNAP_WAL_SIZE_MB 64     /* checkpoint when WAL exceeds this (MB) */
#define DEFAULT_FINISHED_RETAIN_S 3600  /* retain finished jobs (s) */
#define DEFAULT_WORKER_RETAIN_S  86400  /* retain unavailable workers (s) */

/* --- Derived --- */
#define MAX_STANDBYS  3  /* Standalone constant — NOT derived from MAX_HA_CANDIDATES.
                          * Deployment ceiling is ≤3 controllers (1 primary + 2 standbys).
                          * Defined independently so bumping MAX_HA_CANDIDATES for structural
                          * reasons does not silently increase MAX_STANDBYS beyond the
                          * validated deployment ceiling. */

Design rule: If a constant affects more than one module or is an operator-visible limit, it belongs in config.h. If it is a purely internal implementation detail of a single module (e.g., an internal struct size or a local retry count), it may stay in the module header. The test: if an operator would ask "what's the maximum?" or a developer would search for the constant when debugging, it goes in config.h.