myxce/src/docs/persistence-and-ha-design.md

XCE Controller: State Persistence & High Availability Design

Status: Reviewed Last Updated: 2026-05-07 Target: src/controller/ (C daemon)

---

Table of Contents

1. Part 1: Unused Code Cleanup
2. Part 2: Single-Controller Persistence
3. Part 3: Checkpoint — Dual zarray COW
4. Part 4: Deferred Entity Cleanup at Checkpoint
5. Part 5: HA Failover — LSF-Style Priority Election
6. Part 6: Fan-Out TCP Replication
7. Part 7: Mixed Sync/Async Replication Protocol
8. Part 8: Hook-Driven Architecture
9. Part 9: Worker Connection Strategy
10. Part 10: libxce Smart Discovery
11. Implementation Phases
12. Appendix: Data Structures & Signatures

---

Part 1: Unused Code Cleanup

The controller has no database code. It uses kvmgr (in-memory hash tables) for all state. The following are dead stubs and unused code that should be removed.

1.1 Dead Code in job.c

Location	Code	Action
job.c:18	#define NJOBS (100)	Remove — unused constant
job.c:20	uint64_t jobid = 100;	Remove — unused global
job.c:63-89	load_jobs() body inside #if 0	Remove dead block, keep function signature (will be rewired in Part 2)

1.2 Unused Function in worker.c

Location	Code	Action
worker.c:367-391	restore_worker()	Keep — currently unused but will be the persistence restore entry point (Part 2). The existing restore_worker(void* cls, xce_worker_t host) signature is unchanged. A new adapter function worker_from_capnp() (see P0-12) will bridge Cap'n Proto deserialization to restore_worker() — the recovery path calls worker_from_capnp(), not restore_worker() directly.

1.3 Empty Function in cluster.c

Location	Code	Action
cluster.c:400	cluster_build_groups() — empty body (return 0;)	Keep for now (groups are a future feature), but add a /* TODO: implement group building */ comment

1.4 Stale Frontend References

The Go API server already removed PostgreSQL (lib/pq removed from go.mod). Remaining stale references:

• src/frontend/docs/ — Update any references to lib/pq or PostgreSQL
• src/frontend/apiserver/store/audit_test.go — Already has forbidden-pattern checks for DBConnPool, DBHandle, InitDB, QueryFromDB; keep these as regression guards

---

Part 2: Single-Controller Persistence

2.1 Architecture: WAL + Periodic Checkpoint

                    ┌──────────────┐
  State mutations ──▶│  Hook System │
                    └──────┬───────┘
                           │ AFTER_EVENT
                    ┌──────▼───────┐
                    │  persist.c    │
                    │  (WAL Writer) │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
       ┌──────────┐ ┌──────────┐ ┌──────────────┐
       │ WAL File │ │ Checkpoint│ │ Replication  │
       │ (append) │ │ (COW snap)│ │ (Part 6)     │
       └──────────┘ └──────────┘ └──────────────┘

The design draws from etcd's architecture, simplified for XCE's single-writer model:

etcd Concept	XCE Adaptation	Why Simplified
WAL (append-only log)	Adopted directly	Proven durability pattern
Chained CRC	Adopted (links each record to previous)	Detects torn/truncated writes
Pre-allocated segments	Adopted (64 MB segments, rotate)	Avoids filesystem allocation latency
8-byte alignment	Adopted	Direct I/O compatibility
BoltDB snapshot	Not adopted — use dual zarray COW	Application-level COW avoids fork() pitfalls
MVCC key-value store	Not adopted	Single-writer, no concurrent readers needing versioning
Raft consensus	Not adopted — LSF-style priority (Part 5)	~300 LOC vs ~5000 LOC

2.2 WAL Entry Format

/* WAL entry header — 40 bytes, 8-byte aligned */
typedef struct {
    uint64_t term;          /* Election term (incremented on failover) */
    uint64_t lsn;           /* Log sequence number within this term */
    uint64_t timestamp;     /* Epoch microseconds */
    uint32_t type;          /* wal_entry_type enum */
    uint32_t payload_len;   /* Bytes following this header */
    uint32_t crc;           /* CRC32 of (header + payload) */
    uint32_t prev_crc;      /* Previous record's CRC (chained); WAL_CRC_NONE (0xFFFFFFFF) for first record in segment */
} wal_entry_hdr_t;
static_assert(sizeof(wal_entry_hdr_t) == 40, "wal_entry_hdr_t must be 40 bytes for 8-byte alignment and on-disk format stability");

/* Sentinel value for prev_crc: no predecessor (first record in segment).
 * 0xFFFFFFFF is used instead of 0 because CRC32 can legitimately produce 0,
 * which would be indistinguishable from the sentinel. */
#define WAL_CRC_NONE 0xFFFFFFFF

/* Bytes covered by CRC in the header — all fields except crc and prev_crc.
 * Used for both on-disk WAL validation and replication CRC verification. */
#define WAL_HDR_CRC_SIZE 32  /* sizeof(wal_entry_hdr_t) - 2*sizeof(uint32_t) */

2.3 WAL Entry Types

enum wal_entry_type {
    /* Job lifecycle — the high-frequency path */
    WAL_JOB_SUBMIT    = 1,  /* SYNC replication */
    WAL_JOB_START     = 2,  /* ASYNC */
    WAL_JOB_FINISH    = 3,  /* ASYNC */
    WAL_JOB_FAIL      = 4,  /* ASYNC */
    WAL_JOB_STOP      = 5,  /* ASYNC */
    WAL_JOB_RESUME    = 6,  /* ASYNC */
    WAL_JOB_UPDATE    = 7,  /* ASYNC */
    WAL_JOB_RUSAGE    = 8,  /* ASYNC */
    WAL_JOB_CKPT      = 9,  /* ASYNC */
    WAL_JOB_PREASON   = 10, /* ASYNC */

    /* Queue and host control — infrequent */
    WAL_QUEUE_CTRL    = 20, /* ASYNC */
    WAL_WORKER_CTRL   = 30, /* ASYNC */

    /* Marker */
    WAL_CHECKPOINT    = 100 /* Marker — no payload */
};

Note: There are no WAL_JOB_DELETE, WAL_WORKER_DELETE, or WAL_QUEUE_DELETE entry types. Entity cleanup is deferred to checkpoint (see Part 4). This means WAL never contains delete records — only creates and updates — simplifying replay logic.

WAL_CHECKPOINT replication skip — code boundary: WAL_CHECKPOINT (type=100) is written to the local WAL only via persist_write_checkpoint_marker() — a dedicated function in persist.c that calls pwrite()/fsync() directly and never calls repl_broadcast(). By contrast, persist_wal_async() and persist_wal_sync_append() call repl_broadcast() after writing. The invariant persist_wal_async(WAL_CHECKPOINT, ...) MUST NOT be called — enforce with assert(type != WAL_CHECKPOINT) at the top of both persist_wal_async() and persist_wal_sync_append() in debug builds. This makes the code-path boundary explicit rather than relying on callers to never pass WAL_CHECKPOINT to the replication path. The last_replicated_crc field in persist_t (distinct from last_crc which covers all entries including checkpoints) is updated only by the replication send paths, never by persist_write_checkpoint_marker() — this is the mechanism that maintains a continuous CRC chain for standbys that skip checkpoint entries. The standby's mirror of this field is ha_state_t.last_received_crc (Appendix A.4), which tracks the CRC of the last replicated entry the standby received; together these two fields implement the chain validation described in Part 6.4.

Mapping to existing hook detail enums:

Hook Detail (cluster_def.h)	WAL Entry Type	Replication	Notes
HOOK_DETAIL_JOB_NEW	WAL_JOB_SUBMIT	SYNC
HOOK_DETAIL_JOB_START	WAL_JOB_START	ASYNC
HOOK_DETAIL_JOB_EXIT	WAL_JOB_FINISH	ASYNC
HOOK_DETAIL_JOB_FAIL	WAL_JOB_FAIL	ASYNC
HOOK_DETAIL_JOB_STOP	WAL_JOB_STOP	ASYNC	Not in current enum — must be added (see footnote 1)
HOOK_DETAIL_JOB_RESUME	WAL_JOB_RESUME	ASYNC	Not in current enum — must be added (see footnote 2)
HOOK_DETAIL_JOB_UPDATE	WAL_JOB_UPDATE	ASYNC
HOOK_DETAIL_JOB_CKPT	WAL_JOB_CKPT	ASYNC
HOOK_DETAIL_JOB_RUSAGE_UPDATE	WAL_JOB_RUSAGE	ASYNC
HOOK_DETAIL_SETPREASON	WAL_JOB_PREASON	ASYNC	Both map to same type — see footnote 3
HOOK_DETAIL_CLRPREASON	WAL_JOB_PREASON	ASYNC	Both map to same type — see footnote 3
HOOK_DETAIL_QUEUE_UPDATE	WAL_QUEUE_CTRL	ASYNC
HOOK_DETAIL_WORKER_JOIN	WAL_WORKER_CTRL	ASYNC
HOOK_DETAIL_WORKER_UPDATE	WAL_WORKER_CTRL	ASYNC
HOOK_DETAIL_WORKER_DEAD	WAL_WORKER_CTRL	ASYNC

Footnotes:

1. HOOK_DETAIL_JOB_STOP: Does not exist in the current cluster_def.h enum. It must be added as a prerequisite for P0-6. Codebase reality: on_job_stop() currently calls mark_job_update() which fires HOOK_DETAIL_JOB_UPDATE — it does NOT fire HOOK_DETAIL_JOB_EXIT. The implementation in P0-4a must update on_job_stop() to call xce_run_hooks directly with HOOK_DETAIL_JOB_STOP, bypassing mark_job_update() for this event. Search grep mark_job_update src/controller/job.c to find the current call sites.

2. HOOK_DETAIL_JOB_RESUME: Does not exist in the current cluster_def.h enum. It must be added as a prerequisite for P0-6. The existing on_job_resume() function in job.c must be updated to call hooks with this new detail.

3. Pending reason deduplication: Both HOOK_DETAIL_SETPREASON and HOOK_DETAIL_CLRPREASON map to WAL_JOB_PREASON without payload differentiation. This is acceptable because pending reasons are transient — after a controller restart or standby promotion, the scheduler re-evaluates pending jobs and publishes fresh pending reasons when scheduling resumes. On recovery, jobs in JOB_STAT_PEND state are assigned a default pending reason ("Job is waiting to be scheduled") until the scheduler updates them. The critical invariant is that the job's pend/run/exit state is preserved, not the specific pending reason text. Ambiguity logging: When apply_wal_entry() processes a WAL_JOB_PREASON entry during WAL replay, it logs at DEBUG level: "WAL_JOB_PREASON replay for jobid %lu — pending reason text is not preserved across recovery" to make the ambiguity visible in logs rather than silent.

WAL payload serialization format: Each WAL entry's payload is serialized as Cap'n Proto using the controller.capnp schemas. The mapping from entry type to schema is:

WAL Entry Type	Cap'n Proto Schema	Key fields serialized
WAL_JOB_SUBMIT	Job	All fields: jobid, name, queue, num_proc, command, environment, submit_time, stat, resource limits
WAL_JOB_START	Job (partial)	jobid, stat, start_time, exec_host
WAL_JOB_FINISH	Job (partial)	jobid, stat, end_time, exit_code
WAL_JOB_FAIL	Job (partial)	jobid, stat, end_time, fail_reason
WAL_JOB_STOP	Job (partial)	jobid, stat
WAL_JOB_RESUME	Job (partial)	jobid, stat
WAL_JOB_UPDATE	Job (partial)	jobid, changed fields only
WAL_JOB_CKPT	Job (partial)	jobid, ckpt_time
WAL_JOB_RUSAGE	Job (partial)	jobid, rusage fields
WAL_JOB_PREASON	Job (partial)	jobid, preason
WAL_QUEUE_CTRL	QueueConf	All fields for the affected queue
WAL_WORKER_CTRL	Worker	All fields for the affected worker
WAL_CHECKPOINT	(none)	No payload — payload_len = 0

"Partial" payloads serialize only changed fields using Cap'n Proto's standard optional field encoding — unset fields are omitted. apply_wal_entry() applies partial payloads by merging into the existing in-memory object (not replacing it). Merge semantics (CRITICAL): For partial payloads, apply_wal_entry() sets only the fields present in the Cap'n Proto payload. Fields that are not set (absent/optional in the Cap'n Proto message) are not touched in the in-memory object — they retain their previous value. This is naturally supported by Cap'n Proto's optional field encoding: a field that was never set in the message has has<Foo>() == false, and the merge logic skips it. Implementers must NOT use a "replace entire struct" approach, which would zero out absent fields and corrupt in-memory state. The merge rule is: if (payload.has<Field>()) in_memory->Field = payload.get<Field>(); for each optional field.

2.4 WAL Segment Management

effective_dir/data/wal/
├── seg-0000000000000000.wal   # Pre-allocated 64 MB
├── seg-0000000000000064.wal   # LSN offset in filename
└── ...

• Segments are pre-allocated to 64 MB (configurable)
• When a segment is full, rotate to the next
• fsync() at configurable interval (wal_sync_ms, default 10ms) — batched, not per-write. If fsync() fails, persist_wal_sync_append() and persist_wal_async() return an error. For sync replication, the failed write is not sent to standbys and the client receives an error reply. For async replication, the error is logged and the controller continues operating — the WAL entry is in the OS page cache but not on disk, so it may be lost on crash. Monitoring should alert on persistent fsync failures.

  Implementation note: delaymgr uses sleep(precision) with precision=2 seconds (see cluster.c:604), giving 2-second minimum granularity — far too coarse for wal_sync_ms=10. The WAL fsync timer must use a dedicated pthread with nanosleep() or a timerfd_create(CLOCK_MONOTONIC) loop, not delaymgr. The periodic checkpoint timer (P0-3) can still use delaymgr since snap_interval_s is measured in seconds.

• WAL trim guard: WAL segments are only deleted up to min(acked_lsn) across all connected standbys that are not in catch-up mode (standby_conn_t.acked_lsn — updated on each OP_replAck received, but only from standbys with in_catch_up == 0). For catching-up standbys, use their OP_replJoin.applied_lsn (the LSN they reported on reconnect) as the floor — WAL segments from that LSN forward must be retained until the standby sends OP_replCatchUpDone. In-catch-up standbys have stale acked_lsn values (from before their catch-up started); using stale acked_lsn as the trim point could trim entries the standby is about to request, forcing snapshot transfer when only a short WAL replay was needed. Concretely: trim_lsn = min(min_non_catchup(acked_lsn), min_catchup(join_applied_lsn), checkpoint_lsn). This prevents trimming segments that a lagging standby still needs for catch-up. When a standby reconnects after a gap, it reports its last applied LSN, and the primary streams missing entries from that point. If the standby's LSN is older than the primary's oldest surviving WAL segment, a full snapshot transfer is required (see Part 6 standby catch-up). Each standby heartbeat includes the standby's applied_lsn so the primary always knows the safe trim point. Empty-standby fallback: When no standbys are connected (replicator.nstandbys == 0), use the primary's own checkpoint_lsn as the trim point — WAL segments before the latest confirmed checkpoint can be safely deleted. Standby disconnect trim delay: When a standby disconnects, the primary should delay WAL trimming by one checkpoint interval before falling back to checkpoint_lsn-based trimming. This gives the standby a window to reconnect and resume catch-up without requiring a full snapshot transfer. The implementation uses per-slot tracking in replicator_t.last_disconnected_lsn[] and replicator_t.last_disconnect_time[] (see Appendix A.3). When a standby disconnects, its acked_lsn and current time are recorded in its slot. The trim point is computed as min(min_over_disconnected_slots(last_disconnected_lsn[i]), checkpoint_lsn) for one checkpoint interval after each disconnect, then falls back to checkpoint_lsn per slot. Slot expiry (CRITICAL): A disconnected slot's last_disconnected_lsn[i] hold on the trim guard expires exactly when now >= last_disconnect_time[i] + snap_interval_s. At that point, last_disconnected_lsn[i] is reset to 0 and last_disconnect_time[i] is cleared, and that slot no longer contributes to the trim guard min(). Permanent decommission: When a standby is removed from the --peers candidate list (its slot index is no longer valid), its last_disconnected_lsn[i] and last_disconnect_time[i] must be zeroed immediately in replicator_remove_candidate(). Without explicit slot invalidation, a decommissioned standby's disconnect record holds the WAL trim guard indefinitely — a WAL growth leak with no upper bound other than wal_max_size_mb.
• WAL growth bound during standby disconnect: The worst-case WAL accumulation when no standbys are connected is: wal_growth_MB ≈ snap_wal_size_mb + (submit_rate_per_sec * avg_WAL_submit_KB * snap_interval_s * 2 / 1024). The * 2 accounts for the one-interval trim delay before checkpoint_lsn-based trimming takes effect. At default settings (snap_interval_s = 60, snap_wal_size_mb = 64) with 10 submits/sec and ~1 KB average WAL_JOB_SUBMIT payload, worst-case growth is ≈65 MB — well within a single segment. At pathological rates (1000 submits/sec, 4 KB average payload), worst-case growth is ≈535 MB, requiring ~9 segments. Mitigation: If WAL disk usage exceeds wal_max_size_mb (default: 2 GB), log an ERROR and trigger an immediate checkpoint to advance checkpoint_lsn and enable trimming. The one-interval trim delay is skipped when replicator.nstandbys == 0 and WAL usage exceeds wal_max_size_mb / 2 — in this scenario, preserving disk space takes priority over the reconnection catch-up window.
• Segments before the latest snapshot are deleted during recovery, subject to the trim guard above
• WAL integrity model (explicit boundary): CRC32 (chained) detects accidental corruption — bit flips, torn writes, truncated segments. It is NOT tamper-resistant: an attacker with write access to the WAL directory can modify payload bytes and recompute a valid CRC32. Tamper resistance would require HMAC-SHA256 keyed on a controller-local secret. XCE's integrity model relies on OS-level filesystem permissions (mode 0700, controller-UID-owned) as the access control boundary, not on cryptographic authentication. Implementers must not assume WAL CRC provides any security guarantee beyond accidental-corruption detection. If tamper resistance is required for a deployment, use encrypted storage (LUKS/dm-crypt) which protects both confidentiality and integrity.
• Chained CRC detects corruption: if prev_crc doesn't match previous record's CRC, stop replay at last valid entry. The first record in a WAL segment uses prev_crc = WAL_CRC_NONE (defined as 0xFFFFFFFF, not 0 — CRC32 can legitimately produce 0 for certain inputs, which would be indistinguishable from the "no predecessor" sentinel if 0 were used). Recovery validates by checking that each record's prev_crc matches the previous record's crc field, starting from WAL_CRC_NONE for the first record. Segment boundary handling: When recovery encounters prev_crc == WAL_CRC_NONE at a position that is NOT the absolute first record in the file (i.e., at the start of a rotated segment), this is treated as a valid chain break — the CRC chain restarts from this point. The recovery algorithm must NOT treat WAL_CRC_NONE at any record position that is 8-byte aligned and at the beginning of a segment file as corruption. Each segment file is independently CRC-chained from its first record. Writer-side obligation (ADV-11): persist_wal_sync_append() and persist_wal_async() MUST set hdr.prev_crc = WAL_CRC_NONE when writing the first record of a new segment (i.e., when persist_t.seg_offset == 0 after segment rotation). The implementation MUST NOT carry persist_t.last_crc across a segment boundary — the new segment starts a fresh chain. This is the mirror of the recovery-side rule; without it, recovery would see a non-WAL_CRC_NONE prev_crc at a segment boundary and enter 8-byte linear scan mode unnecessarily.
• Truncated segment recovery: If a crash occurs mid-write, the last WAL segment may contain a partial record. The recovery algorithm handles two cases:
  1. Valid header CRC, truncated payload: Read the full 40-byte header, validate hdr_crc against WAL_HDR_CRC_SIZE bytes. If the header CRC is valid but payload_len exceeds the remaining bytes in the segment, truncate at the previous valid record. Do NOT use the declared payload_len to advance the read position — a partially-written payload_len could be corrupted, causing the scanner to jump past valid records.
  2. Invalid header CRC (corrupted header): Enter 8-byte linear scan mode — skip forward 8 bytes at a time, checking for a valid hdr_crc and matching prev_crc at each alignment boundary. This ensures that a corrupted header cannot cause the scanner to skip past valid data.
• WAL trim and total_wal_bytes accounting: After a successful checkpoint, the WAL trim step deletes segment files whose last LSN is ≤ checkpoint_lsn. On each segment deletion, total_wal_bytes is decremented by the segment file size (fstat(fd).st_size). On each WAL write, total_wal_bytes is incremented by the bytes written. On segment rotation, the new segment file size is not counted until bytes are written to it. This ensures total_wal_bytes accurately reflects the on-disk WAL size at all times. In both cases, recovery stops at the end of the WAL segment (the segment file size) when it cannot find a valid record. The seg_offset field in persist_t tracks the last fsync point at runtime but is in-memory and lost on crash — recovery must NOT rely on it as the trust boundary. Instead, recovery validates entries up to the physical end of the segment file using chained CRC: records with invalid CRC or prev_crc mismatch are treated as corruption. The recovery algorithm scans from the beginning of the segment and stops when it encounters an unrecoverable CRC failure at or near the end of the file. seg_offset is only used during normal operation to track how far the fsync thread has flushed — it is updated to the post-write position only after fsync() succeeds, ensuring that on crash + restart (before the next fsync), the runtime can still distinguish flushed from unflushed data if needed.

Data at rest: WAL segments and snapshot files contain job submission payloads (which may include environment variables, command lines, and user-specified data). If the storage directory is on a shared or compromised filesystem, this data is accessible. XCE relies on OS-level filesystem permissions (effective_dir/data/wal and effective_dir/data/snap directories should be owner-read-only, mode 0700, where effective_dir is derived from --workdir and -n name CLI flags) and physical host security rather than encryption at rest. If encryption at rest is required, it should be provided by the underlying filesystem (e.g., LUKS, encrypted EBS volumes) rather than by the controller — adding application-level encryption would double the encryption cost with filesystem-level encryption already in place for most production deployments.

Data retention and scrubbing: Sensitive data (environment variables, job command lines) persists in WAL files until those segments are trimmed (after checkpoint), and in snapshot files until the next checkpoint replaces them. The retention window is bounded by finished_retain_s (default 1h) for job payloads: once a job is GC'd at checkpoint, its data no longer appears in new snapshots, and its WAL segment is trimmed after the checkpoint that removed it. Old WAL segments and snapshot files are deleted with unlink() — no explicit overwrite/scrubbing of file data is performed. If the storage device requires explicit data destruction (e.g., SSDs with wear leveling where unlink does not guarantee physical erasure), this must be handled at the storage layer (encrypted volumes with key rotation, or secure erase tooling). XCE does not implement application-level data scrubbing of WAL or snapshot files.

WAL payload logging policy: The persistence and replication modules must never log WAL payload content. Only entry metadata (type, LSN, term, payload_len, CRC) may appear in log messages. This is a security hardening invariant: WAL_JOB_SUBMIT payloads contain job environment variables which frequently hold API keys, database passwords, and other credentials. All logging call sites in persist.c and repl.c that reference WAL entries must log only hdr->type, hdr->lsn, hdr->term, hdr->payload_len, and CRC values — never the payload pointer or its contents. Code review must verify this as a gating check before merge. The CRC mismatch error path already follows this pattern: ERROR("replication CRC mismatch: expected %u, got %u, lsn %lu", hdr.crc, computed_crc, hdr.lsn). Structural enforcement (required — P0-1 deliverable): Define a persist_log_entry_metadata(const wal_entry_hdr_t* hdr) helper in persist.h that accepts only the header struct and logs type, lsn, term, payload_len, and CRC — it has no payload parameter and cannot log payload content. All log call sites in persist.c and repl.c that emit WAL entry information must use this helper. This converts credential-logging from a review-time catch to a build-time enforcement: a developer cannot format-print a payload pointer without explicitly calling an unwrap function that clearly signals intent. Relying solely on code review is a human control that will eventually fail given how frequently debug log statements are added during development. Add persist_log_entry_metadata() to the P0-1 deliverable table.

Filesystem type requirement: effective_dir/data/wal and effective_dir/data/snap must be on a local filesystem (ext4, xfs, etc.), not on NFS or other network filesystems. (where effective_dir is derived from --workdir and -n name CLI flags) The fsync() durability guarantee depends on the OS writing to stable storage — on NFS, fsync() may only flush to the server's cache, not to persistent storage, undermining the WAL's crash-recovery guarantee. Run a startup check using statfs() and reject f_type values matching NFS_SUPER_MAGIC (0x6969) or SMB_SUPER_MAGIC (0x517B) — log ERROR and abort if detected.

2.5 Recovery Sequence

On startup, cluster_init() calls persist_recover():

1. Find latest snapshot: Scan effective_dir/data/snap/ for the snapshot with the highest LSN
2. Load snapshot: Read the snapshot file into memory 2a. Verify snapshot checksums: (i) Recompute CRC32 over the header fields (lsn, timestamp, term, version, nJobs, nQueues, nWorkers) and compare against SnapshotHeader.headerChecksum. If mismatch, the header has been tampered or corrupted — log an error and try the next-oldest snapshot. (ii) Recompute CRC32 over the Cap'n Proto payload and compare against SnapshotHeader.checksum. If mismatch, log an error and try the next-oldest snapshot. If all snapshots fail verification and the WAL contains entries that predate any valid snapshot (i.e., WAL segments have already been trimmed), the controller must abort with a fatal error — silently booting with empty state would silently discard live job/worker state and is data loss. The fatal error message must include: the number of corrupt snapshots found, the LSN range of surviving WAL segments, and the instruction "Contact support or restore from backup before restarting". Do not fall back to empty state in this scenario. The empty-state fallback is only valid on first boot (see below). Snapshot version check: After verifying both checksums, check SnapshotHeader.version. If the version is higher than the current controller's WAL_FORMAT_VERSION (defined in persist.h as 1), log a fatal error and abort — a snapshot written by a newer controller cannot be safely loaded by an older one. This catches version mismatches earlier than encountering unknown WAL entry types during replay. Fallback WAL verification: When falling back from the latest snapshot to an older snapshot (due to checksum failure), the recovery code must first verify that WAL segments exist from older_snapshot.lsn + 1 forward. If WAL segments have already been trimmed past the older snapshot's LSN (i.e., only segments covering [latest_snapshot.lsn, ...] exist), the fallback cannot replay the gap and the controller must abort with a fatal error. Do NOT silently accept an older snapshot with no WAL to replay — this would start the controller with stale state, silently discarding all changes between the older snapshot and the trimmed WAL. 2b. Deserialize: Deserialize Cap'n Proto data, rebuild kvmgr instances. Set capnp::ReaderOptions.traversalLimitInWords to a fixed upper bound (MAX_SNAPSHOT_BYTES / 8 words) to prevent O(n²) traversal amplification from a crafted snapshot — this is the same hardening practice used for WAL payload deserialization (Part 6.4). Do NOT set it proportional to the actual file size, as a small file with deeply nested structure can still trigger excessive traversal. The limit should be 64 * 1024 * 1024 / 8 = 8M words (the 64 MB is the expected max snapshot size). If the snapshot file is larger than MAX_SNAPSHOT_BYTES (default: 256 MB — 4× the expected 64 MB for 10K jobs × 1 KB + overhead), log an ERROR and reject the snapshot.
   • Jobs → makeJob(c->jobmgr, ...) (reuse existing create_job())
   • Workers → restore_worker(cls, host) (already exists in worker.c:367)
   • Queues → create_queue(c, qconf) (already exists)
   • Resources → makeResource(c->resmgr, ...) (already exists)
   • Configs → add_config(c, key, val) (already exists)
3. Replay WAL: Starting from the snapshot's LSN, read and apply WAL entries forward via apply_wal_entry(). Hooks are not yet registered during recovery — persistence hooks are registered only after persist_recover() completes in cluster_init(), so no bypass flag is needed. This avoids re-persisting (WAL write during recovery) and prevents replication sends during replay. For WAL_JOB_SUBMIT entries, replay calls restore_job(c, jobid, payload) — a dedicated function that inserts the job at its original jobid without allocating a new one. Calling job_submit() here would assign a new jobid, breaking replay by creating duplicate or wrong-ID jobs. Job ID allocation note: The controller does NOT allocate jobids — job IDs are assigned by the Go API server via idgen (clib.go) before the submit request reaches the controller. Recovery merely restores jobs at their original IDs from the WAL; no idgen re-seeding is required in the controller. Recovery is single-threaded: persist_recover() runs before cluster_init() starts the iomgr, vcpu pool, or any other threads — all restore_job()/xkvmgr_add() calls during recovery are serialized and require no additional locking beyond what the functions already provide. Worker behavior during recovery: iomgr is not yet listening during recovery, so workers attempting to connect receive "connection refused". libxce's xce_reconnect() backoff+jitter loop handles this transparently — no protocol-level "not ready" response is needed. Workers connect normally once iomgr starts after recovery completes.
4. Trim WAL: Delete WAL segments fully covered by the loaded snapshot
5. Retention pass: After replay, run the retention/GC check on all entities (equivalent to a thaw pass) to clean up any entities whose TTL has expired during the controller's downtime. This handles the case where a controller was down for longer than finished_retain_s or worker_retain_s — expired entities are cleaned immediately on recovery.
6. Restore election term: Set ha.term = max(snapshot_header.term, max_term_from_replayed_wal). This is a security-critical invariant: a controller must never start with a term lower than any term it has previously used, as this would undermine split-brain fencing. On first boot (empty state), ha.term is initialized to 0. P0 scope note: ha_state_t and its term field are defined in Part 5.3 (HA design), which is implemented in P1. P0 implementers must nonetheless allocate a uint64_t placeholder for the term in the snapshot header (SnapshotHeader.term) and in WAL entry headers (wal_entry_hdr_t.term) — these fields exist in the serialized format even before HA is wired up. During P0 (single-controller), ha.term is always 0; the restore step is a no-op. The field is read and written correctly in P0 so that P1 can wire ha_state_t.term to it without format changes. Snapshot version backward compatibility: If SnapshotHeader.version is lower than the current WAL_FORMAT_VERSION, this indicates a snapshot written by an older controller. New schema fields must be optional with defaults to ensure backward compatibility. If the schema has breaking changes between versions, a migration function (e.g., migrate_snapshot_v1_to_v2()) must be written before bumping WAL_FORMAT_VERSION. Do not accept older versions without verifying field compatibility — the default behavior of loading an older snapshot with missing fields is safe only if all new fields are optional with sensible defaults.

Forward compatibility warning — unknown WAL entry types during recovery: apply_wal_entry() logs a WARN and returns 0 (skips) unrecognized wal_entry_type values. This allows a controller running an older version to recover from WAL segments written by a newer version — the old controller boots, skips entries it cannot interpret, and continues operating. However, the state changes intended by those skipped entries are silently lost. This is a downgrade risk: if a newer controller writes entry types the old controller does not understand, and the old controller is started on that WAL, the cluster state will be incomplete (missing features, incorrect job/worker state for data written by the newer controller). Deployments should never roll back to an older controller version after a newer version has written persistent state. If rollback is required, the only safe procedure is: drain all jobs, stop all controllers, delete WAL and snapshot directories on all candidates, then start the old controller in empty-state-first-boot mode.

WAL entry type version guard: To convert silent data loss into a loud failure, persist_recover() tracks the maximum wal_entry_type value encountered during WAL replay. If any entry has type > WAL_ENTRY_TYPE_MAX (defined in persist.h as the highest known entry type in the current version — initially WAL_CHECKPOINT = 100), persist_recover() logs a FATAL error and aborts: "WAL contains entry type %u beyond current version's maximum %u — controller version is too old to safely recover; upgrade to a newer version or drain and reset WAL". This mirrors the snapshot version check pattern (step 2a) and prevents an older controller from silently booting with incomplete state. Operators who genuinely need to force-boot against a newer WAL (e.g., emergency recovery) can use the --force-wal-version startup flag, which downgrades the abort to a WARN per unknown entry type.

Empty state (first boot after upgrade): If both effective_dir/data/snap/ and effective_dir/data/wal/ are empty (no snapshots, no WAL segments), persist_recover() returns 0 (success) with no state loaded — equivalent to today's in-memory-only boot. This ensures backward compatibility: existing deployments upgrading from in-memory-only operation need no migration steps. The controller simply starts with empty state and begins accumulating WAL entries.

Valid snapshot, empty WAL (clean shutdown): If effective_dir/data/snap/ contains a valid snapshot file but effective_dir/data/wal/ has no segments, persist_recover() loads the snapshot with no WAL replay. This is the expected state after a clean shutdown where the last checkpoint has already trimmed all WAL segments. No error or warning is generated.

Last-known-good snapshot retention: When a new checkpoint completes successfully, the controller retains the previous snap_keep_n - 1 snapshots on disk (default snap_keep_n = 2, i.e., current + one previous). When this fallback works: If the controller crashes mid-checkpoint (before the new snapshot is fsync'd and the WAL is trimmed), the previous snapshot and its follow-on WAL segments are intact — recovery falls back to the previous snapshot and replays the bridging WAL. When this fallback does NOT work: If a checkpoint completed successfully (new snapshot fsynced, WAL trimmed up to new checkpoint_lsn), and the resulting snapshot is corrupt (e.g., silent disk error), the WAL entries covering the previous→new snapshot gap are already trimmed and unavailable. In this case the fallback fails at the WAL-existence check (see step 2a) and the controller aborts with a fatal error — snap_keep_n provides no protection against completed-checkpoint corruption. The fallback chain walks snapshots from newest to oldest, stopping at the first valid one where WAL covers the gap. Snapshot rotation: after writing a new snapshot, delete snapshots beyond snap_keep_n, oldest first.

2.6 Configuration

typedef struct {
    /* Paths are derived from --workdir and -n name CLI flags at startup.
     * Not stored as config fields. Derivation rules (name is mandatory):
     *   effective_dir = workdir + "/" + name
     *   wal_path      = effective_dir + "/data/wal"
     *   snap_path     = effective_dir + "/data/snap"
     *   cluster_cfg   = effective_dir + "/conf/current/cluster.bin"
     *   pid_file      = effective_dir + "/run/controller.pid"
     */
    uint32_t wal_sync_ms;      /* fsync interval (default 10) */
    uint32_t snap_interval_s;  /* checkpoint interval (default 60) */
    uint32_t snap_wal_size_mb; /* checkpoint when WAL exceeds this (default 64) */
    uint32_t snap_keep_n;      /* number of snapshots to retain (default 2 — current + one previous) */
    uint32_t transient_job_timeout_s; /* seconds after failover before JOB_STAT_TRANSIENT jobs are
                                         auto-terminated (default 300 = worker_reconnect_timeout_s).
                                         0 = no auto-termination (operator must intervene manually).
                                         After timeout: job → JOB_STAT_EXIT with reason
                                         "worker did not reconnect after failover"; worker →
                                         WORKER_UNAVAILABLE. Increments ctrl_transient_jobs_stuck
                                         metric for each auto-terminated job. */
    uint32_t finished_retain_s;/* retain finished jobs for N sec (default 3600) */
    uint32_t worker_retain_s;  /* retain unavailable workers for N sec (default 86400) */
    uint32_t sync_timeout_ms;  /* sync replication ACK timeout (default 500) */
} persist_config_t;

Working Directory Layout (unified for dev and production):

${XCE_WORKDIR}/                    e.g. /var/lib/xce (prod) or /tmp/xce (dev)
├── ctrl-1/                        effective_dir for -n ctrl-1
│   ├── conf/
│   │   ├── current/
│   │   │   └── cluster.bin        controller-specific config (WAL params, HA timeouts)
│   │   └── previous/
│   │       └── cluster.bin        rollback target (atomic rename)
│   ├── data/
│   │   ├── wal/                   mode 0700
│   │   └── snap/                  mode 0700
│   └── run/
│       └── controller.pid
│
├── ctrl-2/                        effective_dir for -n ctrl-2
│   └── ...
│
└── ctrl-3/                        effective_dir for -n ctrl-3
    └── ...

Controller startup:

xce_controller \
  --workdir /var/lib/xce \
  -n ctrl-1 \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000

The -n flag is mandatory. Candidate list is passed via --peers CLI flag (no config file for peers). cluster.bin contains only controller-specific config (WAL/HA parameters), not the candidate list.

Added to cluster_config_t in the Cap'n Proto schema and myconf.c YAML parsing.

2.7 Controller Connection Context

The controller only handles connections from three peer types:

1. Workers — compute node daemons reporting job status and receiving dispatch
2. Scheduler — job placement service
3. API servers — Go REST frontend (1–3 instances)

End users never connect directly to the controller — the API server provides traffic control, rate limiting, and request validation. This has important implications:

• Sync replication: Only triggered by job submit from API servers. At 1–3 API server connections, vcpu pool exhaustion from blocking repl_wait_ack() is not a concern — the wait is bounded by sync_timeout_ms (default 500ms) and only affects the vcpu handling that specific API server's request.
• Connection count: Workers (100s–1000s) + scheduler (1–2) + API servers (1–3). The scheduler and API server connections are long-lived and stable.
• All connections use SSL: The controller's iomgr enforces SSL on all connections (see netio.c). Replication channels must also use this pipeline, not raw TCP.

---

Part 3: Checkpoint — Dual zarray COW

SUPERSEDED. This section (originally: a copy-on-write dual-zarray checkpoint mechanism with freeze/thaw semantics and a two-level vget/vput/vstore lookup) described a design that was later replaced. See: src/docs/persistence-and-ha-design.md findings and the closed spec ha-entity-snapshots in .kiro/specs/ for what shipped instead — the actual checkpoint path (persist_checkpoint() / write_snapshot() in src/controller/persist.c) collects entities directly via collect_job_frames() / collect_queue_frames() / collect_host_frames() and writes Cap'n Proto–encoded frames; zarray.h has no freeze/thaw/COW/shadow API.

---

Part 4: Deferred Entity Cleanup at Checkpoint

4.1 Design Philosophy

No entity is ever deleted during normal operation. All entities follow the same lifecycle:

User-visible behavior: This is intentionally different from eager deletion. Finished jobs and unavailable workers remain visible in API queries (filtered by their terminal status) for their configured retention period — this is the desired behavior. Users need to see recently completed jobs to check exit status and history. Abruptly removing entities on state transition (e.g., a job disappearing the moment it finishes) would confuse users and break operational workflows that depend on querying recent job results. The retention periods (finished_retain_s, worker_retain_s) control how long this historical data remains visible.

alive state ──▶ terminal state ──▶ [TTL/retention expires] ──▶ cleaned at next checkpoint

This eliminates vclear from the hot path. The del_key callback in kvmgr is still registered as NULL (the existing design in kvmgr.c:286), but the deferred cleanup mechanism replaces it: instead of xkvmgr_del() being called at state transition time, expired entries are removed during checkpoint GC via a kvtab_gc() scan that iterates over all kvmgr entries, calls should_retain() on each, and removes those that fail retention. This is safe because:

1. Checkpoint GC holds the kvmgr write lock (serializing all other access)
2. kvtab removal is O(n) but n = total entries (10K jobs, etc.) — fast enough for the infrequent checkpoint path
3. Eliminates the need for del_key entirely — the existing NULL registration is the intended design, with deletion deferred to checkpoint GC instead of the hot path.

Design invariant: del_key is deliberately NULL. The deferred GC at checkpoint handles entity removal via kvtab_gc(), not through hot-path key deletion. The RTTI interface's del_key = NULL is correct — no change is needed.

This simplifies COW (no tombstones, no three-state flags — see Part 3)

4.2 Terminal States by Entity Type

Entity	Terminal States	Trigger	Cleanup Criteria
Job	JOB_STAT_DONE, JOB_STAT_EXIT, JOB_STAT_PERR	Job finishes naturally	No dependents AND finish_time + finished_retain_s < now
Job	JOB_STAT_TRANSIENT (0x400)	Primary failover — job was RUNNING on old primary but no state update in standby's WAL	Worker reconnection (sets actual state) OR transient_since + transient_job_timeout_s < now — auto-terminate with JOB_STAT_EXIT and reason "worker did not reconnect after failover" (see config transient_job_timeout_s, default 300s)
Worker	WORKER_UNAVAILABLE	Worker daemon dies / disconnects	dead_since + worker_retain_s < now
Worker	WORKER_HIDDEN	Admin explicit remove (off-hours maintenance only)	Immediate cleanup at next checkpoint (no TTL)
Queue	QUEUE_HIDDEN	Admin deletes queue (off-hours maintenance only)	Immediate cleanup at next checkpoint
Config	CONFIG_HIDDEN	Admin deletes config key	Immediate cleanup at next checkpoint
Resource	(never deleted)	N/A	Resources persist permanently

Important: In production clusters, deleting queues or removing hosts while the cluster is running can cause job loss and is not acceptable. WORKER_HIDDEN and QUEUE_HIDDEN are intended for off-hours maintenance windows only, when no jobs are running on the affected resources. The deferred cleanup design exists so that removals during maintenance can be batched and completed efficiently at the next checkpoint.

4.3 How Terminal States Work

Entities in terminal states remain inuse=1 in the zarray and still have their kvtab entries. Business logic ignores them through status-based filtering that already exists:

• Scheduler skips JOB_STAT_DONE/JOB_STAT_EXIT/JOB_STAT_PERR jobs (already happens)
• Scheduler skips WORKER_UNAVAILABLE workers (already happens)
• API queries filter by status (already happens)

The new states WORKER_HIDDEN, QUEUE_HIDDEN, CONFIG_HIDDEN follow the same pattern — business logic and queries skip them, but the data structure entries persist until the next checkpoint.

4.4 New HostState / EntityState Values

Worker status uses bitmask flags (e.g., XCE_WORKER_UNAVAIL = 0x00010000), not a state enum. Add new bitmask flags and a QueueStatus hidden value:

/* In types.h — new bitmask flags for deferred cleanup (XCE_WORKER_UNAVAIL already exists at 0x00010000) */
#define XCE_WORKER_HIDDEN    (0x00200000)  /* Admin explicitly removed — invisible but not yet cleaned up */

/* Job status — add transient bitmask for HA failover.
 * NOTE: 0x200 is already JOB_STAT_WAIT (types.h:351) — do NOT reuse it.
 * JOB_STAT_TRANSIENT uses 0x400 which is currently free. */
#define JOB_STAT_TRANSIENT   (0x400)       /* Job state uncertain after failover — awaiting worker re-join */

/* Queue status — add hidden bitmask */
#define QUEUE_STAT_HIDDEN    (0x0100)       /* Admin deleted — invisible but not yet cleaned up */

In the Cap'n Proto schema, Worker.status is UInt32 (bitmask-compatible). QueueStatus may also need a hidden value added depending on its representation.

XCE_WORKER_UNAVAIL already exists in types.h as 0x00010000. The new additions are XCE_WORKER_HIDDEN, JOB_STAT_TRANSIENT, and QUEUE_STAT_HIDDEN (or equivalent).

4.5 Retention Policy at Checkpoint

The should_retain callback passed to xzarray_thaw() implements entity-specific retention.

Supporting functions (both implemented as part of P0-13):

/* job_is_terminal — returns true if job is in a terminal state */
bool job_is_terminal(const job_t* j);
/* Implementation: return (j->stat & (JOB_STAT_EXIT | JOB_STAT_DONE | JOB_STAT_PERR)) != 0; */

/* job_has_dependents — returns true if any other job depends on this jobid.
 * Dependency sources:
 *   - Array jobs: if j is an array master, any array element with stat != terminal
 *   - Job dependencies: any pending job with pending_ctrl referencing j->jobid
 *     (i.e., held because this job must finish first)
 * Called only on terminal jobs (job_is_terminal() == true) during checkpoint.
 * Caller holds kvmgr write lock (during thaw), so this function must not
 * re-acquire any kvmgr lock.
 *
 * CRITICAL: During thaw, shadow entries are merged into active one at a time.
 * When should_retain() is called for shadow[i], entries at higher indices
 * (shadow[j] where j > i) have NOT yet been merged into active[]. If
 * job_has_dependents() scans only active[], it will miss dependents that
 * reside in not-yet-merged shadow slots, causing premature GC of jobs that
 * have dependents. To avoid this, xzarray_thaw() MUST use a two-pass
 * approach: (1) merge ALL shadow entries into active[] (pointer copies for
 * untouched entries, skip entries where active[i].inuse == 1 and free shadow[i].data),
 * then (2) run should_retain() checks on the fully-merged active[] array.
 * This ensures that all entities are visible in active[] before any retention
 * decision is made. The two-pass approach is safe because merge (step 1) only
 * copies pointers and sets inuse flags — it does not call should_retain() or
 * free any memory.
 *
 * Returns false if the jobid is not found (treated as no dependents).
 *
 * LOCKING REQUIREMENT: should_retain() MUST be called inside the kvmgr WRITE
 * lock section (during thaw). It MUST NOT be called off the critical path
 * (e.g., in a background GC thread) because concurrent kvtab lookups via
 * xkvmgr_find() (holding only a READ lock) would encounter freed entities
 * still referenced by hash entries until kvtab_gc runs. The window between
 * entity free and kvtab cleanup must be atomic with respect to READ-lock
 * lookups. The checkpoint flow (freeze → serialize → thaw/GC → kvtab_gc →
 * unlock) guarantees this atomicity by holding the WRITE lock throughout. */
bool job_has_dependents(cluster_t* c, uint64_t jobid);

/* Job retention: keep if it has dependents OR TTL not expired.
 * Also handles JOB_STAT_TRANSIENT auto-termination via transient_job_timeout_s. */
static bool job_should_retain(void* arg, void* item) {
    cluster_t* c = (cluster_t*)arg;
    job_t*     j = (job_t*)item;

    /* Auto-terminate transient jobs whose timeout has expired.
     * JOB_STAT_TRANSIENT is NOT terminal (job_is_terminal() would return false),
     * so we handle it before the terminal check. transient_job_timeout_s=0 disables
     * auto-termination (operator must intervene manually). j->transient_since is set
     * when the job enters JOB_STAT_TRANSIENT at promotion time. */
    if ((j->stat & JOB_STAT_TRANSIENT) &&
        c->persist.config.transient_job_timeout_s > 0) {
        time_t now = time(NULL);
        if (j->transient_since > 0 &&
            now > j->transient_since + (time_t)c->persist.config.transient_job_timeout_s) {
            /* Convert to terminal state so it falls through to normal GC below.
             * LOCK ORDERING NOTE: Do NOT write a WAL entry here.
             * job_should_retain() is called from xzarray_thaw() which is called
             * from persist_checkpoint() while holding the kvmgr WRITE lock.
             * persist_wal_async() requires persist.lock — acquiring persist.lock
             * while holding the kvmgr WRITE lock would introduce an undocumented
             * lock ordering dependency that can deadlock with vcpu threads that
             * acquire persist.lock first then attempt xkvmgr_find() (READ lock).
             * Instead: record the auto-terminated jobid in a local heap array
             * declared in persist_checkpoint():
             *   uint64_t* auto_term = calloc(MAX_AUTO_TERM_PER_CHECKPOINT, sizeof(uint64_t));
             *   int n_auto_term = 0;
             * (MAX_AUTO_TERM_PER_CHECKPOINT defined in config.h, default 10240.)
             * This is a local variable in persist_checkpoint(), NOT a field of persist_t.
             * After thaw completes and the kvmgr WRITE lock is released, walk the array
             * and call persist_wal_async() for each jobid. On persist_wal_async() failure,
             * log ERROR("auto-term WAL write failed for jobid %lu: %s", jobid, strerror(e))
             * and continue. The job reverts to JOB_STAT_TRANSIENT after recovery, but this
             * does NOT loop forever: the recovery retention pass (Part 2.5 step 5) treats
             * any JOB_STAT_TRANSIENT job older than transient_job_timeout_s as immediately
             * auto-terminatable. If the WAL write keeps failing across restarts, the job is
             * cleaned up on the NEXT successful checkpoint after recovery — bounded by one
             * checkpoint interval (snap_interval_s), not across unbounded restart cycles.
             * Increment ctrl_transient_jobs_stuck for each failed WAL write so monitoring
             * can detect persistent I/O problems that prevent clean GC.
             * If n_auto_term >= MAX_AUTO_TERM_PER_CHECKPOINT, stop recording (remaining
             * jobs survive this checkpoint and are auto-terminated in the next one — bounded
             * delay is acceptable). free(auto_term) after WAL writes complete. */
            j->stat = JOB_STAT_EXIT;
            j->finish_time = now;
            WARN("transient job %lu on worker %s auto-terminated after %us: "
                 "worker did not reconnect after failover",
                 j->jobid, j->exec_host, c->persist.config.transient_job_timeout_s);
            /* ctrl_transient_jobs_stuck metric incremented by caller after lock release */
        } else {
            return true;  /* transient timeout not yet expired */
        }
    }

    if (!job_is_terminal(j))
        return true;                          /* still active, keep */

    if (job_has_dependents(c, j->jobid))
        return true;                          /* other jobs depend on it */

    time_t now2 = time(NULL);
    if (j->finish_time + c->persist.config.finished_retain_s > now2)
        return true;                          /* TTL not expired */

    return false;                             /* safe to clean up */
}

/* Worker retention: keep if TTL not expired (or hidden = immediate) */
static bool worker_should_retain(void* arg, void* item) {
    cluster_t* c = (cluster_t*)arg;
    worker_t*  w = (worker_t*)item;

    if ((w->host.status & XCE_WORKER_HIDDEN) != 0)
        return false;                          /* admin removed, cleanup immediately */

    if (!IS_WORKER_UNAVAIL(w->host.status))
        return true;                          /* still connected, keep */

    time_t now = time(NULL);
    if (w->host.dead_since + c->persist.config.worker_retain_s > now)
        return true;                          /* TTL not expired */

    return false;                             /* safe to clean up */
}

CLOCK_MONOTONIC vs wall-clock time for retention: time(NULL) returns wall-clock time, which is vulnerable to NTP step adjustments (backward clock jumps can cause premature GC of terminal entities whose TTL appears to have expired). For production deployments, either: (a) use clock_gettime(CLOCK_MONOTONIC) for retention comparisons and store a monotonic timestamp alongside finish_time/dead_since, or (b) document the sensitivity to NTP step changes and require slewing-only NTP configuration (e.g., chrony with the smoothtime directive, which never steps the clock backward). The heartbeat module already uses CLOCK_MONOTONIC (Part 5.5) for exactly this reason. Option (a) is preferred but requires storing a struct timespec or time_t from CLOCK_MONOTONIC per entity (one extra 8-byte field per job/worker). Option (b) is acceptable for deployments where NTP is controlled by the cluster administrator.

4.6 kvtab Cleanup at Checkpoint

When xzarray_thaw frees an entity (returns it from should_retain as false), the kvtab still has a key→index mapping for it. After thaw completes, walk the kvtab and prune stale entries:

void kvmgr_gc_keys(meta_obj_t* obj) {
    xkvmgr_t* kvm = RTTI_OBJ_TO_DATA(xkvmgr_t, obj);

    /* Walk all hash chains. For each entry, check if the
       zarray slot at the mapped index is still inuse.
       If inuse=0, the entity was GC'd — remove hash entry. */
    kvtab_gc_stale_indices(kvm->kvtab, kvm->data);
}

This runs while the write lock is still held (during thaw), so it's safe and bounded. For 10K entries, the hash chain scan is ~tens of microseconds.

4.7 Index Reuse After Cleanup

After checkpoint cleanup, freed zarray slots become inuse=0 with their kvtab entries removed. The zarray's circular scan (pos) finds these freed slots, so new vstore calls reuse them. A new job can be assigned index 42 after the old job 42 was cleaned up — the kvtab gets a fresh key→42 entry for the new job. No confusion because the old key→42 entry was removed.

4.8 Benefits of Deferred Cleanup

Aspect	Before (eager delete)	After (deferred to checkpoint)
COW complexity	Requires tombstones or three-state flags	Simple — inuse=0 always means "not touched"
WAL entry types	Need WAL_*_DELETE types	No delete types — WAL is purely additive
WAL replay	Must handle insert-delete-reinsert correctly	Insert or update — always correct
Delete spikes	Random (worker disconnect, admin action)	Bounded — only at checkpoint intervals
Memory overhead	Freed immediately	Jobs retained for up to snap_interval_s; workers retained for up to worker_retain_s (default 86400s = 24h). Memory sizing: a typical worker_t in XCE is ~500 bytes. Retaining 2000 disconnected workers for 24h consumes ~1 MB — negligible. Job retention at 1h (default finished_retain_s = 3600) with 100K finished jobs × ~1 KB/job = ~100 MB at the high end; operators with very high job throughput may reduce finished_retain_s to limit peak memory.
zarray fragmentation	Holes appear unpredictably	Compacted at checkpoint during merge
kvtab fragmentation	Hash chains shrink unpredictably	Pruned at checkpoint — predictable

4.9 Endpoint Deferred Cleanup

Endpoints (the endpoint_t objects in the controller's c->endpoints zarray) also follow the deferred cleanup pattern, but with an important difference: the network resource (fd, SSL object) must be released immediately, while the slot cleanup is deferred.

When a worker or scheduler disconnects:

1. Immediate: Close fd and free SSL* — these are OS resources that cannot be retained
2. Immediate: Set ep->busy = 0 (unblock any pending vtask queue entries)
3. Deferred: Set ep->state = ENDPOINT_DISCONNECTED but do NOT call xzarray_clear() on the slot — defer cleanup to checkpoint

The endpoint slot is cleaned up at checkpoint via the same should_retain mechanism:

static bool endpoint_should_retain(void* arg, void* item) {
    endpoint_t* ep = (endpoint_t*)item;
    /* Disconnected endpoints have fd=-1 and SSL=NULL (already freed).
       The slot just needs to be reclaimed by the zarray. */
    return (ep->state != ENDPOINT_DISCONNECTED);
}

Why not delete immediately? Calling xzarray_clear() during handle_endpoint_error() would set inuse=0 on the active array during COW mode, creating a "hole" that doesn't exist in the shadow. This would cause vget two-level lookup to incorrectly fall through to stale shadow data. Deferring to checkpoint avoids this entirely.

Note on array type: The current c->endpoints uses plain zarray_t. As part of P0-2, zarray_t will be merged into xzarray_t (this merge is already partially completed), which enables endpoints to participate in COW. Until the merge is complete (P0-0a), the immediate-cleanup path is used: handle_endpoint_error() calls zarray_del() as it does today. After the merge, the deferred cleanup flow applies: set ep->state = ENDPOINT_DISCONNECTED, close fd/SSL immediately, and clean up the slot at checkpoint via should_retain. The P0-8a step in the implementation phases enforces this ordering — it depends on P0-0a (zarray merge) being complete first.

Why immediate fd/SSL close is safe during COW: The fd and SSL pointer are on the endpoint object itself (not on the zarray slot). Setting ep->state = ENDPOINT_DISCONNECTED and closing ep->fd + ep->ssl are mutations on the endpoint struct, not on the zarray metadata. The zarray slot remains inuse=1 pointing to the endpoint — the deferred cleanup at thaw/merge will free the struct and set inuse=0. Since vget/vput check inuse on the active array first (which still has inuse=1), reads still find the disconnected endpoint, and business logic filters on ep->state != ENDPOINT_DISCONNECTED.

Connection to vtaskmgr: When endpoint_block_tasks(ep) sets ep->busy = 1, and later endpoint_unblock_tasks(ep) sets ep->busy = 0, this per-endpoint serialization is maintained regardless of whether the endpoint is in COW mode. The busy flag is on the endpoint object itself, not on the zarray slot.

4.10 Existing Code Alignment: Worker Unavailable State

The mark_worker_unavail() function already exists in worker.c:516-526. It currently:

1. Sets the XCE_WORKER_UNAVAIL flag on the worker
2. Runs AFTER_EVENT hooks (which will trigger the WAL write for WAL_WORKER_CTRL)
3. Does not remove the worker from the workermgr — the worker remains in the zarray with inuse=1

This matches the deferred cleanup design: after mark_worker_unavail(), the worker is in WORKER_UNAVAILABLE state and will be garbage-collected at the next checkpoint when worker_retain_s expires.

The new additions needed:

1. Set w->host.dead_since = time(NULL) in mark_worker_unavail() so the TTL check works
2. Add WORKER_HIDDEN state for admin-initiated removal (sets state, never deletes from zarray)
3. The handle_worker_leave() path currently calls mark_worker_unavail(c, w) and sets w->ep = NULL — this is correct and needs no change

---

Part 5: HA Failover — LSF-Style Priority Election

SUPERSEDED. This section (originally: a non-quorum, priority-list election model — "am I the highest-priority alive candidate?" — chosen instead of Raft consensus, gated by a mandatory ha.split_brain_risk_accepted: true config key) described a design that was later replaced. See: src/docs/persistence-and-ha-design.md findings and the closed specs raft-library, raft-ha-core-safety, ha-prevote, election-iomgr-refactor, raft-log-replication, raft-commit-safety, ha-term-safety, and ha-single-leader-role in .kiro/specs/ for what shipped instead — the actual implementation is a full quorum-based Raft protocol (src/lib/raft/raft.c, ha_has_write_quorum(), ha_start_election_async() in src/controller/ha.c). The split_brain_risk_accepted config key was never shipped.

---

Part 6: Fan-Out TCP Replication

6.1 Topology Choice: Fan-Out vs Chain

Factor	Fan-Out	Chain
Latency to all standbys	max(RTT_1..N) — single hop	RTT_1 + RTT_2 + ... RTT_N — cumulative
Primary load	N non-blocking sends	1 send (lowest load)
Failure isolation	Standby failure only affects that standby	One slow standby blocks the chain
Implementation	Simple	Must handle chain reordering
At XCE scale (~3 standbys)	Primary sends 3 copies — trivial	Marginal load savings

Decision: Fan-out. At 3 standbys, the primary does 3 non-blocking send() calls per WAL entry — negligible overhead. The latency advantage (single hop) and failure isolation are decisive.

6.2 Replicator Data Structure

/* MAX_STANDBYS and MAX_HA_CANDIDATES defined in include/config.h (see Appendix A.10) */

**Constant reconciliation**: The existing `src/include/client.h` defines `N_MAX_MASTERS = 3` for the `xce_client_config_t.masters[]` array. Both `MAX_HA_CANDIDATES = 4` and `MAX_STANDBYS = 3` are defined directly in `include/config.h` (see Out of Scope — `MAX_HA_CANDIDATES` reduced from 16 to 4, `MAX_STANDBYS` reduced from 15 to 3). `MAX_STANDBYS` is defined as a standalone constant (`= 3`) in `config.h`, NOT derived as `MAX_HA_CANDIDATES - 1` in `repl.h` — there is one source of truth. The deployment ceiling advisory (≤3 controllers) aligns with `N_MAX_MASTERS = 3` which was already chosen for that purpose. The Go CGo bindings see both constants — they must be clearly separated in header exports.

typedef struct {
    endpoint_t*  ep;            /* Full endpoint (SSL, FSM, outq) — uses iomgr/vtask pipeline */
    uint64_t     acked_lsn;    /* Last LSN this standby has ACKed. ACCESS RULE: on 64-bit
                                   platforms (LP64), acked_lsn is read/written atomically as a
                                   single 8-byte value. On ILP32 or any platform where uint64_t
                                   is not naturally atomic, this field MUST be accessed under
                                   repl.ack_lock. All write paths (repl_on_ack, standby
                                   disconnect tracking) and all read paths (repl_wait_ack
                                   predicate, WAL trim guard min-over-standbys) must use the
                                   same lock or use _Atomic uint64_t with acquire/release
                                   ordering. The recommended implementation is: use C11
                                   _Atomic uint64_t for acked_lsn with relaxed loads for
                                   advisory reads (lag monitoring, trim guard) and
                                   release/acquire for the ack signal path. Alternatively,
                                   gate ALL reads and writes through repl.ack_lock. */
    int          lagging;      /* Flag: standby is behind threshold */
    char         host[XCE_MAXNAME_LEN];
    uint32_t     port;
} standby_conn_t;

typedef struct {
    standby_conn_t  standbys[MAX_STANDBYS];
    int             nstandbys;
    mylock_t        lock;         /* Protects standbys array */
    uint64_t        pending_lsn;  /* LSN we're waiting ACK for (sync mode) */
    pthread_cond_t  ack_cond;     /* Signaled when ACK arrives */
    pthread_mutex_t ack_lock;     /* Protects ack_cond + pending_lsn. MUST be pthread_mutex_t
                                   * (not mylock_t) because pthread_cond_timedwait requires a
                                   * pthread_mutex_t* — same rationale as checkpoint_mutex. */
} replicator_t;

6.3 Replication Flow

Primary                              Standby-1    Standby-2
  │                                      │              │
  │  WAL write + fsync                   │              │
  │──────────────────────────────────────│              │
  │                                      │              │
  │  repl_broadcast(entry)               │              │
  │─── send(WAL_ENTRY) ─────────────────▶│              │
  │─── send(WAL_ENTRY) ─────────────────────────────────▶│
  │                                      │              │
  │  [for SYNC mode only]                │              │
  │  repl_wait_ack(target_lsn, timeout)  │              │
  │                                      │              │
  │◀── ACK(acked_lsn) ──────────────────│              │
  │◀── ACK(acked_lsn) ─────────────────────────────────│
  │                                      │              │
  │  Condition signal → resume           │              │
  │                                      │              │

6.4 Standby WAL Receiver

IMPLEMENTATION GUARD: The pseudocode below is a logical sketch of WAL entry processing on the standby. The actual implementation must use the endpoint/iomgr/vtask SSL pipeline (see Part 6.5 and Part 6.6). Do NOT implement standalone receive-loop threads with raw read()/write() on file descriptors. All controller connections — including replication — go through the iomgr event loop. A raw-socket implementation would bypass SSL enforcement, per-endpoint message ordering, and backpressure; any such implementation must be rejected in code review. The standby_recv_thread() function below is pseudocode only — replace with handle_msg() dispatch for OP_replEntry in the iomgr/vtask pipeline.

Serialization note: apply_wal_entry() on the standby (including xkvmgr_add() calls inside it) is effectively single-threaded. All OP_replEntry messages arrive over a single primary→standby endpoint; the iomgr/vtask pipeline's endpoint_block_tasks()/endpoint_unblock_tasks() mechanism ensures messages on a given endpoint are dispatched to one vcpu thread at a time and in order. Therefore apply_wal_entry() does not require additional locking beyond what the functions it calls already provide for the normal (non-COW) path.

The standby processes replication entries through its iomgr event loop:

/* On standby: receive WAL entries from primary
 * NOTE: This is PSEUDOCODE showing the logical processing steps.
 * The actual implementation uses iomgr/vtask/endpoints — see Part 6.5. */
static void* standby_recv_thread(void* arg)
{
    cluster_t* c = (cluster_t*)arg;
    ha_state_t* ha = &c->ha;

    for (;;) {
        wal_entry_hdr_t hdr;
        void* payload;

        /* Read WAL entry from primary (pseudocode — actual: iomgr recv via ha->primary_ep) */
        if (read(ha->primary_ep->fd, &hdr, sizeof(hdr)) != sizeof(hdr))
            break;  /* Primary connection lost */

        if (hdr.payload_len > MAX_WAL_PAYLOAD_BYTES) {
            ERROR("replication: oversized payload %u > MAX_WAL_PAYLOAD_BYTES", hdr.payload_len);
            break;  /* Treat as protocol error — drop connection */
        }
        /* Guard against malloc(0) for WAL_CHECKPOINT entries (payload_len=0).
         * For zero-payload entries, skip malloc entirely — no read or free needed.
         * For non-zero payloads, always check for OOM. */
        payload = NULL;
        if (hdr.payload_len > 0) {
            payload = malloc(hdr.payload_len);
            if (payload == NULL) {
                ERROR("replication: OOM allocating %u bytes for lsn=%lu",
                      hdr.payload_len, hdr.lsn);
                break;  /* Disconnect — cannot safely process entry */
            }
            if (read(ha->primary_ep->fd, payload, hdr.payload_len) != hdr.payload_len) {
                free(payload);
                break;
            }
        }

        /* Validate term — reject stale entries but allow multi-promotion jumps.
         * A primary term lower than standby's term means the standby has a
         * higher term (possibly from a previous self-promotion or a different
         * primary) — reject the entry, do not advance applied_lsn, and
         * continue processing. The standby should not follow a lower-term primary.
         * NOTE: ha->term is _Atomic uint64_t — all reads use atomic_load_explicit. */
        if (hdr.term < atomic_load_explicit(&ha->term, memory_order_acquire)) {
            free(payload);
            continue;
        }
        /* Accept any term >= ha->term, bounded by MAX_TERM_JUMP. During a
         * multi-promotion partition, the primary's term may have advanced by
         * more than 1 relative to the standby's ha->term. Instead of
         * disconnecting, update ha->term to match the primary's term — the
         * primary is the authoritative term leader.
         *
         * Security: cap term jumps at MAX_TERM_JUMP (default: 10000) to prevent
         * a corrupted or malicious packet from setting ha->term to UINT64_MAX,
         * which would permanently lock out legitimate primaries and cause
         * term wraparound (UINT64_MAX + 1 = 0) on next promotion. This is a
         * liveness invariant: the fencing token must never be driven to an
         * unusable value by a single bad packet.
         *
         * Accepted risk: MAX_TERM_JUMP prevents single-packet term exhaustion
         * but NOT cumulative drift. An attacker on the management LAN (which
         * is the trust boundary — see Part 5.9) could send repeated packets to
         * advance ha->term by MAX_TERM_JUMP per packet, eventually reaching
         * UINT64_MAX. Given the trusted-LAN security model (network-level ACLs
         * restrict access), this cumulative attack requires sustained access to
         * the management network and is accepted as an out-of-scope risk. mTLS
         * peer authentication (Part 5.9) would eliminate this attack surface. */
        #define MAX_TERM_JUMP 10000
        if (hdr.term > atomic_load_explicit(&ha->term, memory_order_acquire) + MAX_TERM_JUMP) {
            ERROR("replication: term too high: got %lu (max allowed: %lu) — "
                  "possible corrupted packet or tampering",
                  hdr.term, atomic_load_explicit(&ha->term, memory_order_acquire) + MAX_TERM_JUMP);
            free(payload);
            break;  /* Disconnect — term is implausible */
        }
        if (hdr.term > atomic_load_explicit(&ha->term, memory_order_acquire) + 1) {
            WARN("replication: term jump from %lu to %lu (multi-promotion partition heal)",
                 atomic_load_explicit(&ha->term, memory_order_acquire), hdr.term);
            atomic_store_explicit(&ha->term, hdr.term, memory_order_release);
        }

        /* Validate LSN sequence — entries must arrive in strictly increasing order.
         * A repeat means the primary re-sent without a term bump (unexpected).
         * A backward LSN is always an error.
         * A forward gap is expected for WAL_CHECKPOINT (type=100) entries, which are
         * local-only markers not replicated to standbys — they create gaps in the
         * LSN sequence from the standby's perspective. Therefore the check is
         * strictly increasing (hdr.lsn > applied_lsn), NOT strictly consecutive
         * (hdr.lsn == applied_lsn + 1). A missing-entry gap (network loss) is
         * detected by CRC chain validation, not LSN sequence — and a true gap
         * (more than one missing entry) indicates a network issue requiring
         * reconnection and catch-up. A single-entry gap (typical for a skipped
         * WAL_CHECKPOINT) is benign and must be accepted.
         *
         * Note: when applied_lsn == 0 (standby has not yet applied any entry),
         * both guards are skipped — the first replicated entry is accepted with
         * no LSN validation whatsoever. This is correct because (a) the first
         * entry arrives via catch-up which establishes a known starting LSN, and
         * (b) catch-up mode already performs CRC chain validation on the
         * transferred entries. Any gap in the initial stream is caught by the
         * catch-up protocol, not by the normal-mode LSN check. */
        uint64_t applied = atomic_load_explicit(&ha->applied_lsn, memory_order_acquire);
        if (applied != 0 && hdr.lsn <= applied) {
            ERROR("replication: LSN sequence error: got %lu, expected > %lu",
                  hdr.lsn, applied);
            free(payload);
            break;
        }
        if (applied != 0 && hdr.lsn > applied + 1) {
            /* LSN gap — distinguish single-skip (checkpoint marker) from multi-skip (data loss).
             *
             * Single-skip (hdr.lsn == applied + 2): most likely a skipped WAL_CHECKPOINT
             * (type=100) which is local-only and not replicated. Log WARN and continue —
             * the CRC chain validation below provides a second safety net.
             *
             * Multi-skip (hdr.lsn > applied + 2): indicates actual entry loss (network
             * packet loss, buffer overflow, or protocol bug). WAL_CHECKPOINT is a single
             * entry; two consecutive checkpoint skips would still produce a gap of 2 per
             * checkpoint, not a gap > 2 in a single replication step. Disconnect and
             * trigger catch-up — accepting multi-entry gaps silently would allow data loss
             * to go undetected even if the CRC chain restarts (e.g., at a segment boundary).
             */
            if (hdr.lsn == applied + 2) {
                WARN("replication: single LSN skip: expected %lu, got %lu (likely WAL_CHECKPOINT marker)",
                     applied + 1, hdr.lsn);
                /* continue — CRC chain below provides second validation */
            } else {
                ERROR("replication: multi-entry LSN gap: expected %lu, got %lu — possible entry loss, disconnecting for catch-up",
                      applied + 1, hdr.lsn);
                goto disconnect_for_catchup;
            }
        }

        /* Validate chained CRC — detect missing/reordered entries that LSN alone
         * cannot catch (since LSNs are "strictly increasing, not consecutive").
         * Each entry's prev_crc must match the CRC of the immediately preceding
         * received entry. On the first entry (applied_lsn == 0), prev_crc must
         * be WAL_CRC_NONE (established by catch-up protocol).
         * This mirrors the on-disk chained-CRC validation (Part 2.4) but operates
         * on the live replication stream. Without it, a reordered or replayed entry
         * with a valid per-entry CRC but wrong position in the chain would pass
         * the LSN check (since LSN gaps are expected) and corrupt the standby's
         * in-memory state and local WAL.
         *
         * On segment boundaries the primary sends entries with prev_crc ==
         * WAL_CRC_NONE as the first entry of the new segment. The standby treats
         * this as a valid chain break (same as on-disk recovery, Part 2.4). */
        if (atomic_load_explicit(&ha->applied_lsn, memory_order_acquire) == 0) {
            /* First entry after catch-up: prev_crc must be WAL_CRC_NONE */
            if (hdr.prev_crc != WAL_CRC_NONE) {
                ERROR("replication: first entry after catch-up has non-NULL prev_crc %u",
                      hdr.prev_crc);
                free(payload);
                break;
            }
        } else {
            /* Normal mode: prev_crc must match the CRC of the last received entry.
             * NOTE: WAL_CHECKPOINT entries are written to the primary's WAL but NOT
             * replicated. The primary maintains two separate CRC trackers: last_crc
             * (updated for every WAL entry including checkpoints) and
             * last_replicated_crc (updated only for replicated entries, never for
             * WAL_CHECKPOINT). The post-checkpoint replicated entry therefore carries
             * prev_crc = last_replicated_crc, which equals the CRC of the last
             * non-checkpoint WAL entry — not the CRC of the checkpoint entry itself.
             * Since ha->last_received_crc on the standby also tracks only replicated
             * entries, the chain remains unbroken across checkpoint boundaries.
             * See Part 2.3 for last_replicated_crc field documentation. */
            if (hdr.prev_crc != ha->last_received_crc && hdr.prev_crc != WAL_CRC_NONE) {
                ERROR("replication: chained CRC mismatch: expected %u, got %u, lsn %lu "
                      "(possible reordered/missing entry)",
                      ha->last_received_crc, hdr.prev_crc, hdr.lsn);
                free(payload);
                break;  /* Disconnect; primary may have diverged — reconnect + catch-up will resync */
            }
            /* WAL_CRC_NONE in normal mode (applied_lsn > 0): accepted as a valid
             * chain break. The code above does NOT disconnect when prev_crc ==
             * WAL_CRC_NONE — it passes the `if` condition silently. There are three
             * legitimate sources of WAL_CRC_NONE in normal mode:
             *
             * 1. Segment rotation: the primary MUST emit prev_crc = WAL_CRC_NONE for
             *    the first entry of each new WAL segment (per Part 2.4 writer-side
             *    obligation). This creates a chain break in the replication stream
             *    whenever a segment rotation occurs during live replication.
             *
             * 2. Post-promotion: when a standby promotes to primary, it resets
             *    last_replicated_crc = WAL_CRC_NONE (per Part 5.6 step 3). The first
             *    entry it broadcasts carries prev_crc = WAL_CRC_NONE. Already-connected
             *    standbys with applied_lsn > 0 accept this as a term-change chain break.
             *
             * 3. After catch-up: the first live entry after snapshot application is
             *    handled by the applied_lsn == 0 check above (not this branch).
             *
             * The term check earlier in this function guards against accepting entries
             * from a stale primary. WAL_CRC_NONE itself is not an error indicator —
             * the CRC chain exists to detect missing or reordered entries, and
             * WAL_CRC_NONE explicitly signals a legitimate restart of the chain. */
        }

        /* Validate CRC — detect corruption in replication path.
         * WAL_HDR_CRC_SIZE is defined in persist.h (Part 2.2) — covers all
         * header fields except crc and prev_crc.
         * A static_assert ensures sizeof(wal_entry_hdr_t) == 40 so this
         * computation remains correct regardless of field ordering. */
        uint32_t computed_crc = crc32(0, (void*)&hdr, WAL_HDR_CRC_SIZE);
        computed_crc = crc32(computed_crc, payload, hdr.payload_len);
        if (computed_crc != hdr.crc) {
            /* Corrupted entry — log and skip */
            ERROR("replication CRC mismatch: expected %u, got %u, lsn %lu",
                  hdr.crc, computed_crc, hdr.lsn);
            free(payload);
            continue;
        }

        /* Write to local WAL.
         * WAL_JOB_SUBMIT (SYNC) entries must be fsynced before ACK so the
         * primary's durability guarantee holds on the standby as well — if the
         * standby crashes before fsync, the primary's ACK to the client was
         * premature. All other entry types use async (fire-and-forget) writes.
         *
         * LSN handling: replicated entries use the PRIMARY's LSN and CRC — the
         * standby does NOT assign local LSNs. This is critical because:
         *   (a) After promotion, the former standby must serve catch-up to other
         *       standbys using the original LSN space (applied_lsn).
         *   (b) The WAL trim guard uses min(acked_lsn) across standbys — if
         *       LSNs were remapped locally, the trim guard would trim entries
         *       that a lagging standby still needs.
         *   (c) The chained CRC chain must match the primary's CRC values.
         *
         * The standby's persist_t.current_lsn must be updated to hdr.lsn after
         * each replicated write so the next locally-generated entry (after
         * promotion) gets hdr.lsn + 1. See persist_wal_replicated_entry() below.
         *
         * CRITICAL: Always check the return value of persist_wal_sync_append().
         * If the standby's local WAL write fails (disk full, I/O error), the
         * entry must NOT be applied, applied_lsn must NOT advance, and NO ACK
         * may be sent. A failed fsync with an ACK would violate the sync
         * replication durability guarantee — the primary would believe the
         * standby has the entry durably, but it is not on disk. */
        if (hdr.type == WAL_JOB_SUBMIT) {
            int rc = persist_wal_replicated_entry(&c->persist, &hdr, payload);
            if (rc != 0) {
                ERROR("replication: standby WAL write failed lsn=%lu rc=%d — dropping entry, not sending ACK",
                      hdr.lsn, rc);
                free(payload);
                break;  /* Disconnect; the primary will re-send on reconnect */
            }
        } else {
            if (persist_wal_replicated_entry(&c->persist, &hdr, payload) != 0)
                WARN("replication: async WAL write failed lsn=%lu — entry in page cache only", hdr.lsn);
        }

        /* Apply to in-memory state — MUST NOT fire AFTER_EVENT hooks.
         * apply_wal_entry() mutates state directly (same as persist_recover()).
         * If hooks fired here, the persistence hook would write another WAL
         * entry and send another replication message — an infinite loop.
         * Hooks are not registered on standbys (registered only on the
         * primary path after persist_recover() completes), so no bypass
         * flag is needed — the hook system is simply empty.
         *
         * Note: hdr.payload_len was already validated against MAX_WAL_PAYLOAD_BYTES
         * above (line 1210) before malloc. apply_wal_entry() applies the same bound
         * internally as defense-in-depth — it reads from the local WAL during
         * recovery (also user-controllable if disk is compromised).
         *
         * CRITICAL: Check the return value of apply_wal_entry(). If it fails
         * (OOM during deserialization, malformed payload), do NOT update
         * applied_lsn or last_received_crc, and do NOT send an ACK for sync
         * entries. The replication stream must be broken on apply failure —
         * the primary must re-establish with catch-up to ensure the standby's
         * state is consistent. */
        int rc = apply_wal_entry(c, &hdr, payload);
        if (rc != 0) {
            ERROR("replication: apply_wal_entry failed lsn=%lu rc=%d — breaking replication stream",
                  hdr.lsn, rc);
            free(payload);
            break;  /* Disconnect; catch-up will resync on reconnect */
        }

        atomic_store_explicit(&ha->applied_lsn, hdr.lsn, memory_order_release);
        ha->last_received_crc = hdr.crc;  /* Update chained CRC tracker */

        /* Send ACK for sync-replicated entries — only when not in catch-up mode.
         * During catch-up the standby is replaying historical entries; sending ACKs
         * then would prematurely signal readiness to the primary (see Part 6.6).
         * ha->in_catch_up is the STANDBY-LOCAL field in ha_state_t (set by the
         * standby when it enters catch-up, cleared when it sends OP_replCatchUpDone).
         * This is distinct from the PRIMARY-SIDE per-slot standby_conn_t.in_catch_up
         * (Appendix A.3), which is the primary's mirror of this field. */
        if (hdr.type == WAL_JOB_SUBMIT && !atomic_load_explicit(&ha->in_catch_up, memory_order_acquire)) {
            send_repl_ack(ha->primary_ep, hdr.lsn);
        }

        free(payload);
    }

    /* Primary connection lost — start monitor to check for promotion */
    ha_monitor_thread(c);
    return NULL;
}

6.5 Replication Uses the Endpoint/Iomgr/Vtask SSL Pipeline

All controller connections use SSL — the iomgr (netio.c) enforces this. Replication between controllers is no exception. The raw read/write calls in the standby receiver pseudocode above are simplified; the actual implementation must use the full endpoint/iomgr/vtask pipeline:

Standby                              Primary
  │                                     │
  │  iomgr_monitor(ep)                 │
  │  ─── endpoint established via SSL ──│
  │                                     │
  │  io_worker (epoll)                  │
  │    └─ read_fsm (SSL_read)           │
  │       └─ handle_msg()               │
  │          └─ vtask_handle_msg()      │
  │             └─ OP_replEntry          │
  │                └─ apply_wal_entry()  │
  │                └─ persist_wal_async()│
  │                                     │
  │  For SYNC: vtask sends OP_replAck   │
  │  └─ outq (per-endpoint serialized)  │
  │     └─ write_fsm (SSL_write)        │
  │       ──────────────────────────────▶│

Why the full pipeline?

1. SSL enforcement: All connections in netio.c go through SSL_new()/SSL_set_fd(). Creating a raw TCP socket for replication would bypass this requirement.
2. Per-endpoint serialization: The vtaskmgr endpoint_block_tasks()/endpoint_unblock_tasks() mechanism ensures that messages on a given endpoint are processed in order. This guarantees that WAL entries from the primary are applied in order on the standby.
3. Backpressure: If the standby is slow (e.g., during checkpoint serialization), the endpoint's outq fills up, and io_worker naturally applies backpressure via the write FSM. Raw TCP would require a separate flow-control mechanism.
4. Connection management: Reconnection, heartbeat, and error handling all reuse the existing handle_connection()/handle_endpoint_error() code path.

standby_conn_t update: Replace int fd with endpoint_t* ep — the standby connection is a full endpoint managed by iomgr, not a raw file descriptor.

typedef struct {
    endpoint_t*  ep;            /* Full endpoint (SSL, FSM, outq) — not raw fd */
    uint64_t     acked_lsn;    /* Last LSN this standby has ACKed */
    int          lagging;      /* Flag: standby is behind threshold */
    char         host[XCE_MAXNAME_LEN];
    uint32_t     port;
} standby_conn_t;

The primary initiates the replication connection as a client (using iomgr's outgoing connection support), and the standby accepts it via its own iomgr. The OP_replEntry and OP_replAck opcodes flow through the same SSL/endpoint/vtask pipeline as all other controller messages.

The standby_recv_thread pseudocode above would instead be handled by the iomgr event loop: the standby's handle_msg() dispatches OP_replEntry to vtask_handle_msg(), which applies the WAL entry and sends OP_replAck back via the endpoint's outq.

6.6 OP_replAck Must Bypass vtaskmgr (ACK Deadlock Prevention)

OP_replAck is the only opcode that must be handled directly in the io_worker read callback (handle_msg()) instead of being dispatched to vtaskmgr. All other opcodes go through vtaskmgr.

Why: When sync replication is active, vtask threads executing handle_jobsubmit() block inside repl_wait_ack() waiting for an ACK. If OP_replAck were also dispatched via vtaskmgr, the incoming ACK message would sit in the vtask queue — but no vcpu thread would be free to process it (they are all blocked waiting for ACKs). This is a classic deadlock.

Implementation: handle_msg() must check the opcode before calling vtask_handle_msg():

/* handle_msg is the io_worker read callback. It receives a protocol message,
 * NOT a WAL entry header. The first field is the OPCode (see Part 5.10 for the
 * opcode enum: OP_replEntry=23, OP_replAck=24, OP_redirect=25, OP_heartbeat=26).
 * For OP_replEntry, the payload is a WAL entry (wal_entry_hdr_t + data).
 * Do not confuse the protocol opcode with wal_entry_type values (Part 2.3). */
void handle_msg(cluster_t* c, endpoint_t* ep, msg_hdr_t* hdr, void* payload) {
    if (hdr->opcode == OP_replAck) {
        /* Handle directly in io_worker — do NOT dispatch to vtaskmgr */
        repl_on_ack(&c->ha.repl, decoded_lsn);
        return;
    }
    vtask_handle_msg(c, ep, hdr, payload);
}

repl_on_ack() is safe to call from io_worker: it only updates acked_lsn and calls pthread_cond_signal() under repl.ack_lock, with no vtaskmgr interaction.

Standby catch-up after reconnection: When a standby reconnects to the primary (either after a network disruption or on initial startup), it must tell the primary its last applied_lsn so the primary can stream missing WAL entries. The catch-up protocol is:

1. Standby sends OP_replJoin with its applied_lsn and term
2. Primary validates the peer: The primary verifies that the peer's host:port appears in its candidate list (from --peers CLI flag). An OP_replJoin from an unknown endpoint is rejected with an error reply.
3. Primary checks if the standby's applied_lsn is still within available WAL segments (not yet trimmed by a checkpoint). If yes, stream entries from applied_lsn + 1 forward via OP_replEntry messages on the existing SSL endpoint pipeline
4. If the standby's applied_lsn is too old — i.e., applied_lsn < oldest_surviving_segment_start_lsn (the WAL segments needed for catch-up have been trimmed) — the primary transfers the latest snapshot via the same SSL endpoint pipeline using a new OP_replSnapshot opcode. The snapshot is chunked at MAX_WAL_PAYLOAD_BYTES per message: the first chunk includes a snapshot header { snapshot_lsn, total_size, chunk_seq=0, final=false }, subsequent chunks carry { chunk_seq, data }, and the last chunk sets final=true and includes a CRC32 of the full snapshot payload. The standby reassembles chunks into a buffer, verifies the CRC32, then applies the snapshot to its kvmgrs. Snapshot size validation on standby receive: The total_size field in the first chunk must not exceed MAX_SNAPSHOT_BYTES (default: 256 MB). If total_size > MAX_SNAPSHOT_BYTES, the standby logs an ERROR and tears down the replication connection — a compromised or buggy primary could otherwise exhaust the standby's memory by sending an unbounded snapshot. This mirrors the local recovery path's snapshot size validation (Part 2.5 step 2b). After snapshot application, the primary streams WAL entries from snapshot_lsn + 1 forward. No separate file transfer channel or scp is required — all data flows through the established SSL endpoint.
5. If the standby's term is higher than the primary's, the primary has stale data — the standby should not follow this primary (fencing)

Replication during catch-up: While the primary is streaming historical WAL entries (WAL replay or snapshot + WAL) to a catching-up standby, the primary pauses live OP_replEntry broadcasts to that standby until the standby sends OP_replCatchUpDone. This is the definitive design choice: live entries are NOT buffered in the catching-up standby's sendq during catch-up. Buffering live entries during a 256 MB snapshot transfer (10–30 seconds at disk speed) at 1000 submits/sec would require 10–30 MB of in-flight WAL in the sendq — unbounded memory growth with no application-level control. The simpler and safer design: the catching-up standby is excluded from repl_wait_ack() via in_catch_up gating until it sends OP_replCatchUpDone. After catch-up, the standby requests any WAL entries it missed during the catch-up window via a second WAL-only catch-up phase (the primary's WAL covers this gap since WAL trim is held back by the standby's join LSN). The standby does not send OP_replAck during catch-up — only after fully caught up and processing live entries in normal mode.

Catch-up receive ordering: (1) OP_replSnapshot chunks arrive first and must be fully received and CRC-verified before any new WAL entries are applied to the kvmgrs. (2) The primary must NOT send live OP_replEntry messages to the catching-up standby until the snapshot transfer is complete — the standby sends OP_replCatchUpDone (or equivalent signal) after successfully loading the snapshot. (3) After snapshot application, the primary streams WAL entries from snapshot_lsn + 1 forward, and the standby applies them in strict LSN order. Only after reaching the primary's committed_lsn does the standby begin sending OP_replAck and transition to normal replication. This ensures the standby's state is self-consistent at each phase boundary: snapshot-loaded → historical-WAL-replayed → caught-up.

Snapshot catch-up flow control: The OP_replSnapshot transfer sends the full snapshot in chunks of MAX_WAL_PAYLOAD_BYTES through the SSL endpoint pipeline. The primary sends chunks as fast as the endpoint pipeline allows, and the standby must receive, buffer, and reassemble all chunks before applying the snapshot. The existing iomgr backpressure mechanism (endpoint write FSM, outq buffering, SSL-level flow control via SSL_write partial writes) provides implicit flow control: if the standby's receive path is slow, the primary's endpoint outq fills, and subsequent SSL_write calls block or return partial writes, naturally throttling the sender. No additional application-level flow control is needed beyond the iomgr/SSL pipeline's built-in backpressure. io_worker pool sizing caveat: if the io_worker pool is too small, a slow snapshot transfer (256 MB at disk speed, 10–30 seconds) can starve the OP_replAck path for healthy standbys — sync replication ACKs may experience queue delays that exceed sync_timeout_ms, triggering spurious degraded-mode for job submits that should have been durably replicated. Operators should size the io_worker pool to at least MAX_STANDBYS + 2 workers (default: MAX_STANDBYS=3 catch-up endpoints + scheduler + 1 reserve). Add a startup log at INFO level that reports the configured io_worker count and the recommended minimum. A P3 improvement would be to rate-limit snapshot chunk sends per-endpoint using a configurable snapshot_bandwidth_limit_kbps parameter, leaving headroom for sync replication ACK traffic. However, operators should be aware that a snapshot transfer (up to MAX_SNAPSHOT_BYTES = 256 MB) temporarily increases the standby's memory usage for reassembly. The standby allocates a receive buffer of total_size bytes (validated against MAX_SNAPSHOT_BYTES at first chunk, per Part 6.6) before receiving subsequent chunks — this memory is freed after snapshot application. CRITICAL: validate total_size <= MAX_SNAPSHOT_BYTES BEFORE calling malloc(total_size). Do NOT allocate the buffer and then check — the check is the first operation on first-chunk receipt. Aggregate memory budget: With MAX_STANDBYS = 3, up to 3 standbys may catch up concurrently on the primary side. Each primary-side sendq for an in-flight snapshot transfer holds at most one pending chunk (MAX_WAL_PAYLOAD_BYTES ≈ 1 MB), so the primary's total per-catch-up memory is negligible. On the standby side, only one standby receives a snapshot at a time (a standby can only be catching up from one primary), so the worst-case standby-side allocation is a single total_size ≤ MAX_SNAPSHOT_BYTES buffer. The 768 MB = 3 × 256 MB scenario (all standbys catching up simultaneously) can only occur on the primary if it were somehow buffering outbound snapshots for all standbys at once — but the primary streams each snapshot through an already-established SSL endpoint, holding at most MAX_WAL_PAYLOAD_BYTES per chunk in the endpoint outq. Therefore the realistic aggregate memory bound is MAX_SNAPSHOT_BYTES per standby (one reassembly buffer) plus MAX_WAL_PAYLOAD_BYTES × MAX_STANDBYS on the primary for in-flight chunks.

Catch-up trigger: When a standby reconnects (or initially joins), it includes its applied_lsn in the heartbeat to the primary. The primary compares this against its available WAL segments and streams missing entries from applied_lsn + 1 forward. If the standby's applied_lsn is older than the primary's oldest surviving WAL segment, the primary initiates snapshot transfer (see OP_replSnapshot framing above) before streaming WAL entries.

MAX_WAL_PAYLOAD_BYTES: The maximum payload size per WAL entry (and per OP_replEntry message) is 1 MB (1,048,576 bytes). This bounds the memory allocation in the replication receive path (line payload = malloc(hdr.payload_len) above) to a safe maximum. The largest expected payload is a job submit with a large environment (~64 KB typical, ~512 KB pathological). The 1 MB cap provides 2× headroom over the pathological case while preventing unbounded allocation from a corrupted or malicious entry. Defined in persist.h as MAX_WAL_PAYLOAD_BYTES (see Appendix A.1). A #ifndef include guard allows test override.

Cap'n Proto traversal limit: apply_wal_entry() must set a Cap'n Proto traversal limit when constructing the reader for WAL payloads. Set capnp::ReaderOptions.traversalLimitInWords proportional to MAX_WAL_PAYLOAD_BYTES / 8 (the word count for the maximum payload). This prevents O(n²) or deeper traversal amplification from a crafted payload that is within the 1 MB byte limit but triggers excessive Cap'n Proto structural nesting. This is standard Cap'n Proto hardening practice and is a one-line change per deserialize call site.

This catch-up mechanism is analogous to etcd's snapshot + WAL catch-up, simplified for XCE's single-writer model. The OP_replJoin, OP_replSnapshot opcodes and catch-up protocol will be added as part of P2-5 (repl.c — standby catch-up implementation; distinct from P2.5 which is the libxce smart discovery phase).

---

Part 7: Mixed Sync/Async Replication Protocol

SUPERSEDED. This section (originally: single-standby-ACK durability — "wait for at least one standby ACK" — with a degraded mode that ACKs the client on local WAL alone if zero standbys ACK within sync_timeout_ms) described a design that was later replaced. See: src/docs/persistence-and-ha-design.md findings and the closed specs ha-commit-engine, ha-commit-durability, ha-commit-barrier-rollout, and ha-consistency-invariants in .kiro/specs/ for what shipped instead — the commit engine (src/controller/commit.c, repl_wait_majority_ack() in src/controller/repl.c) requires majority-quorum ACK and removed the zero-ACK-succeeds short-circuit.

---

Part 8: Hook-Driven Architecture

8.1 Existing Hook Pattern

XCE's hook system already integrates with the controller. Examples from apiserver.c and sched.c:

/* apiserver.c — existing pattern */
static int apiserver_job_hook_fun(int type, int situation, int detail,
                                  void* data, va_list ap)
{
    cluster_t* c = va_arg(ap, cluster_t*);
    /* ... handle job events ... */
}

/* Registration in apiserver_init() */
xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
             (hook_fn_t)apiserver_job_hook_fun, NULL);

8.2 Persistence Hook Functions

Persistence follows the exact same pattern:

/* persist.c — new hook functions */
int persist_job_hook_fun(int type, int situation, int detail,
                         void* data, va_list ap)
{
    cluster_t* c;
    persist_t* p;

    /* Only act on AFTER_EVENT — state mutation is complete */
    if (situation != HOOK_AFTER_EVENT)
        return 0;

    c = va_arg(ap, cluster_t*);
    p = &c->persist;

    switch (detail) {
    case HOOK_DETAIL_JOB_NEW:
        /* SYNC: WAL write + replicate + wait ACK */
        return persist_wal_sync_append(p, WAL_JOB_SUBMIT,
                                        data, compute_payload_len(data));
    case HOOK_DETAIL_JOB_START:
        return persist_wal_async(p, WAL_JOB_START, data, ...);
    case HOOK_DETAIL_JOB_EXIT:
        return persist_wal_async(p, WAL_JOB_FINISH, data, ...);
    case HOOK_DETAIL_JOB_FAIL:
        return persist_wal_async(p, WAL_JOB_FAIL, data, ...);
    /* ... etc ... */
    default:
        return 0;
    }
}

int persist_queue_hook_fun(int type, int situation, int detail,
                           void* data, va_list ap)
{
    /* ... same pattern for queue events ... */
}

int persist_worker_hook_fun(int type, int situation, int detail,
                            void* data, va_list ap)
{
    /* ... same pattern for worker events ... */
}

8.3 Critical Change: Reply After Hooks

Current code in handle_jobsubmit() (job.c:455-525):

/* CURRENT — reply BEFORE hooks */
xce_run_hooks(BEFORE_EVENT, HOOK_DETAIL_JOB_NEW, c, submit);
jobid = job_submit(c, submit);

reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = jobid;
xce_reply(ep, c->myhost, &reply);       /* ← ACK before hooks */

/* Collect jobs into list... */
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist);  /* ← persistence happens here */

Modified code — move xce_reply() after AFTER_EVENT:

/* MODIFIED — reply AFTER hooks */
xce_run_hooks(BEFORE_EVENT, HOOK_DETAIL_JOB_NEW, c, submit);
jobid = job_submit(c, submit);

if (jobid == 0) {
    reply.status = -errno;
    reply.data_which = RequestReply_data_nodata;
    xce_reply(ep, c->myhost, &reply);   /* Error reply is immediate */
    return -errno;
}

/* Collect jobs into list... */
/* AFTER_EVENT hook errors are tolerated per XCE hook design: pre-hook (BEFORE_EVENT)
 * failures abort the operation; post-hook (AFTER_EVENT) failures are logged but do not
 * require transaction rollback.
 *
 * WAL result is communicated via a PER-REQUEST stack variable passed via xce_run_hooks
 * extra argument. Using a shared persist_t field would race: multiple vcpu threads can
 * execute handle_jobsubmit() concurrently (one per API server connection), and each
 * would overwrite the shared field with its own result, causing thread A to read
 * thread B's -EIO and reject a durable job, or vice versa.
 *
 * Implementation: persist_job_hook_fun() receives &sync_rc via va_list as the last
 * argument (after cluster_t* c and the jlist pointer), writes the WAL result there,
 * and returns 0. handle_jobsubmit() reads sync_rc after hooks return. */
int sync_rc = 0;    /* per-request WAL result — not shared, no race */
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist, &sync_rc);
/* ↑ SYNC replication happens inside the persistence hook.
 * The hook always returns 0 (AFTER_EVENT errors tolerated at the hook level).
 * WAL write result is in sync_rc (stack-allocated, this vcpu thread only):
 *   0           — WAL written + fsynced (+ standby ACKed if standbys connected)
 *  -EIO         — local WAL write failed (disk full, permanent)
 *  -ETIMEDOUT   — wrote locally but no standby ACKed within sync_timeout_ms */
if (sync_rc == -EIO) {
    /* Permanent WAL failure — do not ACK to client.
     * The job was inserted into kvmgr by job_submit() but is not durable.
     * CRITICAL: mark the job terminal so checkpoint GC cleans it up. Without
     * this, the job survives every GC pass (job_is_terminal() returns false)
     * and accumulates as a phantom entry the client was told doesn't exist.
     *   job_mark_wal_error_exit(c, jobid);   <- sets JOB_STAT_EXIT + finish_time=now
     * This makes it eligible for GC after finished_retain_s and prevents
     * the scheduler from dispatching work to a nonexistent job. */
    job_mark_wal_error_exit(c, jobid);   /* see job.h — sets stat=JOB_STAT_EXIT */
    reply.status = -EIO;
    reply.data_which = RequestReply_data_nodata;
    xce_reply(ep, c->myhost, &reply);
    ERROR("job submit: WAL write failed with -EIO — job %lu marked terminal for GC", jobid);
    return -EIO;
}
if (sync_rc == -ETIMEDOUT) {
    /* Locally durable but no standby ACK — degraded durability. Accept. */
    WARN("job submit: no standby ACK within timeout — accepting with degraded durability");
}

/* Reply AFTER hooks — client ACK implies durability */
reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = jobid;
xce_reply(ep, c->myhost, &reply);

Why this works: AFTER_EVENT hook failures are tolerated (no transaction rollback needed). The persistence hook uses the per-request stack variable sync_rc (passed via va_list as an extra argument to xce_run_hooks()) to communicate WAL write results back to handle_jobsubmit() without relying on hook return values. Because sync_rc is stack-allocated in handle_jobsubmit(), it is thread-private — concurrent vcpu threads each have their own sync_rc with no sharing or race conditions. This decouples the sync replication result from the hook system's return-value semantics.

No reply reordering concern: The controller processes requests from each API server connection one at a time — a request is fully handled (including hooks and xce_reply()) before the next request on the same connection is processed. SYNC replication adds latency to the AFTER_EVENT hook, which delays the xce_reply() for that specific request, but it does not cause out-of-order replies because the next request on that connection doesn't begin processing until the current one completes. Replies to different API servers may have different latencies, but each API server sees its own replies in request order.

Lock safety analysis: The current handle_jobsubmit() acquires lock(c->jobmgr, RTTI_LOCK_READ) for job lookup, then operates on the per-job object. No kvmgr write lock is held during the hook call or xce_reply() for job submit. The change to move xce_reply() after hooks is therefore safe (note: handle_worker_join() in join.c:58-69 is the exception — it holds the workermgr WRITE lock during AFTER_EVENT hooks, but this is acceptable because worker join uses async replication only):

1. job_submit() takes the jobmgr READ lock for lookup, then per-object lock for modification — no change here
2. AFTER_EVENT hooks (including persist_wal_sync_append()) acquire their own locks internally (persist.lock for WAL, repl.lock for replication) — no nesting with jobmgr
3. xce_reply() writes to the endpoint's outq — no kvmgr lock interaction
4. handle_worker_join() in join.c:58-69 takes the workermgr WRITE lock for add_worker(). Worker join is always async-replicated (never blocks on repl_wait_ack), so holding the write lock during the hook call is safe — the hook only does a non-blocking WAL write and fire-and-forget replication send. The xce_reply() for worker join is also inside the write lock (at line 65), which risks blocking on endpoint outq backpressure under extreme conditions. add_worker() is infrequent enough that this is acceptable in practice, but if endpoint backpressure becomes a concern, xce_reply() should be moved after unlock(c->workermgr).

Enforcement: The persistence hook for HOOK_DETAIL_WORKER_JOIN must use persist_wal_async() — never persist_wal_sync_append(). To make this self-documenting and prevent a future refactor from accidentally switching to sync replication while the WRITE lock is held, add a persist_wal_async_only() wrapper that asserts type != WAL_JOB_SUBMIT at runtime (under persist.lock), or log a CRITICAL error and abort if a sync-type entry is attempted through the worker path. This is a belt-and-suspenders guard: the hook mapping in Part 2.3 already maps worker events to async-only WAL types, but the runtime assertion prevents regression.

Replication scope: Sync replication (repl_wait_ack) only triggers for WAL_JOB_SUBMIT entries, which originate from API server connections. The controller only handles connections from workers, scheduler, and API servers — never end users. At 1–3 API server connections, the vcpu pool is not at risk of exhaustion from blocking repl_wait_ack() calls.

Array job sync replication latency bound (CRITICAL): Array jobs submit multiple sub-jobs in a single handle_jobsubmit() call. With AFTER_EVENT firing once per sub-job (the existing hook dispatch model), 10,000 sub-jobs each triggering repl_wait_ack() would block the vcpu for 500 seconds. Mitigation — accumulation buffer: For array submits (submit->array_size > 1), handle_jobsubmit() allocates a persist_batch_ctx_t on the heap and passes it as a second extra va_list argument (after &sync_rc), always NULL for scalar submits:

/* Scalar submit: sync_rc only (batch_ctx = NULL) */
int sync_rc = 0;
persist_batch_ctx_t* batch_ctx = NULL;
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist, &sync_rc, batch_ctx);

/* Array submit: sync_rc + batch_ctx (non-NULL) */
int sync_rc = 0;
persist_batch_ctx_t* batch_ctx = persist_batch_ctx_alloc(submit->array_size, 100 /* batch_size */);
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist, &sync_rc, batch_ctx);
/* ... iterate sub-jobs, each fires AFTER_EVENT ... */
free(batch_ctx);

The hook reads both arguments: int* sync_rc_p = va_arg(ap, int*); persist_batch_ctx_t* batch = va_arg(ap, persist_batch_ctx_t*);. When batch != NULL, the hook accumulates WAL entries and flushes every 100 sub-jobs. The hook sets batch->sub_jobs_done++ and writes the final sync_rc into *sync_rc_p on the last sub-job (batch->sub_jobs_done == batch->total_sub_jobs). This bounds latency to (array_size / 100) × 50ms — 5 seconds for a 10K array. The accumulation buffer is per-request, not shared, so concurrent submits are fully independent.

Async hook error handling: For async-replicated events (all events except WAL_JOB_SUBMIT), persist_wal_async() may return a non-zero error code. These errors fall into two categories: (1) -EIO (local WAL write failure — disk full, I/O error) and (2) other errors. For all async events, the caller (e.g., handle_worker_join(), on_job_start(), on_job_finish()) must follow this error handling convention:

• -EIO (local WAL write failure): Log an ERROR with the event type, LSN (if available), and error code. The in-memory state mutation has already completed — the entity exists in the kvmgr but is not durable on local WAL. The error is not recoverable without operator intervention (disk full or permanent I/O failure). Do NOT undo the in-memory mutation — it is the source of truth for the scheduler and API queries. Do NOT reply to the client with an error for async events (they have no reply expected from the persistence path — xce_reply() for async events has already happened before the hook).
• Async replication failure (send to standby failed): Log a WARN. The entry is locally durable (WAL + fsync). The standby will receive it on reconnection catch-up. No action needed — this is the expected behavior for async replication.
• Summary: Async hook callers should log errors but never undo in-memory state or block the vcpu thread. The self-healing property of async events (workers re-notify on reconnect) means that a missed async WAL entry is recoverable without manual intervention unless the local WAL itself has failed.

8.4 Registration in cluster_init()

Hook registration is conditional on role and happens after persist_recover() completes. This resolves two constraints:

• Part 2.5: "hooks not registered until after persist_recover() completes" — recovery replay must not trigger WAL writes
• Part 6.4: apply_wal_entry() on standbys must not fire hooks — standbys must never register persistence hooks

/* In cluster_init(), after xce_install_hooks(): */

if (persist_config_enabled(&c->myconf->cluster_config->persist)) {
    rc = persist_init(&c->persist, &c->myconf->cluster_config->persist, c);
    if (rc < 0) return rc;

    /* Step 1: Recover BEFORE registering hooks.
       persist_recover() replays WAL via the same mutation functions
       (job_submit, mark_worker_unavail, etc.) but hooks are not yet
       registered, so no WAL writes occur during replay. */
    rc = persist_recover(&c->persist, c);
    if (rc < 0) return rc;

    /* Step 2: Register persistence hooks ONLY on primary path.
       Standbys apply WAL entries via apply_wal_entry() — hooks are
       not registered on standbys, so no infinite loop occurs.
       Hook registration happens here for primary, or in
       ha_promote_to_primary() step 5 when a standby promotes. */
    if ((ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_PRIMARY) {
        xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
                     (hook_fn_t)persist_job_hook_fun, NULL);
        xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_QUEUE,
                     (hook_fn_t)persist_queue_hook_fun, NULL);
        xce_add_hook(&(c->hooksys), XCE_CONTROLLER_HOOK_WORKER,
                     (hook_fn_t)persist_worker_hook_fun, NULL);
    }
    /* Standbys: no hooks registered here. apply_wal_entry() mutates state
       directly — persistence hooks are registered only after cluster recovery
       completes, and only on the primary path. */
}

---

Part 9: Worker Connection Strategy

9.1 Current Design: Worker-to-Controller Long TCP

Workers maintain a persistent TCP connection to the controller. This provides:

1. Fast failure detection: If a worker dies, the controller gets an immediate TCP RST — no polling needed
2. Easy push from controller: The controller can push job dispatches, config updates, and heartbeats over the existing connection without initiating a new one
3. Bidirectional heartbeat: Both sides can detect liveness via the same connection

The main concern is the OS connection limit: each TCP connection consumes a file descriptor and kernel socket buffer. However, for typical HPC cluster scales (100s–low 1000s of workers), this is a configuration issue, not an architectural one:

# /etc/sysctl.conf — sufficient for a 2000-node cluster
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
fs.file-max = 200000

# In the controller process startup:
# ulimit -n 65535

Linux handles tens of thousands of idle TCP connections with minimal overhead. The practical limit is ~65K connections per IP/port tuple, which is far beyond any single XCE cluster.

9.2 Decision: Workers Connect Directly to Primary

Three options were evaluated:

Option A — Direct-only: Workers connect to primary only.

Option B — Standby as full proxy: Workers connect to any candidate; standbys forward all traffic (both directions) to primary.

Option C — Standby as discovery + redirect: Workers can connect to any candidate at startup; standbys send a redirect response pointing to the primary; worker reconnects to primary.

Aspect	A (Direct-only)	B (Full proxy)	C (Redirect)
Normal latency	1 hop	2 hops (doubles)	1 hop
Complexity	Minimal	High — bidirectional relay, reply correlation, read-after-write consistency	Low — just a redirect opcode
Failover reconnection	Workers must discover new primary	Workers on promoting standby: zero cost; others: must reconnect	Workers reconnect with libxce auto-retry
Worker startup	Must know primary	Pick any candidate	Pick any candidate, get redirected
Read-after-write	No issue	Critical issue — standby forwards write to primary, but its own state is stale until replicated back	No issue — all writes go to primary directly
Connection load	Primary only	Distributed	Primary only (discovery is transient)

Decision: Option C (Redirect). It gives the "worker doesn't need to know the primary" benefit at startup, without paying the 2-hop latency and read-after-write consistency cost on every message. The redirect is a one-time cost during connection setup — amortized to zero over a long-lived connection.

This pattern is used by etcd (proxy mode), Redis Sentinel, and MongoDB replica sets.

9.3 Full Communication Topology

                     ┌─────────────────┐
                     │  Controller     │
                     │  Candidates     │
                     │  (--peers flag) │
                     └────────┬────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
        ┌──────────┐   ┌──────────┐   ┌──────────┐
        │ ctrl-1   │   │ ctrl-2   │   │ ctrl-3   │
        │ PRIMARY  │   │ STANDBY  │   │ STANDBY  │
        │ P=0      │   │ P=1      │   │ P=2      │
        └────┬─────┘   └────┬─────┘   └────┬─────┘
             │              │              │
    ┌────────┼──────┐       │              │
    │        │      │       │              │
    ▼        ▼      ▼       ▼              ▼
  Worker   Worker  Worker  (accepts discovery
   W1       W2      W3     connections only —
                            sends REDIRECT)
                                │
                         New Worker W4
                         startup flow:
                         1. Try ctrl-2 (random)
                         2. Get REDIRECT → ctrl-1
                         3. Connect to ctrl-1

9.4 Standby Redirect Handling

When a standby receives a join request from a non-controller peer (worker, scheduler, API server), it sends a redirect response instead of accepting the connection:

/* In handle_join() — modified for HA redirect */
int handle_join(cluster_t* c, endpoint_t* ep, xce_request_t* request)
{
    xce_peerinfo_t* peer = request->body.joinmsg->peer;
    int role = peer->role;

    /* If I'm a standby and the peer is a worker/scheduler/apiserver,
       redirect them to the primary */
    if ((ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire) == CTRL_STANDBY) {
        switch (role) {
        case HostRole_worker:
        case HostRole_scheduler:
        case HostRole_apiserver: {
            xce_reply_t reply;
            init_reply_by_req(&reply, request);
            /* Guard: primary_host is zero-initialized on startup and only populated
             * after the standby receives the first heartbeat from the primary.
             * If we have never seen a primary (cold start, all controllers starting
             * simultaneously), return E_XCE_NOMASTER instead of a garbage redirect. */
            if (c->ha.primary_host[0] == '\0' || c->ha.primary_port == 0) {
                reply.status = E_XCE_NOMASTER;
                xce_reply(ep, c->myhost, &reply);
                return 0;
            }
            reply.status     = E_XCE_REDIRECT;
            reply.data_which = RequestReply_data_redirect;
            reply.data.redirect.host = c->ha.primary_host;
            reply.data.redirect.port = c->ha.primary_port;
            xce_reply(ep, c->myhost, &reply);
            return 0;
        }
        default:
            break;
        }
    }

    /* Normal join processing (I am primary) */
    /* ... existing code ... */
}

Redirect authentication: All controller connections use SSL enforced by iomgr (netio.c), which means a redirect response can only come from a peer that has completed the SSL handshake with a valid certificate. A rogue endpoint that has not established an SSL connection cannot inject redirect responses. Additionally, the RedirectInfo contains only host/port of a known candidate — the receiving peer validates that the redirect target is in its candidate list (from --peers CLI flag) before following it.

9.5 Thundering Herd Mitigation on Failover

When the primary dies, all workers connected to it get TCP RST simultaneously. If they all reconnect to the new primary at once, the listen backlog and accept loop could be overwhelmed.

Mitigation: Exponential backoff with random jitter in libxce (detailed in Part 10). This spreads worker reconnections over a time window instead of a single instant.

For 2000 workers with 100ms base backoff + 50% jitter:

• First wave: ~0-150ms — ~500 workers reconnect
• Second wave: ~150-300ms — ~300 workers
• Third wave: ~300-600ms — remaining workers
• Total reconnection window: ~600ms — well within the new primary's capacity

9.6 Worker Disconnect — Deferring to Terminal State

When a worker disconnects, instead of immediately removing it from data structures, the controller transitions it to WORKER_UNAVAILABLE:

/* In handle_worker_disconnect() — modified for deferred cleanup */
void handle_worker_disconnect(cluster_t* c, worker_t* w) {
    /* Instead of: destroy(w); kvmgr_del(...); */
    w->state = WORKER_UNAVAILABLE;
    w->host.dead_since = time(NULL);

    /* Worker remains in zarray (inuse=1) and kvtab.
       Scheduler already ignores UNAVAILABLE workers.
       Actual cleanup happens at next checkpoint if
       worker_retain_s has expired. */
}

9.7 Redirect Cap'n Proto Schema

/* New struct for redirect response */
struct RedirectInfo $C.mapname("xce_redirect_t") {
    host @0 :Text $C.mapname("host");
    port @1 :UInt32 $C.mapname("port");
}

/* In RequestReply union: */
redirect @8 :RedirectInfo $C.mapname("redirect");

/* New error code in common/ */
#define E_XCE_REDIRECT (-100)  /* Not an error — redirect instruction */

---

Part 10: libxce Smart Discovery

10.1 Problem

Currently, workers and other components are configured with a single controller address. With HA, they need to:

1. Discover which candidate is the current primary
2. Follow redirects from standbys
3. Reconnect automatically on failover with appropriate backoff
4. Verify they're talking to the primary after connecting

10.2 Configuration API

/* libxce — xce_cluster.h */

typedef struct {
    char     host[XCE_MAXNAME_LEN];
    uint32_t port;
} xce_candidate_t;

typedef struct {
    char     ca_cert_path[PATH_MAX];  /* CA cert path for SSL_VERIFY_PEER (empty = skip) */
} xce_tls_config_t;

typedef struct {
    xce_candidate_t  candidates[MAX_HA_CANDIDATES];
    int              ncandidates;           /* Number of candidates configured */
    uint32_t         connect_timeout_ms;    /* Per-candidate connect timeout (default 2000) */
    uint32_t         retry_base_ms;         /* Backoff base (default 100) */
    uint32_t         retry_max_ms;          /* Backoff cap (default 5000) */
    int              max_retries;           /* Max reconnection attempts (default 10) */
    xce_tls_config_t tls;                  /* TLS options — CA cert path for SSL_VERIFY_PEER.
                                            * Populated from --ca-cert CLI flag (P2.5-5/P2.5-6). */
} xce_cluster_config_t;

/* Connect to the cluster — returns SSL* (not raw fd) connected to primary.
 * The caller must NOT call close() on the returned pointer — use
 * xce_disconnect_cluster() to tear down the SSL connection and free resources.
 * All controller connections use SSL enforced by iomgr; this function performs
 * the TLS handshake (see P2.5-2). */
SSL* xce_connect_cluster(xce_cluster_config_t* conf);

/* Connect + join as specific role (worker, scheduler, apiserver) */
SSL* xce_join_cluster(xce_cluster_config_t* conf, xce_peerinfo_t* myinfo);

/* Handle disconnection — auto-reconnect with backoff + redirect following.
 * On success, *ssl is updated to the new SSL connection. On failure, *ssl is
 * set to NULL and the caller must treat the connection as lost. */
int xce_reconnect(SSL** ssl, xce_cluster_config_t* conf);

/* Disconnect from cluster — SSL shutdown + close + free.
 * Safe to call on NULL or already-disconnected SSL*. */
void xce_disconnect_cluster(SSL* ssl);

/* REMOVED: xce_load_cluster_config() — no YAML parsing in libxce.
 * xce_cluster_config_t is populated by caller (worker main, scheduler main, etc.)
 * from --peers and --ca-cert CLI flags. */

10.3 Connection State Machine

                    ┌─────────────┐
                    │    INIT     │
                    └──────┬──────┘
                           │ Load candidates[] from config
                           ▼
                 ┌─────────────────┐
           ┌────▶│   CONNECTING    │◀──────────────┐
           │     └────────┬────────┘               │
           │   timeout or │                        │
           │   connection │                        │
           │     failed   │ connected              │
           │        │     ▼                        │
           │        │  ┌─────────────┐             │
           │        │  │  CONNECTED  │             │
           │        │  └──────┬──────┘             │
           │        │         │ send join request   │
           │        │         ▼                     │
           │        │  ┌─────────────┐              │
           │        │  │  HANDSHAKE  │              │
           │        │  └──────┬──────┘              │
           │        │         │ receive reply        │
           │        │         ▼                     │
           │        │  ┌──────────────┐             │
           │        │  │ CHECK REPLY  │             │
           │        │  └───┬──────┬───┘             │
           │        │      │      │                 │
           │        │  ACCEPTED  REDIRECT            │
           │        │      │      │                 │
           │        │      ▼      ▼                 │
           │        │  ┌───────┐ ┌────────────┐     │
           │        │  │ READY │ │ Follow      │     │
           │        │  │(prime)│ │ redirect →  │─────┘
           │        │  └───┬───┘ │ reconnect  │
           │        │      │     │ to primary  │
           │        │      │     └────────────┘
           │        │      │
           │        │      │ TCP RST / connection lost
           │        │      ▼
           │        │  ┌────────────┐
           └────────┴──│ RECONNECT  │
              backoff   │ (backoff + │
              + jitter  │  jitter)   │
                        └────────────┘

10.4 Candidate Selection Strategy

Two phases of selection:

Initial connection — Try candidates in priority order:

• The highest-priority alive candidate is most likely the primary
• This minimizes redirects on first connection

Reconnection after failure — Try candidates in round-robin with priority bias:

• Start from the candidate after the one we were connected to (it may be down)
• Round-robin distributes reconnection load across candidates after a failover
• Avoids all workers hammering the same candidate simultaneously

/* Priority-ordered for initial, round-robin for reconnection */
static int select_candidate(xce_cluster_config_t* conf, int* attempt,
                            int last_candidate_idx)
{
    if (*attempt < conf->ncandidates) {
        /* First pass: priority order (index 0 first) */
        return (*attempt)++;
    }

    /* Subsequent passes: round-robin starting after last connected candidate */
    int start = (last_candidate_idx + 1) % conf->ncandidates;
    return (start + (*attempt)++) % conf->ncandidates;
}

10.5 Backoff with Jitter

/* Thread-local RNG seed — each thread initializes its own seed once.
 * Using __thread avoids the data race on a shared global rng_seed when
 * multiple threads call backoff_jitter() concurrently (rand_r with the
 * same shared seed is concurrent read-modify-write, which is a race). */
static __thread unsigned int rng_seed;
static __thread int rng_init = 0;

static void rng_init_once(void) {
    if (rng_init) return;
    /* Seed from getrandom(2) for unpredictable jitter — pthread_self() XOR
     * time(NULL) is predictable (deterministic thread IDs, 1-second resolution)
     * and could allow an attacker to coordinate thundering-herd attacks.
     * getrandom(2) is available on Linux 3.17+; fall back to /dev/urandom. */
    unsigned int seed = 0;
#if defined(__linux__) && defined(SYS_getrandom)
    if (syscall(SYS_getrandom, &seed, sizeof(seed), GRND_NONBLOCK) != sizeof(seed)) {
        /* getrandom unavailable (e.g., early boot entropy depletion) — use urandom */
        int fd = open("/dev/urandom", O_RDONLY | O_CLOEXEC);
        if (fd >= 0) { read(fd, &seed, sizeof(seed)); close(fd); }
        else { seed = (unsigned int)((unsigned long)pthread_self() ^ (unsigned long)time(NULL)); }
    }
#else
    seed = (unsigned int)((unsigned long)pthread_self() ^ (unsigned long)time(NULL));
#endif
    rng_seed = seed;
    rng_init = 1;
}

/* Exponential backoff with random jitter to spread reconnect storm.
 * Uses per-thread rand_r() to avoid data races. */
static void backoff_jitter(int attempt, xce_cluster_config_t* conf)
{
    int64_t delay = (int64_t)conf->retry_base_ms * ((int64_t)1 << (attempt > 30 ? 30 : attempt));

    if (delay > conf->retry_max_ms)
        delay = conf->retry_max_ms;

    rng_init_once();
    int jitter = rand_r(&rng_seed) % ((int)delay / 2 + 1);
    delay += jitter;

    usleep((useconds_t)(delay * 1000));
}

Why jitter matters: Without jitter, all 2000 workers back off for exactly the same duration and reconnect simultaneously — a thundering herd. With 50% jitter, the reconnection window spreads across 1.5× the base backoff, allowing the new primary to accept connections gradually.

10.6 Full Connection Flow

SSL* xce_connect_cluster(xce_cluster_config_t* conf)
{
    int attempt = 0;
    int last_candidate = -1;
    int redirect_count = 0;

    for (; attempt < conf->max_retries; attempt++) {
        /* Note: there is no 'goto next_candidate' in this flow — the loop uses
         * 'continue' and 'attempt' is intentionally incremented on each redirect
         * (redirect_count provides a separate hop cap at 5). */
        /* redirect_count is NOT reset per candidate — it is a global hop counter
         * across the entire connection attempt to prevent redirect loops that cycle
         * through multiple candidates (e.g., A→B→A). */
        int idx = select_candidate(conf, &attempt, last_candidate);
        xce_candidate_t* cand = &conf->candidates[idx];

        /* Establish SSL session inline. tcp_connect_ssl() performs
         * TCP connect + TLS handshake as a single operation. On failure, returns NULL. */
        SSL* ssl = tcp_connect_ssl(cand->host, cand->port,
                                   conf->connect_timeout_ms, &conf->tls);
        if (ssl == NULL) {
            backoff_jitter(attempt, conf);
            continue;
        }

        /* Send join request to discover role */
        xce_request_t req;
        xce_reply_t   reply;
        init_join_request(&req, &conf->myinfo);
        xce_send_request(ssl, &req);
        xce_recv_reply(ssl, &reply);

        if (reply.status == 0) {
            /* Accepted by primary */
            last_candidate = idx;
            return ssl;
        }

        if (reply.status == E_XCE_REDIRECT) {
            /* Standby redirected us — validate redirect target before following.
             * Only follow to a known candidate from --peers list to prevent
             * a rogue endpoint from redirecting us to an arbitrary host. */
            int target_known = 0;
            for (int k = 0; k < conf->ncandidates; k++) {
                if (strcmp(conf->candidates[k].host, reply.data.redirect.host) == 0
                    && conf->candidates[k].port == reply.data.redirect.port) {
                    target_known = 1;
                    break;
                }
            }
            if (!target_known) {
                /* Redirect to unknown host — treat as error, try next candidate */
                WARN("redirect to unknown host %s:%u — ignoring",
                     reply.data.redirect.host, reply.data.redirect.port);
                xce_disconnect_cluster(ssl);
                backoff_jitter(attempt, conf);
                continue;
            }

            /* Follow redirect — no backoff since this is expected during discovery.
             * The redirect target must be in the candidate list (validated above).
             * Increment redirect_count BEFORE tcp_connect_ssl so the hop count is
             * enforced even if every redirect target fails to connect — without this,
             * a cycle of unreachable redirect targets would never increment the counter
             * and could loop until max_retries (not the redirect_count cap of 5). */
            if (++redirect_count >= 5) {
                xce_disconnect_cluster(ssl);
                errno = ELOOP;
                return NULL;  /* EXCE_ERR_REDIRECT_LOOP */
            }
            xce_disconnect_cluster(ssl);
            ssl = tcp_connect_ssl(reply.data.redirect.host,
                                  reply.data.redirect.port,
                                  conf->connect_timeout_ms, &conf->tls);
            if (ssl != NULL) {
                /* Re-send join request to verify this is actually the primary.
                 * TOCTOU protection: the primary may have changed between the
                 * redirect and our connection — the join reply confirms we
                 * reached the actual primary, not another standby. */
                init_join_request(&req, &conf->myinfo);
                xce_send_request(ssl, &req);
                xce_recv_reply(ssl, &reply);

                if (reply.status == 0) {
                    return ssl;  /* Confirmed primary */
                }
                /* redirect_count was already incremented before tcp_connect_ssl;
                 * if we reach here with another redirect reply, the outer loop
                 * will re-check the count on the next iteration. */
            }
            xce_disconnect_cluster(ssl);
            continue;
        }

        /* Unexpected reply — try next candidate */
        xce_disconnect_cluster(ssl);
        backoff_jitter(attempt, conf);
    }

    /* All retries exhausted */
    return NULL;
}

10.7 Reconnection on Failover

When the primary fails, workers detect it via TCP RST or heartbeat timeout. The reconnection flow:

int xce_reconnect(SSL** ssl, xce_cluster_config_t* conf)
{
    xce_disconnect_cluster(*ssl);
    *ssl = NULL;

    /* xce_connect_cluster handles candidate selection,
       redirect following, and backoff internally */
    *ssl = xce_connect_cluster(conf);
    return (*ssl != NULL) ? 0 : -1;
}

The worker's main loop integrates reconnection naturally:

/* Worker main loop — simplified */
for (;;) {
    int rc = xce_process_message(ssl);

    if (rc < 0) {
        /* Connection lost */
        TRACE("connection to controller lost, reconnecting...");

        rc = xce_reconnect(&ssl, &conf);
        if (rc < 0) {
            ERROR("failed to reconnect to cluster");
            /* Could exit, or keep retrying depending on policy */
            continue;
        }

        /* Reconnected — re-register with controller */
        on_worker_join(cluster_cls, ssl_to_endpoint(ssl));
        TRACE("reconnected to controller (primary)");
    }
}

10.8 Candidate List Distribution

Controller: reads candidate configuration from cluster.bin (binary format produced by the config tool from cluster.yaml). The controller also accepts --peers CLI flag for the candidate list:

xce_controller \
  --workdir /var/lib/xce \
  -n ctrl-1 \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000

Workers, schedulers, and API servers do NOT read any config file. They accept CLI flags only:

xce_worker \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000 \
  --log-level info

xce_scheduler \
  --peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7000,ctrl-3=10.0.1.3:7000 \
  --log-level info

libxce's xce_cluster_config_t is populated directly from these CLI flags at startup — no YAML parsing in any component except the config tool.

Distribution: The --peers string is generated by the config tool and distributed via deployment scripts (Ansible, Salt) into systemd unit files or container environment variables. Config management tooling should ensure all nodes receive identical candidate lists atomically before any controller restarts.

Candidate list consistency: A stale --peers value on one node could cause connectivity failures — for example, if a worker's peers list includes only ctrl-1 while ctrl-1 is down, the worker cannot discover ctrl-2 or ctrl-3. Config management tooling should ensure all nodes receive identical --peers updates atomically.

Out of scope: The config version hash in heartbeat messages is deferred to P3 (see Out of Scope section). Do not implement the configVersion field, the hash embedding in OP_heartbeat, or the reload-on-mismatch logic during P2.5. The description below is retained for future reference only.

Config freshness control (deferred to P3 — not in scope for P2.5): The controller could embed a config version hash (CRC32 of the serialized candidate list, +1 to avoid 0 sentinel) in OP_heartbeat. When a worker receives a mismatched hash, it logs a warning (config changes require a restart to take effect since candidates come from CLI flags). The configVersion @23 :UInt32 Cap'n Proto field in Request is reserved for this feature. Do not add this field until P3.

10.9 libxce Changes Summary

File	Change
src/lib/include/xce/cluster.h	Add xce_cluster_config_t, xce_candidate_t types
src/lib/include/xce/cluster.h	Add xce_connect_cluster() (returns SSL*), xce_disconnect_cluster(), xce_join_cluster(), xce_reconnect() (takes SSL**) API. No xce_load_cluster_config() — callers populate xce_cluster_config_t from --peers/--ca-cert CLI flags.
src/lib/cluster.c (new)	Implement candidate selection, redirect following, backoff+jitter, reconnection. Accept pre-populated xce_cluster_config_t — no config file reading.
src/worker/main.c	Replace hardcoded controller address with xce_cluster_config_t; populate from --peers and --ca-cert CLI flags; add reconnection loop; defer worker removal (set WORKER_UNAVAILABLE instead of eager delete)
src/scheduler/	Same pattern as worker — parse --peers and --ca-cert CLI flags, populate xce_cluster_config_t, use libxce discovery.
src/frontend/apiserver/backend/clib.go	Use xce_cluster_config_t for controller discovery in RegisterToBackend()

---

Implementation Phases

P0: Foundation (Single-Controller Persistence)

Goal: Controller survives restart with job/queue/worker state intact.

P0 is broken into seven sub-phases with clear deliverables and dependency ordering. Each sub-phase produces a buildable, testable increment.

---

P0-A: Cleanup & Prerequisites

Deliverable: Clean codebase + rtti COW interface ready. No persistence behavior yet.

Step	File	Description
P0-0	src/controller/job.c + worker.c + cluster.c	Remove dead code (Part 1): unused NJOBS/jobid constants, load_jobs() #if 0 body. Do this first so P0 builds on a clean foundation
P0-0a	src/lib/zarray.c + include/zarray.h	Merge zarray_t into xzarray_t (partially completed per git history) — enables uniform COW for all data structures including endpoints. Scope: move the zarray_t definition into xzarray_t, update all zarray_* call sites to use xzarray_* equivalents, verify that existing tests still pass. This step is a prerequisite for P0-2 (header move of xzarray_t) and P0-8a (deferred endpoint cleanup). Owner: this must be completed or explicitly handed off before P0-2 begins; if the merge is not yet complete, P0-8a falls back to the immediate-cleanup path (see Part 4.9 note). IMPORTANT: P0-0a and P0-0b must be developed on the same branch or merged together before P0-2 — both steps modify zarray.c compound-literal initializers and the xzarray_type registration. P0-0a renames zarray_* functions to xzarray_*; P0-0b adds two new slots to intf_varray_t. If developed in parallel on separate branches, the initializer updates will conflict. Merge P0-0a first, then P0-0b (which builds on the renamed functions), or develop both on a single P0-A feature branch.
P0-0b	rtti/typeinfo.h + zarray.c + kvmgr.c + kv.c	rtti library enhancement (see Appendix A.9): Check the installed rtti header first — vfreeze_fn_t, vthaw_fn_t, and retain_fn_t may already be present in the installed /nix/store/...-rtti-0.1.0/include/rtti/typeinfo.h from a prior pass. Verify with grep vfreeze_fn_t $(nix eval --raw .#rtti)/include/rtti/typeinfo.h. If the types are already present, skip the header additions and go directly to the zarray.c implementation work. If not present: add retain_fn_t (bool (*)(void*, void*)), vfreeze_fn_t, and vthaw_fn_t to typeinfo.h; grow intf_varray_t from 6 to 8 slots; add vfreeze()/vthaw() global API wrappers. Remaining work in all cases (whether or not header types were already present): (1) add shadow and shadow_caps fields to xzarray_private_t in zarray.c; (2) update all compound-literal intf_varray_t initializers in zarray.c, kvmgr.c, and kv.c to include the two new slots — first convert all positional initializers to designated-initializer style ({ .vstore = ..., .vfreeze = (vfreeze_fn_t)NULL, .vthaw = (vthaw_fn_t)NULL }) before adding the new slots, since adding members to a struct with positional initializers is a compile error. Run grep -rn 'intf_varray_t' src/ to find all initializer sites before starting. Update nix derivation version from rtti-0.0.1 to rtti-0.1.0. This is a prerequisite for P0-2 which wires vfreeze/vthaw to the xzarray implementations. Requires P0-0a to be merged first (same-branch constraint above). rtti source: the rtti library source is maintained by the XCE team (not a third-party dependency). The implementer has direct access to modify typeinfo.h and bump the Nix derivation. Locate the source via nix flake show or the project's flake.nix inputs. BREAKING ABI CHANGE: both intf_varray_t (grows from 6 to 8 pointers) and xzarray_private_t (gains shadow and shadow_caps fields, changing privateSize) change size. Since no version of XCE has been released, backward binary compatibility is not required — all components are rebuilt together. Trigger a full ninja -C build clean rebuild after this step.
P0-5	src/include/xce/cluster_def.h + include/config.h	Add persist_t persist; to cluster_t. Pre-addition audit: run grep -rn 'cluster_t|sizeof.*cluster|offsetof.*cluster' src/ and verify no code assumes sizeof(cluster_t) or uses field offsets before adding the new field — adding a large struct may shift subsequent field offsets. Note c->endpoints is currently zarray_t endpoints (not xzarray_t); the deferred endpoint cleanup (P0-8b) requires a type change in P0-0a. Note: ha_state_t ha; is added in P1-5 (HA wiring) — it is not needed in P0 since no P0 codepath references c->ha. P0-5 adds only persist_t persist;. Also augment the existing include/config.h (do NOT create a new file — the file already exists with XCE_MAXNAME_LEN and job ID bit fields) with all system-level tunable constants (see Appendix A.10): add MAX_STANDBYS=3, MAX_HA_CANDIDATES=4, WAL_FORMAT_VERSION=1, WAL_ENTRY_TYPE_MAX=100, MAX_WAL_PAYLOAD_BYTES=(1<<20), MAX_SNAPSHOT_BYTES=(256*1024*1024), DEFAULT_WAL_SYNC_MS=10, DEFAULT_SNAP_INTERVAL_S=60, etc.

---

P0-B: WAL & COW Infrastructure

Deliverable: WAL writes work; COW freeze/thaw round-trip on zarray. No checkpoint or recovery yet. Requires: P0-A.

Step	File	Description
P0-1	src/controller/persist.c + include/persist.h	WAL writer, segment management, fsync batching. persist_init() receives workdir and name parameters (from CLI --workdir and -n flags). It derives wal_path = effective_dir/data/wal and snap_path = effective_dir/data/snap (where effective_dir = workdir / name). Must validate derived paths against path traversal using realpath() — reject any path that resolves outside the intended base directory or contains .. components — then create directories with mode 0700 (owner-read-only) if they do not exist, and verify permissions on existing directories at startup — refuse to start if derived wal_path or snap_path is not mode 0700 and owned by the controller UID (rather than log-and-continue). Job payloads written to WAL/snapshot files may include environment variables and command lines containing credentials; this permission gate prevents a misconfigured mount from exposing them. Runtime permission re-check: persist_checkpoint() (P0-3) must also call fstatat(dirfd(wal_dir), ".", &st, 0) and verify (st.st_mode & 0777) == 0700 && st.st_uid == getuid() before writing each new snapshot. Log ERROR and skip the snapshot write if the check fails — a post-startup permission change (e.g., misconfigured backup agent chmod) is caught within one checkpoint interval (default 60s) rather than persisting until the next restart. Add persist_t.wal_dirfd and persist_t.snap_dirfd (open directory file descriptors) so the check uses fstatat() rather than stat() to avoid TOCTOU race against path traversal. Dedicated fsync thread required: wal_sync_ms defaults to 10ms; delaymgr has 2-second granularity (sleep(precision) where precision=2) so the WAL fsync timer must be a dedicated pthread using nanosleep() or timerfd_create(CLOCK_MONOTONIC) — do not use delaymgr for this. Encryption note: WAL and snapshot files are stored unencrypted by the controller. persist_init() must log an INFO message at startup: "WAL/snapshot stored unencrypted; ensure effective_dir/data/wal and effective_dir/data/snap reside on an encrypted filesystem (e.g., LUKS/dm-crypt) if job payloads contain sensitive data". File-level encryption (e.g., AES-GCM per-record) is out of scope for this design.
P0-2	src/lib/zarray.c + include/zarray.h	COW extensions: shadow, shadow_caps, xzarray_freeze(), xzarray_thaw(); modify xzarray_iter_next() for shadow-aware merge walk (see Part 3.5); modify xzarray_freeze() to set pos = shadow_caps (see Part 3.5 vstore invariant). Header move required: xzarray_t is currently defined as a file-private struct in zarray.c; it must be moved to include/zarray.h so persist.c, repl.c, and other callers can embed or reference it. xzarray_private_t (the internal struct containing mylock_t) is also currently file-private in zarray.c and must be moved to include/zarray.h as well. _Atomic _Bool inuse audit (mandatory): changing xzarray_item_t.inuse from unsigned int :1 to _Atomic _Bool (required by Part 3.5 memory ordering) affects every read and write of this field in zarray.c. Run grep -n '\.inuse' src/lib/zarray.c before starting — expect ~15 call sites across xzarray_destroy, xzarray_store, xzarray_get, xzarray_clear, xzarray_iter_next, and the new COW paths. Every read must use atomic_load_explicit(..., memory_order_acquire) and every write must use atomic_store_explicit(..., memory_order_release). This is a checklist item — do not mark P0-2 complete until all sites are converted and verified. Requires: P0-0a — P0-0a merges zarray_t into xzarray_t; P0-2 must not begin until P0-0a is merged and tested.
P0-2a	src/lib/zarray.c	COW lock protocol: verify xzarray mylock_t mutex is taken correctly inside vget/vput during COW mode; kvmgr my_rwlock_t is the outer coordination lock (see Part 3.9)
P0-2b	src/controller/delay.c + src/controller/delay.h + all callers	delaymgr precision enhancement (prerequisite for P0-3): rename precision field to precision_ms (milliseconds); replace sleep(precision) with struct timespec ts = {0, precision_ms * 1000000L}; nanosleep(&ts, NULL);. Important: delay_task() delay parameter remains in seconds — only the sleep granularity changes. Audit all delaymgr_init() call sites (grep -rn 'delaymgr_init' src/) and update each to pass precision_ms = 200. Verify no existing delay_task() call passes seconds-based values that assume the old 2-second tick — the time_t arithmetic (ts_nextrun = now + delay) is unchanged. Update cluster.c:604 from delaymgr_init(&c->delaymgr, 2, run_delay_tasks) to delaymgr_init(&c->delaymgr, 200 /* precision_ms */, run_delay_tasks).

---

P0-C: Checkpoint & Recovery

Deliverable: Full persist/recover round-trip — controller survives restart. Requires: P0-B.

Step	File	Description
P0-3	src/controller/persist.c	Requires: P0-2b (delaymgr precision enhancement — 1-second checkpoint poll requires nanosleep granularity). Checkpoint via dual zarray freeze → serialize shadow → thaw + GC; use delaymgr for periodic checkpoint timer (see Part 3.10). checkpoint_requested inline trigger polling: The checkpoint timer thread also polls persist_t.checkpoint_requested on each timer wake. The timer callback is registered with a 1-second interval via delay_task(&c->delaymgr, 1, -1, persist_checkpoint_timer_fn, c). Each callback checks checkpoint_in_progress first (skip if set), then checkpoint_requested (run checkpoint if set), then snap_interval_s countdown (run periodic checkpoint when countdown reaches zero). The WAL fsync timer (10ms intervals) and HA heartbeat watchdog both require dedicated pthreads — they cannot use delaymgr.
P0-4	src/controller/persist.c	Recovery: load snapshot + replay WAL. Includes: (1) snapshot version check — reject snapshots where SnapshotHeader.version > WAL_FORMAT_VERSION (see Part 2.5 step 2a); (2) Cap'n Proto traversal limit — set ReaderOptions.traversalLimitInWords to the fixed upper bound MAX_SNAPSHOT_BYTES / 8 words (see Part 2.5 step 2b; do NOT use file_size / 8); (3) snapshot CRC verification over both header (headerChecksum covering lsn, term, version, counts) and body (checksum); (4) WAL entry type max guard — abort if any entry type exceeds WAL_ENTRY_TYPE_MAX (see Part 2.5).
P0-12a	job.c	Implement restore_job(cluster_t* c, uint64_t jobid, const void* capnp_payload) — inserts a job at its original jobid without allocating a new one. Used by apply_wal_entry() for WAL_JOB_SUBMIT during recovery and standby replay. Unlike job_submit(), restore_job() must accept an explicit jobid, bypass the jobid allocator, and skip BEFORE/AFTER_EVENT hooks. Duplicate jobid handling: if the jobid already exists in the kvmgr (e.g., from a duplicate WAL entry or a corrupt WAL that replays the same submit twice), restore_job() must log a WARN and return -EEXIST without overwriting the existing entry. apply_wal_entry() treats -EEXIST as a non-fatal skip. Required before P0-4 (recovery replay) and P2-2 (standby apply). Note on -EEXIST occurrence: During clean recovery (snapshot + WAL replay from a trimmed WAL), -EEXIST should never occur — the snapshot represents a consistent point-in-time state, and WAL entries after the snapshot's LSN contain only new state changes. -EEXIST is a safety guard for crash-recovery edge cases where: (1) a checkpoint failed mid-write but the WAL was not yet trimmed, causing the recovery algorithm to replay entries already present in the snapshot; or (2) a standby receives a duplicate replication entry due to reconnection. If -EEXIST occurs during what should be a clean recovery, it indicates a WAL trim bug or snapshot/WAL inconsistency and should be investigated.
P0-12	worker.c	Wire restore_worker() into recovery path. Adapter requirement: the existing restore_worker(void* cls, xce_worker_t host) takes xce_worker_t by value (struct copy), but Cap'n Proto deserialization produces a pointer-based Worker struct. Implement a wrapper worker_from_capnp() that: (1) receives the deserialized Worker Cap'n Proto object, (2) allocates and populates an xce_worker_t by deep-copying pointer fields (char* name → strdup, xce_config_t** resources → allocate and copy each config item, xce_res_threshold_t** thresholds → same), (3) calls restore_worker(cls, host) with the populated struct. The existing restore_worker() function signature is unchanged — the adapter sits between Cap'n Proto and restore_worker().

---

P0-D: Hooks & Reply Ordering

Deliverable: Persistence hooks fire on AFTER_EVENT; xce_reply() moved after hooks for sync durability. Requires: P0-C (hooks need WAL to write to).

Step	File	Description
P0-4a	src/include/xce/cluster_def.h + src/controller/job.c	Add HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME to the hook detail enum. Critical codebase fact: on_job_stop() and on_job_resume() in job.c currently call mark_job_update() which fires HOOK_DETAIL_JOB_UPDATE — they do NOT currently fire HOOK_DETAIL_JOB_EXIT. (Footnote 1 in Part 2.3 incorrectly states they fire HOOK_DETAIL_JOB_EXIT.) The audit directive must search for mark_job_update calls in the stop/resume paths, not HOOK_DETAIL_JOB_EXIT. The implementation must update on_job_stop() and on_job_resume() to call xce_run_hooks directly with HOOK_DETAIL_JOB_STOP / HOOK_DETAIL_JOB_RESUME (bypassing mark_job_update() for those events). Audit: grep -n 'mark_job_update|HOOK_DETAIL_JOB' src/controller/job.c — confirm all non-terminal paths are correctly remapped. This is a prerequisite for P0-6 WAL_JOB_STOP and WAL_JOB_RESUME hook wiring.
P0-5a	src/include/xce/cluster_def.h + types.h	Add entity terminal states: XCE_WORKER_UNAVAIL (already exists in types.h:165), XCE_WORKER_HIDDEN, QUEUE_STAT_HIDDEN; add JOB_STAT_TRANSIENT = 0x400 to job status bitmask (must not use 0x200 — JOB_STAT_WAIT already occupies 0x200, see types.h:351). This fits in the existing uint32_t state field (current highest bitmask is 0x40000 for JOB_STAT_ERR). The scheduler must skip workers that own transient jobs (no new work dispatched to them). API queries for transient jobs must NOT return the job's last-known state (e.g., PEND) as that would actively mislead users who submitted running jobs. The Go API server mapping of JOB_STAT_TRANSIENT to a user-visible TRANSIENT response is tracked separately — see P2.5-7 (or a dedicated API server story). Worker reconnection resolves the transient state by setting the job's actual terminal/running status (the worker reports job completion/exit). If the worker never reconnects, the job remains transient until operator intervention (same as existing WORKER_UNAVAILABLE behavior for workers).
P0-6	src/controller/persist.c	Hook functions: persist_job_hook_fun, persist_queue_hook_fun, persist_worker_hook_fun (requires P0-4a for HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME wiring)
P0-7	src/controller/job.c + all handler files	Move xce_reply() after AFTER_EVENT hooks in handle_jobsubmit(); verify no kvmgr lock held during hooks (see Part 8.3 lock safety analysis). Full audit required: grep for all xce_run_hooks(AFTER_EVENT + xce_reply pairs across all handler files (job.c, worker.c, join.c, cluster.c, etc.) to identify any handler that still replies before post-hooks. va_list compatibility audit: P0-6 adds persist_job_hook_fun() as an AFTER_EVENT hook for HOOK_DETAIL_JOB_NEW that reads two extra va_list arguments (int* sync_rc and persist_batch_ctx_t*). All existing registered AFTER_EVENT HOOK_DETAIL_JOB_NEW hooks (e.g., in apiserver.c, sched.c) must be audited to verify they do NOT call va_arg() for these extra arguments (they should only consume the arguments they were registered to expect). Document for each: (a) does the hook call va_arg(), (b) for which arguments, (c) whether adding the new args shifts its position. If any existing hook calls va_arg() past the standard args, it must be updated or given its own hook detail to avoid type mismatches. For async-replicated events, reply ordering is less critical, but consistency is preferred. Document each handler's reply ordering decision and rationale as part of this step.

---

P0-E: Deferred Cleanup & GC

Deliverable: Entity lifecycle management — terminal entities cleaned at checkpoint, endpoints deferred. Requires: P0-C (checkpoint GC needs thaw infrastructure).

Step	File	Description
P0-8	src/controller/worker.c	Extend existing mark_worker_unavail() to set w->host.dead_since = time(NULL) (field already exists in xce_worker_t); add XCE_WORKER_HIDDEN bitmask path; already deferred — no zarray_del removal needed (see Part 4.10)
P0-8a	src/controller/cluster.c	Fallback path (no dependency on P0-0a): Keep the existing zarray_del()/zarray_clear() calls in handle_endpoint_error() — this is the safe baseline that ships with P0-E. Mark these call sites with /* TODO P0-8b: replace with deferred cleanup after P0-0a */ comments so they are easy to find. P0-8a is complete when these comment markers are in place.
P0-8b	src/controller/cluster.c	Deferred cleanup path — requires P0-0a to be merged and green first. Replace the TODO P0-8b sites from P0-8a: close fd/SSL immediately, set ENDPOINT_DISCONNECTED state, clean up the slot at checkpoint via endpoint_should_retain(). P0-8b must not be started until P0-0a is merged and all zarray_t → xzarray_t migration tests pass. This ordering gate prevents the deferred path from shipping before the zarray infrastructure is ready.
P0-13	job.c	Implement job_is_terminal() (check `JOB_STAT_EXIT
P0-14	kvmgr.c + htab.c	Implement kvtab_gc_stale_indices() — walk hash chains, remove entries whose mapped zarray index has inuse=0. Called during thaw while kvmgr write lock is held. Internally uses ht_del() (not del_key — remains NULL per Part 4.1 invariant)
P0-14a	kvmgr.c	Implement kvmgr_capture_keys(meta_obj_t* kvmgr, slot_key_t** keys_out, size_t* n_keys_out) — walks the kvtab under caller-held write lock and deep-copies each key string into a malloc'd slot_key_t[] array. Called by persist_checkpoint() in Part 3.8 immediately after xzarray_freeze() while the write lock is still held. Depends on: P0-14 (htab walk primitives) and P0-0b (slot_key_t struct). Required before: P0-3 (checkpoint) can be tested end-to-end. The serialization pthread uses the captured key array to build Cap'n Proto key→index mappings without holding any lock. free(keys) releases the array; individual key strings are freed by the keys array destructor.

---

P0-F: Schema, Config, & Cluster Wiring

Deliverable: Full end-to-end — config parsed from YAML, schemas defined, persistence wired into cluster lifecycle. Requires: P0-D + P0-E.

Step	File	Description
P0-9	controller.capnp	Add PersistConfig, WALEntry, SnapshotHeader, Snapshot schemas; add version field (UInt32, default 1) to SnapshotHeader for format version detection; add snapKeepN @6 :UInt32 = 2 to PersistConfig (see Appendix A.5); add transientJobTimeoutS @7 :UInt32 = 300 to PersistConfig (see Appendix A.5 and persist_config_t.transient_job_timeout_s); add transientSince (UInt64, epoch seconds) to the Job schema — set when a job enters JOB_STAT_TRANSIENT at promotion; needed for the transient_job_timeout_s GC check; add deadSince to Worker schema; add bitmask flags for XCE_WORKER_HIDDEN, QUEUE_STAT_HIDDEN; add jobStatTransient = 0x400 constant comment in Job schema (status remains UInt32 bitmask; not 0x200 which is JOB_STAT_WAIT)
P0-10	cluster.c	Wire persist_init/recover/start/stop/fini into cluster_init/run/fini. Requires: P0-9 — PersistConfig and the snapshot schemas must be defined in controller.capnp before cluster.c can reference them via the generated C bindings.
P0-11	myconf.c + YAML config	Parse persistence configuration

---

P0-G: End-to-End Smoke Tests

Deliverable: Integrated persistence test suite exercising cross-sub-phase interactions. Requires: P0-F.

Gate structure note: These tests run AFTER all P0 sub-phases (P0-A through P0-F) are complete. They are end-to-end integration tests, not per-sub-phase gates. For per-sub-phase incremental verification, each sub-phase's deliverable should be tested immediately after it lands (see the "each sub-phase produces a buildable, testable increment" principle). P0-G tests validate the full assembled system.

Step	File	Description
P0-15a	Tests (WAL E2E)	WAL write/fsync/CRC round-trip: write entries, simulate crash (truncate), recover and verify correct replay up to last valid CRC
P0-15b	Tests (Checkpoint E2E)	Checkpoint + recovery: freeze → serialize → thaw cycle; snapshot-to-disk; restart from snapshot + bridging WAL; verify entity counts match
P0-15c	Tests (Hook E2E)	Hook invocation ordering: BEFORE_EVENT abort path, AFTER_EVENT sync_rc propagation, reply-after-hook sequencing; xce_reply fires after hook returns
P0-15d	Tests (GC E2E)	Retention/GC: terminal entities survive until TTL expires, cleaned at checkpoint; deferred endpoint cleanup; JOB_STAT_TRANSIENT auto-terminate path
P0-15e	Tests (Cap'n Proto E2E)	Cap'n Proto partial-merge semantics: absent optional fields are NOT zeroed; partial WAL entry applies only stated fields to in-memory struct

P1: HA Foundation (LSF-Style Election)

Goal: Standby controller can take over when primary fails.

Step	File	Description
P1-1	src/controller/ha.c + src/controller/netio.c	Verify that HA listeners (election probe, replication accept) reuse the existing iomgr/SSL pipeline without modification. No create_ssl_context() API changes needed — the trusted LAN model (Part 5.9) requires no mTLS. Peer identity validated by host:port match against --peers candidate list.
P1-2a	src/lib/hook.c + src/lib/include/xce/hook.h	Prerequisite for P1-2. Add xce_disable_hook(hooksys_t* hs, int type, hook_fn_t fn) — scans for a matching entry by (type, fn) and stores NULL atomically with atomic_store_explicit(memory_order_release). Add void xce_disable_hook(...) to hook.h. Struct change: change hook_item_t.fn from hook_fn_t fn; to _Atomic hook_fn_t fn; — required so xce_disable_hook() can store NULL with release ordering and xce_run_hooks() can load with acquire into a local before NULL-check and call. Update xce_add_hook() to use atomic_store_explicit(&it->fn, fn, memory_order_release). NULL-fn guard in xce_run_hooks(): after loading hook_fn_t local_fn = atomic_load_explicit(&it->fn, memory_order_acquire), add if (!local_fn) continue; — call local_fn(...) instead of it->fn(...). Slot reuse: xce_add_hook() scans for existing NULL-fn slots of the same type before appending — prevents unbounded list growth on promote→demote cycles. Note: xce_add_hook() currently rejects NULL fn pointers; the slot-reuse scan path must only match slots where an entry was previously registered and then disabled (not freshly zeroed).
P1-2	src/controller/ha.c + include/ha.h	Requires: P1-2a. Role management, am_i_primary(), promotion/demotion. OP_steppingDown handler: on receiving OP_steppingDown from the primary, the handler validates: (1) the sender's term matches ha->term (stale messages ignored), (2) the sender is the current primary (verified via ha->primary_host/ha->primary_port). Priority resolution mechanism: upon validation, each standby immediately runs am_i_primary() — the same election probe used at startup. This naturally resolves which standby is the highest-priority reachable candidate: each standby probes all higher-priority candidates; if none respond within election_timeout_ms, it promotes itself. This avoids the impossible requirement of knowing which other standbys were connected to the stepping-down primary at the time of the message. Non-highest-priority standbys that discover a higher-priority peer is reachable (via the probe returning "you are NOT primary") defer and wait for the new primary's heartbeat. The am_i_primary() call runs in a temporary pthread (same as startup) so the handler does not block the iomgr event loop. The handler runs in the iomgr/vtask thread for initial validation; the actual am_i_primary() probe is dispatched asynchronously.
P1-3	src/controller/ha.c	Heartbeat watchdog: OP_heartbeat exchange via iomgr/vtask pipeline; ha_check_heartbeat() on a dedicated pthread with nanosleep() or timerfd_create(CLOCK_MONOTONIC) (NOT delaymgr — see Part 5.5 heartbeat detection latency note) detects primary failure (replaces tcp_probe). Shutdown protocol (mirrors checkpoint thread pattern): add _Atomic int ha_watchdog_stop to ha_state_t. The watchdog thread checks this flag on each nanosleep() wake and exits cleanly when set. ha_fini() sets the stop flag then calls pthread_join() on the watchdog thread before returning — this prevents the watchdog from accessing freed ha_state_t (embedded in cluster_t) after cluster_fini() frees it. ha_fini() must be called from cluster_fini() before any ha_state_t memory is reclaimed. The watchdog thread must not be cancelled with pthread_cancel() — the same rationale as election probe threads (OpenSSL handlers are not cancellation-safe; see Part 5.4 startup note).
P1-3a	src/controller/ha.c	Promotion sequence: stop replay, increment term, recover, start primary; SIGUSR1 handler for manual failover (see Part 5.8). Requires: P1-2a — ha_demote_to_standby() calls xce_disable_hook() which is added in P1-2a. Do not start P1-3a until P1-2a is merged and tested.
P1-4	controller.capnp	Add term to Request, standby to HostRole, replEntry/replAck/redirect/heartbeat/steppingDown/replSnapshot/replCatchUpDone opcodes (see Part 5.10 for full opcode list), RedirectInfo struct
P1-5	cluster.c	Wire HA into startup sequence; add ha_state_t ha; to cluster_t in this step (note: persist_t persist; field was already added to cluster_t in P0-5)
P1-6	join.c	Add redirect logic in handle_join() — standby sends REDIRECT for worker/scheduler/apiserver join requests
P1-7	Tests	Election logic, promotion sequence, term fencing, redirect response

P2: Replication (Fan-Out TCP)

Goal: Primary replicates state changes to standbys in real time.

Step	File	Description
P2-1	src/controller/repl.c + include/repl.h	Replicator: fan-out send via endpoint/iomgr SSL pipeline, ACK tracking; standby_conn_t.ep replaces standby_conn_t.fd (see Part 6.5)
P2-2	src/controller/repl.c	Standby receiver: OP_replEntry dispatched via iomgr/vtask pipeline; OP_replAck sent via endpoint outq (not raw send()). apply_wal_entry() does not fire hooks — persistence hooks are registered only on the primary path (in cluster_init() after recovery completes, or in ha_promote_to_primary() step 5 on promotion). No bypass flag is needed; the hook system is empty on standbys. Standbys apply WAL entries the same way recovery does (see Part 4.2 recovery)
P2-3	src/controller/repl.c	Sync replication: repl_wait_ack() with timeout; only triggered for WAL_JOB_SUBMIT from API server connections. ACK notification mechanism: when the standby's OP_replAck arrives via iomgr/vtask, it must wake the blocked repl_wait_ack() call. Implement with a per-standby condition variable (pthread_cond_t) under repl.lock: the vtask handler signals it on receiving OP_replAck; repl_wait_ack() parks on it with pthread_cond_timedwait(). This keeps the blocking confined to the vtask thread running the handle_jobsubmit() path. Pool floor note: the default vcpu pool is 8 (myconf.c default). Under burst job submission, 8 simultaneous sync-replicating submits exhaust the pool. Clusters using sync replication should configure vcpus >= 16: the reasoning is that each blocked repl_wait_ack() occupies one vcpu for up to sync_timeout_ms (50ms default); at the typical API server connection count of 1–3, the realistic burst concurrency is low, but with multiple API servers sending simultaneous submits the pool can saturate quickly. 16 vcpus provides 2× headroom over the default and headroom for non-submit vtasks (worker joins, heartbeats) that must not be starved. Clusters with only 1 API server and low submit rates can stay at 8. Startup enforcement: persist_start() must log a WARN at startup if sync replication is enabled (sync_timeout_ms > 0) and the configured vcpus < 16: WARN("sync replication enabled with vcpus=%d (< 16); recommend vcpus >= 16 to prevent repl_wait_ack pool saturation", vcpus). This is a warning, not a hard failure, so low-concurrency deployments can still operate. Operators should monitor ctrl_vcpu_queue_depth and increase vcpus if it stays above 0 during normal operation. ACK deadlock prevention: repl_on_ack() must be callable directly from the io_worker read callback (i.e., from handle_msg() for OP_replAck before vtask dispatch) — it only does pthread_cond_signal() and an acked_lsn update under repl.ack_lock, which is safe to call from io_worker without going through vtaskmgr. If OP_replAck were dispatched via vtaskmgr like all other messages, a pool of 8 vcpus all blocked in repl_wait_ack() would deadlock: the incoming ACK sits in the vtask queue, but no vcpu is free to process it. By handling ACKs in io_worker directly, the signal wakes a sleeping vcpu regardless of pool saturation. Worker join lock constraint: handle_worker_join() holds workermgr WRITE lock during add_worker() + AFTER_EVENT hooks + xce_reply(). The persistence hook for HOOK_DETAIL_WORKER_JOIN must use persist_wal_async() only — worker join is always async-replicated and must never block on repl_wait_ack while the write lock is held
P2-4	src/controller/persist.c	Wire sync/async replication into persist_wal_sync_append() / persist_wal_async()
P2-5	src/controller/repl.c	Standby catch-up: when standby reconnects via endpoint, stream missing WAL entries. The primary trims WAL only up to min(acked_lsn) across connected standbys (standby_conn_t.acked_lsn; falls back to checkpoint_lsn when no standbys — see Part 2.4 WAL trim guard). If a reconnecting standby needs WAL entries older than the primary's oldest surviving segment, transfer the latest snapshot first, then stream WAL entries from the snapshot's LSN forward. Snapshot atomic write on standby receive: the standby writes reassembled snapshot chunks to a temp file snap/snap-<lsn>.tmp, verifies the CRC32 after the final chunk, then uses rename(2) to atomically promote to snap/snap-<lsn>.capnp. This prevents a partial snapshot file from being discovered by persist_recover() on a subsequent restart. The standby snap/ directory must have mode 0700 (same startup check as the primary — verify this is covered by the standby's persist_init() call). OP_replCatchUpDone handler (primary side): when the primary receives OP_replCatchUpDone from a standby, it must: (1) locate the matching standby_conn_t slot by endpoint pointer, (2) clear slot->in_catch_up with atomic_store_explicit(&slot->in_catch_up, 0, memory_order_release) — this unblocks repl_wait_ack() from excluding that standby, so its subsequent OP_replAck messages count toward the "at least one ACK" predicate. The handler runs in the io_worker read callback (same as OP_replAck — not dispatched via vtaskmgr) to avoid a vtask queue latency spike after catch-up. If no matching slot is found (standby disconnected between sending OP_replCatchUpDone and the primary processing it), log WARN and discard silently.
P2-6	Tests	Replication round-trip (via SSL pipeline), sync ACK timeout, standby catch-up

P2.5: libxce Smart Discovery

Note

: P2.5 carries the same implementation scope as P1 or P2 (8 steps). It is numbered 2.5 — not 3 — to reflect that the client library code (P2.5-1 through P2.5-6) depends only on P1 and can be developed in parallel with P2. However, the functional value of P2.5-7 (API server discovery, RegisterToBackend()) is only fully realized when P2 replication is also deployed: without P2, a promoted standby has no live replication from the primary, so reconnecting the API server to a promoted standby via P2.5-7 serves no purpose. Deploy P2 and P2.5 together for a complete HA experience. P2.5 alone (without P2) provides worker/scheduler auto-reconnect after failover but does not provide replication continuity.

Candidate count ceiling: MAX_STANDBYS = 3 and MAX_HA_CANDIDATES = 4 are standalone constants defined directly in include/config.h (see A6-010). The deployment ceiling is ≤3 controllers (1 primary + 2 standbys). xce_cluster_config_t must enforce the structural ceiling when populated from --peers CLI flag (reject configs with more than MAX_HA_CANDIDATES candidates). Operators are strongly advised to deploy no more than 3. Add a WARN log at startup if candidates_count > 3: "HA cluster has %d candidates; LSF-style election is validated for ≤3. Larger clusters increase split-brain risk."

Goal: Workers and schedulers can discover the primary via any candidate (P2.5 alone). API server discovery (P2.5-7) is also included here but only delivers value when deployed together with P2 replication — without P2, the promoted standby has incomplete state and reconnecting the API server to it is misleading rather than helpful. Deploy P2.5-7 only alongside P2. Design decision: libxce auto-discovery and auto-reconnect are foundational, not optional. All client-side controller connection code in the codebase will be migrated to this path — there is no "raw TCP to controller" fallback after this phase. This ensures the scheduler policy enhancement work that follows operates on a stable HA foundation.

Step	File	Description
P2.5-1	src/lib/include/xce/cluster.h	Define xce_cluster_config_t, xce_candidate_t, API functions
P2.5-2	src/lib/cluster.c (new)	Implement xce_connect_cluster(): candidate selection, redirect following, backoff+jitter. SSL: all controller connections use SSL (enforced by netio.c); xce_connect_cluster() must perform the TLS handshake and return an SSL-wrapped connection, not a raw fd. Use the same CA cert path from xce_cluster_config_t for server certificate verification — `SSL_CTX_set_verify(SSL_VERIFY_PEER
P2.5-2a	src/lib/meson.build	Add cluster.c to the libxce_src source list so the new file is compiled into libxce
P2.5-3	src/lib/cluster.c	Implement xce_reconnect(): auto-reconnect with state machine
P2.5-4	src/lib/cluster.c	REMOVED: no xce_load_cluster_config() function. Workers/schedulers/apiserver parse --peers and --ca-cert CLI flags in their main() and populate xce_cluster_config_t directly before calling libxce functions.
P2.5-5	src/worker/main.c	Replace hardcoded controller address with xce_cluster_config_t; populate from --peers and --ca-cert CLI flags in main(); add reconnection loop. Do NOT read any config file.
P2.5-6	src/scheduler/	Same pattern as worker — parse --peers and --ca-cert CLI flags, populate xce_cluster_config_t, use libxce discovery. Do NOT read any config file.
P2.5-7	src/frontend/apiserver/backend/clib.go	Use cluster discovery for controller connection in RegisterToBackend(). Decision: use CGo binding (option a). Expose xce_cluster_config_t and xce_connect_cluster() through the existing clib.go CGo shim — clib.go already imports xce headers and wraps C calls for other controller interactions, so adding cluster discovery there is consistent and avoids duplicating config parsing logic in Go. Add a #include "xce/cluster.h" import and a XceConnectCluster(conf *C.xce_cluster_config_t) int Go wrapper, then call it from RegisterToBackend() instead of the current hardcoded dial.
P2.5-7a	src/frontend/apiserver/	Go API server: JOB_STAT_TRANSIENT serialization — Update the Go jobstatus serializer to map the JOB_STAT_TRANSIENT bitmask to { "status": "TRANSIENT", "status_detail": "job state uncertain after controller failover — awaiting worker reconnect" }. Without this, users querying jobs after failover receive stale PEND state, which actively misleads them. This is a standalone story that does not depend on P2.5-7 controller discovery — it only depends on P0-5a (which defines the JOB_STAT_TRANSIENT bitmask). Should ideally ship concurrent with or immediately after P0. Deliverable: add case JOB_STAT_TRANSIENT: return "TRANSIENT" (or equivalent) to the job status switch in the Go API serializer; add a test case.
P2.5-8	Tests	Candidate selection, redirect following, backoff+jitter, reconnection

P3: Integration & Hardening

Goal: Production-ready persistence + HA.

Step	Description
P3-1	End-to-end failover test: kill primary, verify standby promotes, workers reconnect via libxce discovery
P3-2	Redirect test: worker connects to standby, receives redirect, reconnects to primary
P3-3	WAL corruption handling: test torn writes, partial segments, CRC mismatch
P3-4	Performance: benchmark 1000 job submits/sec with sync replication
P3-5	Checkpoint GC: verify expired jobs/workers/queues are cleaned; verify dependents block cleanup
P3-6	Test the manual failover state machine path by calling ha_demote_to_standby() directly from a test harness (e.g., a test-only ha_trigger_manual_failover(c) shim or a dedicated integration test binary). This validates that the HA state machine correctly handles a manual promotion trigger — role transitions, WAL flush, hook re-registration — without requiring the unimplemented xce hostcontrol --failover CLI (see Out of Scope section).
P3-7	Metrics: WAL size, replication lag, checkpoint duration, standby health, entity GC stats. Metric naming convention: All HA/persistence metrics use a ctrl_ prefix (e.g., ctrl_repl_degraded_count, ctrl_split_brain_detected, ctrl_async_events_lost) to namespace them within the controller's existing metrics perimeter. Metric names must match the pattern ctrl_[a-z_]+ and should be registered in a central metrics table with a description and type (counter/gauge/histogram). The implementation should define this table as part of persist.c or ha.c with a metrics_init() call in persist_start() / ha_init(). Split-brain alert: log a WARN-level message and expose a ctrl_split_brain_detected counter whenever a term conflict is detected (primary receives a message with a higher term than its own, or standby detects two primaries). This is the minimum operator signal that the cluster has entered a split-brain state. Async event loss counter: expose ctrl_async_events_lost — incremented for each async WAL entry that was in-flight (written locally but not yet replicated) when the primary failed. Computed by the newly promoted standby during promotion: events_lost = primary_last_lsn - standby_applied_lsn (where primary_last_lsn is recovered from the old primary's WAL at promotion time, if available; otherwise log WARN("async event loss count unavailable — primary WAL not accessible")). This gives operators visibility into the amount of async data loss after a failover. Transient job auto-termination counter: expose ctrl_transient_jobs_stuck (counter, type: counter) — incremented each time a JOB_STAT_TRANSIENT job is auto-terminated by job_should_retain() after transient_job_timeout_s elapses without worker reconnection. Accompanies a WARN-level log message with jobid and worker name. Allows operators to detect and quantify failover-induced job loss without reviewing individual log entries.

P4: Cleanup & Documentation

Step	Description
P4-1	Update src/frontend/docs/ to remove stale PostgreSQL references
P4-2	Update CLAUDE.md files with persistence/HA architecture
P4-3	Operational runbook: how to add/remove standby, manual failover, disaster recovery. Must include candidate list upgrade sequencing: (1) update --peers on all controllers first and restart them (they continue operating with the new list); (2) then update workers, schedulers, and API servers. Never update workers before controllers — a worker with a candidate list that includes a new controller before that controller knows about itself cannot connect during the transition. Add a section "Changing the candidate list" with this explicit sequence. Also include: what ctrl_split_brain_lost_submits means and how to notify affected users; how to interpret ctrl_reconciliation_stuck_workers and what manual steps to take; the JOB_STAT_TRANSIENT user-facing meaning and when to expect it.

Out of Scope

The following are explicitly not covered by this design document and are deferred to future work:

• Admin CLI for manual failover: xce hostcontrol --failover is not designed here. Manual failover is exercised in P3-6 via ha_demote_to_standby() called from a test harness. A production CLI surface for this operation is deferred to future work
• WAL compaction beyond checkpoint trimming: Checkpoint-based WAL segment deletion is specified, but online compaction (merging small segments, garbage-collecting within segments) is out of scope
• Multi-data-center replication: All replication targets are same-DC controllers connected via a dedicated management network. Cross-DC replication would require bandwidth-aware batching and is not addressed
• Monitoring/alerting integration: P3-7 lists metrics to expose, but the integration with Prometheus, Nagios, or other monitoring systems is out of scope
• Runtime configuration changes: Adding/removing standby candidates at runtime, changing persist_config_t values without restart, and online checkpoint interval adjustment are not designed here
• Partial WAL re-sync: After partition healing, the standby performs full re-sync from the primary. Incremental re-sync (sending only divergent entries) is out of scope
• Snapshot compression: Snapshots are stored as uncompressed Cap'n Proto. Compression (zstd, lz4) is a potential optimization deferred to future work
• Config version hash in heartbeat messages (Part 10.8): Deferred to P3 (operational tooling). Not needed for core discovery, failover, or connection behavior. The configVersion field in the Request Cap'n Proto schema and the CRC freshness signal are removed from P1 scope.
• Degraded-mode restricted state (Part 7.4 repl_degraded_timeout_s): Removed from scope. The "restricted mode" where the primary refuses WAL_JOB_SUBMIT after 300s without a standby ACK is a product policy decision, not a persistence or HA requirement. The existing sync+timeout+local-WAL fallback already meets the stated durability goal. Monitoring and operator escalation handle sustained degradation. repl_degraded_timeout_s is removed from persist_config_t — if HA policy needs a degradation tolerance field, it belongs in ha_config_t.
• MAX_HA_CANDIDATES = 16: Reduced to 4 in include/config.h. The design is explicitly validated for ≤3 controllers. MAX_STANDBYS reduced from 15 to 3. Setting structural limits far beyond operational needs is speculative generality.

Review Resolutions

The following issues were identified during multi-perspective review and resolved in this version:

ID	Finding	Resolution
R-1	traversalLimitInWords set to file_size / 8 provides no security (P0)	Changed to fixed upper bound MAX_SNAPSHOT_BYTES / 8 words
R-2	Job ID allocator not reset after WAL replay (P0)	Superseded by A6-014: the premise was incorrect — jobids are allocated by the Go API server's idgen, not the controller. The controller is a passive consumer that restores jobs at their original IDs during recovery. No max_replayed_jobid tracking or idgen_set_next() call is needed. See recovery step 3 note and A6-014.
R-3	acked_lsn not declared _Atomic in struct (P0)	Changed standby_conn_t.acked_lsn to _Atomic uint64_t
R-4	ha.role not atomic — role-change window incomplete (P1)	Changed ha_state_t.role to _Atomic ctrl_role_t, documented acquire/release requirements
R-5	Missing hook idempotency guard (P0)	Added hooks_registered flag to ha_state_t, required in ha_promote_to_primary()
R-6	checkpoint_mutex missing from persist_t (P1)	Added pthread_mutex_t checkpoint_mutex field
R-7	WALEntry Cap'n Proto schema confused with on-disk format (P1)	Clarified that WALEntry is replication-only; on-disk uses raw C wal_entry_hdr_t
R-8	Snapshot backward compatibility not specified (P1)	Added backward-compatibility note to recovery step 6
R-9	Split-brain detection condition unspecified (P1)	Added detection specification in Part 7.4
R-10	Heartbeat interval minimum inconsistent (P1)	Standardized to heartbeat_interval_ms >= 2000 across all sections
R-11	repl_degraded_timeout_s in wrong config struct (P1)	Removed from scope (see Out of Scope above)
R-12	WAL_CHECKPOINT entries skipped creates CRC chain gaps (P2)	Documented in Part 6.4 that skipped checkpoint entries produce expected single-entry CRC gaps
R-13	Snapshot total_size used for direct malloc without boundary check (P1)	Added explicit requirement: validate total_size <= MAX_SNAPSHOT_BYTES BEFORE malloc()
R-14	CRC32 for WAL integrity not tamper-evident (P2)	Documented threat model: CRC32 protects against accidental corruption; HMAC-SHA256 would be required for adversarial tamper resistance
R-15	WAL payload logging relies on code review only (P1)	Added recommendation for structural enforcement via opaque wal_payload_t wrapper type
R-16	vstore COW growth re-check uses OOB a->data[old_caps].inuse (P0)	Fixed to check a->caps <= old_caps (capacity comparison) instead of OOB memory access; re-read free index after growth
R-17	Standby WAL write return value unchecked before ACK (P0)	Added return value check on persist_wal_replicated_entry() — disconnect on failure, no ACK on error
R-18	Chained CRC absent from replication stream (P0)	Added hdr.prev_crc validation against ha->last_received_crc on standby; last_received_crc field added to ha_state_t
R-19	Replicated entry LSN handling unspecified (P1)	Added persist_wal_replicated_entry() function — writes primary's LSN/CRC directly, updates current_lsn and last_crc to match
R-20	seg_offset in-memory and lost on crash — recovery trust boundary undefined (P1)	Clarified: recovery scans full segment file using CRC validity, not seg_offset — seg_offset is runtime-only for fsync tracking
R-21	WAL segment rotation prev_crc == WAL_CRC_NONE handling during recovery unspecified (P1)	Documented: WAL_CRC_NONE at segment-aligned position is valid chain break, not corruption
R-22	COW vstore find-slot-and-write not atomic — concurrent slot overwrite (P0)	Wrapped entire COW-mode find-slot-and-write in mylock_lock(&priv->lock); simplified growth path
R-23	vput grow path calls xzarray_ensure_items() without mutex during COW (P0)	Resolved by R-22 fix — entire COW write path (including growth) is now under the inner mutex
R-24	Post-growth slot assignment a->caps - 1 double-claimed by concurrent threads (P0)	Resolved by R-22 fix — mutex held around growth + assignment prevents double-claim
R-25	Shadow-only iteration is prose discipline, not enforced API (P0)	xzarray_freeze() builds per-kvmgr slot_key_t[] under write lock — serialization reads this array, not kvtab (Part 3.8)
R-26	malloc(0) for WAL_CHECKPOINT entries → NULL dereference (P0)	Added payload_len > 0 guard before malloc; apply_wal_entry() receives NULL for zero-payload entries
R-27	Fallback to older snapshot with trimmed WAL → silent stale boot (P0)	Added WAL-existence verification before accepting older snapshot fallback
R-28	hdr.term > ha->term + 1 permanently disconnects multi-promotion standby (P0)	Replaced with term-accept-and-update: primary's term is authoritative, standby updates ha->term to match
R-29	_Atomic ctrl_role_t on enum type — C11 semantics implementation-defined (P0)	Changed to _Atomic int32_t with cast to ctrl_role_t at access point
R-30	handle_jobsubmit() ignores xce_run_hooks() return value — false ACK on WAL failure (P0)	Added return value check: -EIO aborts submit, -ETIMEDOUT accepts with degraded durability warning
R-31	malloc(hdr.payload_len) NULL check missing in standby receiver (P0)	Added OOM check; break connection on failure
R-32	No heartbeat during persist_recover() on new primary → split-brain (P1)	Added step 5.5: start heartbeat before persist_recover() with new term, defer cluster_start()
R-33	Catch-up from dead primary specified in promotion step 3 (P1)	Removed catch-up-from-dead-primary — async gaps acknowledged as lost, self-heal via worker reconnect
R-34	No total_wal_bytes field for inline checkpoint trigger (P1)	Added total_wal_bytes counter to persist_t struct
F-1	_Atomic uint32_t cast of unsigned int :1 bitfield — C11 prohibits atomic operations on bitfields (P0)	Changed _Atomic uint32_t to _Atomic _Bool for inuse; added xzarray_private_t.free_idx cache for O(1) slot allocation
F-2/F-5	ha->term missing from ha_state_t — no authoritative term field in struct (P0)	Added _Atomic uint64_t term to ha_state_t; all accesses use atomic_load/store_explicit with acquire/release
F-3	Plain c->ha.role == reads against _Atomic int32_t in pseudocode (P0)	All pseudocode role comparisons updated to use (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire)
F-4	Unbounded term-accept — standby with corrupted term could overflow (P1)	Added MAX_TERM_JUMP (10000) ceiling; terms exceeding ceiling cause disconnect
F-6	Heartbeat starts before cluster_start() — subsystems not initialized (P1)	Step 5.5 clarified: heartbeat starts before cluster_start() with new term; messages contain only term and role until step 8
F-7	persist_t.key_snapshot lifecycle broken — xzarray_freeze() has no access to persist_t, wrong type (P0)	Redesigned as per-kvmgr slot_key_t[] built during xzarray_freeze() under write lock, freed after serialization
F-8	total_wal_bytes decrement on WAL trim not specified (P1)	Added trim accounting: total_wal_bytes decremented by fstat(fd).st_size on segment deletion
F-9	Shadow use-after-free during abnormal shutdown after 5-second deadline (P1)	Added pthread_join() of checkpoint thread before freeing cluster state on shutdown
F-10	promotion_retry_count data race between watchdog and vcpu threads (P1)	Changed to _Atomic int with relaxed atomic_load/store
F-11	a->pos = priv->shadow_caps truncates size_t to int (P1)	Changed loop variable from int idx to size_t idx in vstore COW scan
F-12	Async WAL write failure silently swallowed — no logging (P1)	Added WARN() log on persist_wal_replicated_entry() failure
F-13	Step 5.5 references persist_recover() but promotion doesn't call it (P1)	Changed wording to reference cluster_start() instead
F-14	apply_wal_entry() return value ignored — corrupted entry silently accepted (P0)	Added return value check; break replication stream on failure
F-15	replicator_t.ack_lock is mylock_t but pthread_cond_timedwait requires pthread_mutex_t* (P0)	Changed to pthread_mutex_t — same rationale as checkpoint_mutex (R-6)
F-16	ha->in_catch_up referenced in ACK gate but absent from ha_state_t (P0)	Added in_catch_up field to ha_state_t in Part 5.3 and Appendix A.4 with lifecycle spec
F-17	SIGUSR1 handler calls non-async-signal-safe ha_demote_to_standby() (P0)	SIGUSR1 sets atomic_int failover_requested only; watchdog thread executes actual demotion
F-18	vfreeze_fn_t signature incompatible with slot_key_t capture (P1)	Moved key capture to caller-side kvmgr_capture_keys() after vfreeze() returns, under write lock
F-19	P0-4 step says traversalLimitInWords proportional to file_size / 8, contradicting R-1 (P1)	Updated P0-4 to fixed upper bound MAX_SNAPSHOT_BYTES / 8 words
F-20	1 << attempt signed overflow + shared rng_seed data race in backoff_jitter (P1)	Saturating shift at 30; __thread per-thread seed with pthread_self init
A5-001	Checkpoint timer described as "1-second delaymgr timer" but WAL fsync and heartbeat require sub-second granularity (P1)	Clarified P0-3: checkpoint timer uses delaymgr (1s poll is adequate); WAL fsync and heartbeat use dedicated pthreads for sub-second precision
A5-002	Capnp schema missing opcodes added in P1 milestones (P1)	Added heartbeat @26, steppingDown @27, replSnapshot @28, replCatchUpDone @29 to Part 5.10 schema
A5-003	WAL CRC chain WAL_CRC_NONE at segment start is permissive — could mask corruption (P1)	Clarified: WAL_CRC_NONE at segment boundaries (writer-side rotation, after fsync succeeded) is valid and required; WAL_CRC_NONE encountered mid-segment during catch-up replay is a protocol error
A5-004	Hook list mutation during demotion — xce_remove_hook() could race with concurrent iteration (P1)	Added lock-free/RCU-style removal spec: xce_remove_hook() sets disabled flag, defers memory reclamation
A5-005	xce_connect_cluster() returns int fd but TLS-first architecture requires SSL* (P1)	Changed return type to SSL*; added xce_disconnect_cluster(SSL*); updated xce_reconnect to SSL**
A5-006	GC use-after-free: checkpoint traverses deleted entities without protection (P1)	Added GC use-after-free mitigation section: per-operation READ lock hold for P0; epoch-based reclamation as P3 optimization
A5-007	xzarray_freeze() OOM rollback of a->pos is racy if readers observed new value (P2)	Noted as P2 — freeze under write lock makes rollback window single-threaded
A5-008	ha->in_catch_up plain int — data race between iomgr and watchdog threads (P1)	Changed to _Atomic int in_catch_up with documented access patterns and memory ordering
A5-009	promotion_failed should be _Atomic int for watchdog/vcpu access (P2)	Already _Atomic; existing promotion_retry_count atomic is sufficient
A5-010	Missing OP_steppingDown handler in P1-2 milestone (P1)	Added P1-2 handler spec: term validation, sender verification, priority-based promotion
A5-011	P0/P1 milestone dependency ordering unspecified — implementer could build P1 before P0 (P2)	Dependencies are implied by sequential section numbering; explicit DAG not in scope
A5-012	primary_ep pointer lifecycle on standby — no documented safe access pattern (P1)	Added lifecycle spec: iomgr-owned borrowed reference, NULL-before-free under ha->lock, all access must check NULL
A5-013	repl_remove_standby() leaves stale slot data before nstandbys decrement (P1)	Added cleanup requirement: zero slot fields before decrementing nstandbys
A5-014	Async hook error handling unspecified — what happens when AFTER_EVENT hook returns error for async-replicated events (P1)	Added error convention spec: same as sync hooks (-EIO aborts, -ETIMEDOUT degraded durability)
A5-015	Concurrent snapshot catch-up memory — MAX_STANDBYS × MAX_SNAPSHOT_BYTES aggregate (P2)	Added aggregate memory budget analysis: primary holds negligible per-chunk memory; standby reassembly buffer bounded by single MAX_SNAPSHOT_BYTES
A5-016	ha_check_heartbeat() uses plain ha->role read without atomic_load_explicit (P1)	Fixed to use (ctrl_role_t)atomic_load_explicit(&ha->role, memory_order_acquire) — consistent with F-3 fix
A5-017	Appendix A.4 in_catch_up field typed as plain int instead of _Atomic int (P1)	Fixed to _Atomic int to match Part 5.3 definition and documented access pattern
A5-018	Stale "COW mode growth" snippet (pre-R-22 intermediate design) present alongside final xzarray_store() listing — contradicts the mutex-first approach	Removed the superseded snippet; added a forward-reference note pointing to the definitive R-22 implementation below
A5-019	Plain reads/writes to _Atomic uint64_t ha->applied_lsn in standby receiver pseudocode (Part 6.4) — contradicts documented require of acquire/release ordering	Replaced plain accesses with atomic_load_explicit(acquire) into local applied variable; replaced write with atomic_store_explicit(release)
A5-020	Plain !ha->in_catch_up in standby receiver pseudocode (Part 6.4) — contradicts documented _Atomic int field with required acquire ordering	Changed to !atomic_load_explicit(&ha->in_catch_up, memory_order_acquire)
A5-021	ha_check_heartbeat() reads ha->primary_ep and ha->last_heartbeat without ha->lock — contradicts lifecycle comment requiring mutex; ha_state_t missing lock field entirely	Added pthread_mutex_t lock field to ha_state_t (Part 5.3 and Appendix A.4); updated ha_check_heartbeat() to snapshot primary_ep and last_heartbeat under ha->lock before use
A5-022	Thaw function reads _Atomic _Bool inuse via plain !x.inuse and x.inuse — C11 plain access gives seq_cst ordering inconsistently with relaxed ordering used by surrounding explicit stores	Changed all three plain inuse reads in xzarray_thaw() to atomic_load_explicit(memory_order_relaxed) matching the explicit stores in the same function
A6-001	Part 5.5 code sketch registers heartbeat check via delay_task(&c->delaymgr, ...) — directly contradicts the same section's requirement for a dedicated pthread	Replaced delaymgr sketch with a reference to the dedicated-pthread requirement; noted delaymgr's 2-second tick granularity (precision=2 in cluster.c) as the reason
A6-002	Remaining plain ha->applied_lsn read in Part 6.4 CRC check block (if (ha->applied_lsn == 0)) missed in A5-019 pass	Changed to atomic_load_explicit(&ha->applied_lsn, memory_order_acquire) == 0
A6-003	Part 9.4 redirect sends primary_host/primary_port without nil-check — on cold start before any heartbeat, standby sends garbage redirect (empty host, port 0)	Added guard: if primary_host[0] == '\0' || primary_port == 0, reply with E_XCE_NOMASTER instead of redirect
A6-004	Part 5.4 election probe peer-cert validation describes only application-layer CN check — missing SSL_VERIFY_PEER TLS-layer requirement that Part P2.5-2 already mandates for xce_connect_cluster()	Added explicit SSL_CTX_set_verify(SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT) requirement to Part 5.4; cross-referenced P2.5-2
A6-005	persist_t has last_crc for local WAL chain but no last_replicated_crc for replication chain — WAL_CHECKPOINT entries break the standby's chained-CRC validation unless the primary skips them from the replicated chain	Added last_replicated_crc field to persist_t (Appendix A.1) with documentation of when it is updated vs last_crc
A6-006	slot_key_t.key is const char* (borrowed pointer into kvtab) — kvtab can rehash and move strings after the write lock is released during lock-free serialization, causing use-after-free	Changed slot_key_t.key to char key[XCE_MAXNAME_LEN] (owned deep copy populated by strndup in kvmgr_capture_keys); updated the serialization snapshot code and comment
A6-007	P0-10 description has no prerequisite dependency on P0-9 — implementer can attempt to wire persist_init into cluster.c before Cap'n Proto schemas are generated	Added Requires: P0-9 to P0-10 description
A6-008	P0-0b description did not call out that intf_varray_t and xzarray_private_t struct changes are breaking ABI — partial builds corrupt makeInstance() private section allocations	Added explicit breaking-ABI note to P0-0b; required converting positional initializers to designated style before adding new slots; required full ninja -C build clean rebuild
A6-009	Part 3.9 lock hierarchy described xzarray mylock_t as "not taken in vput" without distinguishing normal/COW/thaw modes — contradicted Part 3.5's COW-mode inner-mutex requirement (R-22)	Replaced single-sentence description with three-mode table: (a) normal — inner mutex skipped; (b) COW vstore — inner mutex required; (c) thaw — write lock sufficient
A6-010	Part 6.2 "constant reconciliation" note still referenced old MAX_HA_CANDIDATES = 16 and MAX_STANDBYS = MAX_HA_CANDIDATES - 1 derivation, contradicting Out of Scope reduction to 4/3	Updated to state both constants defined directly in config.h (MAX_HA_CANDIDATES = 4, MAX_STANDBYS = 3); removed derived formula
A6-012	kvmgr_capture_keys() called in Part 3.8 checkpoint flow but never defined anywhere in the document	Added full function signature and documentation to Appendix A.1
A6-013	Recovery step 3 required idgen_set_next() — a function that does not exist; idgen lives in Go API server (clib.go), not the controller	Removed idgen_set_next() requirement; documented that the controller never allocates jobids (assigned by API server before submission) and recovery only restores jobs at their original IDs from the WAL
A6-014	R-2 "Job ID allocator reset" based on incorrect assumption that the controller allocates jobids	Corrected: jobids are allocated by the Go API server's idgen; the controller is a passive consumer. Recovery single-threaded note added to step 3
A6-015	xce_remove_hook() does not exist in hook.c — required for hook unregistration on demotion	Replaced with xce_disable_hook() approach: set entry->fn = NULL in place; xce_run_hooks() skips NULL entries. No RCU/deferred-reclamation complexity; bounded memory (3 persist hooks max). Requires adding xce_disable_hook() to src/lib/hook.c
A6-016	Part 6.4 did not clarify that apply_wal_entry() on standby is effectively single-threaded	Added serialization note: all OP_replEntry messages arrive over a single endpoint; iomgr/vtask endpoint_block_tasks() serializes dispatch — xkvmgr_add() calls in apply_wal_entry() are never concurrent
A6-017	delaymgr uses sleep(precision) with precision=2 (integer seconds) — makes 1-second checkpoint poll (P0-3) impossible, and adds up to 2s jitter to heartbeat detection	Specified delaymgr enhancement: replace sleep() with usleep(), change precision field to precision_ms (default 200ms). Allows 1-second checkpoint poll (5 ticks × 200ms) without a dedicated pthread. P0-3 notes this as a prerequisite
A6-018	Part 10.8 listed NFS as a valid distribution mechanism for cluster.yaml, contradicting the no-shared-filesystem architecture	Removed NFS from distribution options; documented that XCE assumes no shared filesystem — cluster.yaml must be distributed via config management tooling (Ansible, Salt, deployment pipeline) on each node's local storage
A6-019	Product-lens review flagged "split-brain data loss not surfaced at occurrence" as a P1 omission	Reframed as an intentional CAP trade-off: XCE chooses AP (availability + partition tolerance) over CP (consistency). Real-time split-brain detection is impossible without quorum — implementing it is equivalent to Raft. The split_brain_risk_accepted: true key is the operator's explicit CAP position acknowledgment. Post-partition visibility is provided by ctrl_split_brain_detected counter and ERROR-level logs
A6-020	AFTER_EVENT hook return value used to detect persistence failures — breaks if any subsequent hook overwrites rc; requires transaction rollback to undo in-memory state on failure	Replaced with side-channel pattern: persistence hook always returns 0 (AFTER_EVENT failures tolerated per XCE hook design; pre-hook BEFORE_EVENT failures abort, post-hook AFTER_EVENT failures are logged). WAL result stored in persist_t.last_sync_rc; handle_jobsubmit() reads it directly after xce_run_hooks(AFTER_EVENT) returns. Field added to persist_t struct.
A6-021	sizeof(xzarray_item_t) change from unsigned int :1 to _Atomic _Bool flagged as backward compatibility concern	Dismissed: no released version of XCE exists; all components are rebuilt together. The struct layout can change freely. calloc() sizing in freeze and xzarray_ensure_items() automatically uses the new size after recompile.
A6-022	No recovery path for corrupt-all-snapshots fatal abort scenario — "restore from backup" is undefined	Added snap_keep_n config (default 2) to retain previous snapshots as last-known-good fallback. Recovery walks snapshots newest-to-oldest, falling back if checksum fails and WAL can bridge the gap. snap_keep_n added to persist_config_t and Cap'n Proto PersistConfig schema.
A6-023	rtti source location flagged as unknown — P0-0b appeared to lack an implementation path	Clarified: rtti is maintained by the XCE team (not a third-party dependency). Source is accessible via the project's flake.nix inputs. No external coordination required.
A6-024	Part 1.2 said restore_worker() "signature will need to be updated" — contradicted P0-12 which says "signature is unchanged"	Fixed Part 1.2: signature is unchanged; worker_from_capnp() adapter in P0-12 bridges Cap'n Proto to restore_worker(). Recovery calls the adapter, not restore_worker() directly
A6-025	P2.5 "parallel with P2" claim misleading — P2.5-7 (API server discovery) only has value when P2 replication is live	Clarified: P2.5 code can be developed in parallel with P2; P2.5-7 should be deployed together with P2 for full functionality. P2.5 alone provides worker/scheduler auto-reconnect but not replication continuity
A6-026	COW active array allocated at exact shadow_caps — any vstore() call immediately triggers realloc() which moves a->data, causing dangling pointers in concurrent vget() callers holding READ lock	Pre-grow active array to 1.5 × shadow_caps + delta at freeze time. [shadow_caps, cow_caps) range is immediately available for vstore() — no realloc needed in the common case. If full, vstore() returns -ENOMEM (WARN + client retry after COW closes)
A6-027	P0-8a conditional fallback had no merge-order gate — deferred cleanup path could be permanently skipped	Split into P0-8a (fallback with TODO markers, no P0-0a dependency) and P0-8b (deferred path, explicitly requires P0-0a merged and green before starting)
A6-028	P0-7 reply-after-hooks audit scoped only to handle_jobsubmit() — other handlers with AFTER_EVENT + xce_reply have the same ordering gap	Expanded P0-7 scope to full codebase audit: grep all xce_run_hooks(AFTER_EVENT + xce_reply pairs across all handler files; document each handler's reply ordering decision
A6-029	SIGUSR1 manual failover has no access control — any same-UID process can trigger failover	Documented: acceptable for initial implementation on administratively restricted management nodes. Future xce hostcontrol --failover CLI (SSH-authenticated) is the preferred interface. Added note about future --disable-sigusr1-failover startup flag
A7-001	Part 3.10 "ticks" framing for delay_task() was wrong — delay.c uses time_t arithmetic (seconds), not tick counts; tick description contradicted P0-3 which correctly says delay=1 second	Removed ticks framing; clarified delaymgr enhancement as nanosleep precision change only (delay_task parameters remain seconds throughout); corrected delay_task call to delay=1 second
A7-002	snap_keep_n added to persist_config_t C struct but absent from PersistConfig Cap'n Proto schema in Appendix A.5 — field could never be read from config	Added snapKeepN @6 :UInt32 = 2 to PersistConfig schema; updated P0-9 step to include the field
A7-003	COW vstore() growth path called xzarray_ensure_items() (realloc) despite pre-grow design intent — realloc moves a->data, dangling vget() pointers under READ lock	Changed growth path to return -ENOMEM instead of realloc; documented rationale; -ENOMEM is transient (COW < 100ms) — client retries
A7-004	xce_disable_hook() and xce_run_hooks() NULL guard had no implementation phase step — ha_demote_to_standby() calls a function not yet in hook.c	Added P1-2a step: add xce_disable_hook() to src/lib/hook.c and add if (!it->fn) continue; guard to xce_run_hooks(); marked as prerequisite for P1-3a
A7-005	Part 3.8 said checkpoint_lsn = max LSN across kvmgrs — should be minimum; using max trims WAL entries needed to replay the earliest-frozen kvmgr	Changed to minimum LSN; documented why minimum is correct for cross-kvmgr WAL bridging
A7-006	ha_state_t.promotion_failed was plain int while promotion_retry_count was _Atomic int — mismatch, data race on watchdog vs vcpu reads	Changed to _Atomic int promotion_failed in both Part 5.3 and Appendix A.4 struct definitions
A7-007	P2.5 intro still said MAX_STANDBYS = MAX_HA_CANDIDATES - 1 (defined in repl.h, currently 15) — stale text contradicting A6-010	Fixed to MAX_STANDBYS = 3 standalone in include/config.h; removed derived-formula reference
A7-008	Part 7.4 Degraded Mode contained full implementation spec for repl_degraded_timeout_s despite Out of Scope removing it — contradiction caused scope confusion	Added explicit callout box at top of degraded-mode timeout paragraph marking it out of scope; replaced full spec with concept summary
A7-009	Part 7.4 step 4 said self-healing was via "WAL replay" — misleading after a checkpoint trims the degraded WAL, where full snapshot transfer is required	Added "or full snapshot transfer" to the self-healing description
A7-010	Part 3.11 snapshot format diagram missing version field and headerChecksum field — reviewers and implementers could miss these required fields	Added version: UInt32 (=1) and headerChecksum: CRC32(header fields) to the SnapshotHeader diagram
A7-012	Part 10.8 contained full configVersion CRC hash implementation spec despite Out of Scope deferring it to P3	Added deferred callout; replaced full spec with concept summary; noted configVersion @23 is reserved
A7-013	snap_keep_n fallback description implied protection against completed-checkpoint corruption — actually only protects against crash-during-checkpoint	Clarified: fallback works only when WAL bridging the old→new snapshot gap still exists (crash-during-checkpoint scenario); aborts with fatal error when WAL is already trimmed
A7-014	P0-9 phase step description did not mention snap_keep_n/snapKeepN schema addition	Added snapKeepN @6 :UInt32 = 2 explicitly to P0-9 step description
A7-015	last_sync_rc was a shared int on persist_t — concurrent vcpu threads race on reads/writes, causing false ACKs on WAL failures	Removed from persist_t; replaced with stack-allocated int sync_rc in handle_jobsubmit() passed as extra va_list argument to xce_run_hooks() and read by persist_job_hook_fun() — per-request, no sharing across threads
A7-016	xce_disable_hook() could grow hook list indefinitely (new NULL slot per promote/demote cycle) and fn pointer reads were non-atomic — use-after-NULL-write race	Made hook_item_t.fn _Atomic hook_fn_t; xce_disable_hook() stores NULL with release; xce_run_hooks() loads into local with acquire then checks+calls local; xce_add_hook() reuses existing NULL slots before appending — bounds list to max-ever-registered slots
A7-017	JOB_STAT_TRANSIENT had no TTL or auto-resolution path — permanently stuck scheduling slots if worker never reconnects	Added transient_job_timeout_s config (default 300s) to persist_config_t and PersistConfig schema (@7); after timeout, job → JOB_STAT_EXIT with reason "worker did not reconnect"; worker → WORKER_UNAVAILABLE; increments ctrl_transient_jobs_stuck
A7-018	Array job batch sync mechanism said "accumulate in a single hook call" but AFTER_EVENT fires once per sub-job — no mechanism for cross-call accumulation	Specified heap-allocated persist_batch_ctx_t* passed via same va_list slot as sync_rc for array submits; accumulates WAL entries across calls; flushes every 100 sub-jobs; bounded by per-request allocation (no shared state)
A7-019	ha_init() mutex initialization timing relative to heartbeat thread and promotion path was unspecified	Added explicit ordering note: ha_init() must run before am_i_primary(); mutex is initialized once at startup and never re-initialized on re-promote cycles (re-init with waiters = UB)
A7-020	reconciliation_progress denominator (workers_at_old_primary) was undefined — implementers could not compute it	Specified as: count of workers in promoted standby's kvmgr with status NOT including XCE_WORKER_UNAVAIL or XCE_WORKER_HIDDEN at the moment of promotion — derivable from replicated state, no data from dead primary needed
A7-021	delaymgr precision enhancement had no dedicated phase step — was only noted as a prerequisite inline in P0-3	Added P0-2b step: rename precision → precision_ms, replace sleep() with nanosleep(), audit all delaymgr_init() callers, update cluster.c:604 to precision_ms=200
A8-001	"Why this works" paragraph in Part 8.3 still referenced removed c->persist.last_sync_rc field	Updated to reference per-request sync_rc stack variable and document thread-private design
A8-002	persist_batch_ctx_t was referenced in Part 8.3 and A7-018 but never defined — no struct or Appendix entry	Added full typedef struct persist_batch_ctx_t to Appendix A.1 with all fields documented
A8-003	va_list slot ambiguity: &sync_rc (int*) and persist_batch_ctx_t* described as occupying "same slot" — C UB when va_arg reads different types at same position	Replaced with two distinct va_list arguments (always present): &sync_rc first, batch_ctx (NULL for scalar, non-NULL for array) second; hook always reads both with typed va_arg calls; call site updated with persist_batch_ctx_alloc() example
A8-004	P1-2a listed after P1-2 in phase table but is a prerequisite of P1-2 — table order inverts the dependency	Moved P1-2a before P1-2; added Requires: P1-2a note to P1-2; added _Atomic hook_fn_t and atomic_store/load_explicit requirements to P1-2a
A8-005	P0-9 schema step missing transientJobTimeoutS @7 and transientSince field for Job schema	Added both to P0-9; transientSince is required for the transient_job_timeout_s GC check in job_should_retain()
A8-006	transient_job_timeout_s auto-termination had no implementation path — job_should_retain() returns true for JOB_STAT_TRANSIENT (not terminal per job_is_terminal())	Added transient-job TTL check to job_should_retain() before the job_is_terminal() guard; auto-terminates to JOB_STAT_EXIT with WARN log + ctrl_transient_jobs_stuck increment
A8-007	ctrl_transient_jobs_stuck metric named in config comment and A7-017 but absent from P3-7 metric inventory	Added counter definition to P3-7 with description and WARN log requirement
A8-008	P0-3 had no explicit Requires: P0-2b — the dependency was only in P0-2b's description and prose	Added Requires: P0-2b to P0-3 step row

---

Appendix: Data Structures & Signatures

A.1 Persistence Module

/* persist.h — see Appendix A.10 for system-level tunable constants in include/config.h */

#include "config.h"   /* WAL_FORMAT_VERSION, MAX_SNAPSHOT_BYTES, MAX_WAL_PAYLOAD_BYTES,
                         WAL_ENTRY_TYPE_MAX, WAL_HDR_CRC_SIZE, DEFAULT_WAL_SYNC_MS, etc. */

typedef struct {
    int             wal_fd;         /* Current WAL segment fd */
    uint64_t        current_lsn;    /* Next LSN to assign */
    uint64_t        last_crc;       /* Previous record's CRC (chained) */
    uint64_t        term;           /* Current election term */
    char            seg_path[XCE_MAXPATH_LEN];  /* Current segment path */
    uint32_t        seg_offset;     /* Write offset in current segment */
    uint64_t        checkpoint_lsn; /* LSN of last completed checkpoint (for WAL trim guard) */
    persist_config_t config;
    cluster_t*      cluster;        /* Back-reference */
    mylock_t        lock;           /* Protects WAL write + LSN */
    atomic_int      checkpoint_in_progress; /* Non-zero while checkpoint is active */
    atomic_int      checkpoint_requested;   /* Set by inline WAL size trigger (Part 3.11);
                                               cleared by checkpoint timer after completing */
    pthread_cond_t  checkpoint_done;        /* Signaled when checkpoint completes or is canceled */
    pthread_mutex_t checkpoint_mutex;       /* Dedicated mutex paired with checkpoint_done.
                                               MUST NOT be persist.lock (which is mylock_t) —
                                               pthread_cond_timedwait requires pthread_mutex_t*.
                                               See Part 3.8 for rationale. */
    pthread_t       snap_thread;            /* ID of the active serialization pthread, or 0 if
                                               none is running. Set by persist_checkpoint_timer_fn()
                                               when it spawns the serialization pthread; read by
                                               cluster_fini() for pthread_join() on the deadline-
                                               exceeded shutdown path. Zero-initialized (memset
                                               in persist_init()). cluster_fini() MUST check
                                               snap_thread != 0 before calling pthread_join() —
                                               pthread_join(0, NULL) is undefined behavior. */
    uint64_t        total_wal_bytes;        /* Cumulative WAL size across all segments.
                                               Updated on each write, adjusted on segment
                                               rotation and WAL trimming. Used by the inline
                                               checkpoint trigger (Part 3.11) to decide when
                                               to checkpoint based on snap_wal_size_mb. */
    uint32_t        last_replicated_crc;   /* CRC of the last WAL entry that was sent to standbys.
                                               Distinct from last_crc (which includes WAL_CHECKPOINT
                                               entries that are local-only and never replicated).
                                               Used as prev_crc for the next replicated entry so
                                               that standbys' chained-CRC validation works correctly
                                               across checkpoint boundaries. Updated only when a
                                               non-WAL_CHECKPOINT entry is broadcast to standbys.
                                               See Part 6.4 WAL_CHECKPOINT chain gap handling.
                                               LOCK: protected by persist.lock. All updates to
                                               last_replicated_crc must be made while holding
                                               persist.lock (same lock as last_crc). The WAL fsync
                                               thread (which writes WAL_CHECKPOINT and updates
                                               last_crc) does NOT update last_replicated_crc —
                                               the two fields are always updated on different code
                                               paths, preventing the race where a checkpoint write
                                               interleaves with a replication CRC read. */
    /* NOTE: last_sync_rc was removed (A7-015). WAL write results are now communicated
     * via a per-request stack variable (&sync_rc) passed as an extra va_list argument
     * to xce_run_hooks(AFTER_EVENT). A shared persist_t field races across concurrent
     * vcpu threads processing simultaneous job submits from different API servers. */
} persist_t;

int  persist_init(persist_t* p, persist_config_t* config, cluster_t* c);
int  persist_start(persist_t* p);
void persist_stop(persist_t* p);
void persist_fini(persist_t* p);

/* Flush the current WAL segment to disk (fsync).
 * Called during promotion (Part 5.6 step 3) to ensure all replicated
 * entries are durable before the standby takes over as primary. */
int  persist_flush(persist_t* p);

/* Async WAL write — returns immediately after writing + sending to standbys */
int  persist_wal_async(persist_t* p, uint32_t type, void* payload, uint32_t len);

/* Sync WAL write — writes+fsyncs locally, then waits for at least one standby ACK.
 * Return values distinguish the two failure modes:
 *   0          — success (written + fsynced + ACKed)
 *  -EIO        — local write or fsync failed (permanent; do not send to client)
 *  -ETIMEDOUT  — wrote locally but no standby ACKed within sync_timeout_ms
 *                (transient; client may retry; entry is locally durable)
 * Callers must treat -EIO and -ETIMEDOUT differently: -EIO aborts the operation,
 * -ETIMEDOUT can be surfaced to the client as a "degraded durability" warning.
 *
 * IMPORTANT: -ETIMEDOUT is a durability warning, not a consistency error. When
 * -ETIMEDOUT is returned, the in-memory mutation (e.g., job_submit()) has already
 * completed — the job ID is assigned, the kvmgr entry exists, and the local WAL
 * entry is fsynced to disk. There is NO rollback of the in-memory state. The
 * persistence hook call from AFTER_EVENT cannot undo the business logic that
 * preceded it. Callers that receive -ETIMEDOUT should still reply to the client
 * with the result (e.g., the jobid), and log the degraded durability event.
 * The entry will be replicated to standbys when they reconnect (see Part 7.4
 * degraded mode self-healing). */
int  persist_wal_sync_append(persist_t* p, uint32_t type, void* payload, uint32_t len);

/* Write a replicated WAL entry — uses the primary's LSN and CRC.
 * Called by the standby replication receiver (Part 6.4) to write entries
 * received from the primary. Unlike persist_wal_sync_append/async, this
 * function does NOT assign a new local LSN. Instead it:
 *   1. Writes hdr directly (primary-assigned lsn, crc, prev_crc) to the WAL
 *   2. Updates p->current_lsn = hdr.lsn so the next local entry gets hdr.lsn + 1
 *   3. Updates p->last_crc = hdr.crc so the chained CRC chain remains valid
 *   4. Updates p->term = hdr.term to stay in sync with the primary's term
 *   5. fsyncs only if hdr.type == WAL_JOB_SUBMIT (sync replication)
 * Must NOT be called on the primary — use persist_wal_async or persist_wal_sync_append.
 * Returns 0 on success, -errno on I/O failure.
 * IMPORTANT: After calling this, the caller must NOT advance its own LSN
 * counter — p->current_lsn is already set to hdr.lsn. */
int  persist_wal_replicated_entry(persist_t* p, const wal_entry_hdr_t* hdr, const void* payload);

/* Checkpoint — dual zarray freeze + serialize + thaw + GC */
int  persist_checkpoint(persist_t* p, cluster_t* c);

/* Recovery — load latest snapshot + replay WAL */
int  persist_recover(persist_t* p, cluster_t* c);

/* Hook functions — registered in cluster_init() */
int  persist_job_hook_fun(int type, int situation, int detail, void* data, va_list ap);
int  persist_queue_hook_fun(int type, int situation, int detail, void* data, va_list ap);
int  persist_worker_hook_fun(int type, int situation, int detail, void* data, va_list ap);

/* Apply a single WAL entry to in-memory state — used by both persist_recover()
 * and the standby replication receiver. Persistence hooks are registered only
 * on the primary path (after persist_recover() completes in cluster_init(), or
 * in ha_promote_to_primary() step 5 on promotion), so apply_wal_entry() does
 * not fire hooks — the hook system is empty during recovery and on standbys.
 *
 * Semantics:
 *   WAL_JOB_SUBMIT  — calls restore_job(c, jobid, payload) to insert the job at its
 *                     original jobid without allocating a new one. Do NOT call
 *                     job_submit() here — job_submit() assigns a new jobid, which
 *                     breaks replay by creating duplicate or wrong-ID jobs.
 *                     If the jobid already exists, restore_job() returns -EEXIST;
 *                     apply_wal_entry() logs a WARN and skips the entry (non-fatal).
 *   WAL_JOB_START   — updates job stat, start_time, exec_host in-place
 *   WAL_JOB_FINISH  — updates job stat, end_time, exit_code in-place
 *   WAL_JOB_FAIL    — updates job stat, end_time, fail_reason in-place
 *   WAL_JOB_STOP    — updates job stat to JOB_STAT_USUSP in-place
 *   WAL_JOB_RESUME  — updates job stat from USUSP back to PEND in-place
 *   WAL_JOB_UPDATE  — merges partial Job payload into existing job object
 *   WAL_JOB_CKPT    — updates job ckpt_time in-place
 *   WAL_JOB_RUSAGE  — updates job rusage fields in-place
 *   WAL_JOB_PREASON — updates job preason field in-place
 *   WAL_QUEUE_CTRL  — creates or updates queue from QueueConf payload.
 *                     Idempotent: if the queue already exists (matched by name
 *                     or key), update its config in-place; otherwise create it.
 *                     During WAL replay or standby replication, duplicate
 *                     WAL_QUEUE_CTRL entries for the same queue must not create
 *                     duplicates — the create path checks for existing queues first.
 *   WAL_WORKER_CTRL — creates or updates worker from Worker payload.
 *                     Idempotent: if the worker already exists (matched by hostname
 *                     or endpoint), update its state in-place; otherwise create it.
 *                     Same duplicate-safety guarantee as WAL_QUEUE_CTRL.
 *   WAL_CHECKPOINT  — no-op (marker only, no payload)
 *
 * Returns 0 on success, -errno on deserialization or mutation failure.
 * Unknown entry types are WARN-logged and skipped (return 0) to allow forward
 * compatibility with future WAL types added by newer controller versions.
 * WARNING: Skipping unknown entries during recovery silently loses the
 * state changes those entries represent — see recovery forward-compat note. */
int apply_wal_entry(cluster_t* c, const wal_entry_hdr_t* hdr, const void* payload);

/* Array job batch accumulation context — allocated on handle_jobsubmit() stack for
 * array submits (submit->array_size > 1), freed on return from handle_jobsubmit().
 * Passed via va_list to persist_job_hook_fun() instead of int* sync_rc:
 *   - single job: va_list carries (jlist_ptr, &sync_rc)
 *   - array job:  va_list carries (jlist_ptr, batch_ctx_ptr)
 * The hook distinguishes the two cases by checking detail == HOOK_DETAIL_JOB_NEW
 * and submit->array_size — or by inspecting the cluster_t's in-flight array state.
 *
 * The batch context accumulates WAL entries for N sub-jobs (default batch_size=100),
 * then calls repl_broadcast() once and waits for one ACK at the batch's highest LSN.
 * sync_rc is embedded in the context so the hook can report the final WAL result. */
typedef struct {
    int      sync_rc;         /* Final WAL result — 0, -EIO, -ETIMEDOUT, or -EAGAIN.
                                * -EAGAIN: transient COW-full condition (xzarray_store returned
                                * -ENOMEM during COW window). Unlike -EIO, the job must NOT be
                                * marked terminal via job_mark_wal_error_exit() — the in-memory
                                * state may or may not have been inserted depending on where in
                                * the submit path the -ENOMEM occurred. handle_jobsubmit() must
                                * reply with a retriable error code (EAGAIN/EBUSY) so the client
                                * retries after the COW window closes (< 100ms). */
    uint64_t batch_high_lsn;  /* Highest LSN in the current batch (for repl_wait_ack) */
    uint32_t sub_jobs_in_batch; /* Sub-jobs accumulated in the current batch */
    uint32_t batch_size;      /* Flush threshold (default 100) */
    uint32_t total_sub_jobs;  /* Total sub-jobs in the array (submit->array_size) */
    uint32_t sub_jobs_done;   /* Sub-jobs processed so far (hook call count) */
} persist_batch_ctx_t;

/* Key→index mapping used during checkpoint serialization (Part 3.8).
 * Lives in kvmgr.h — used by both kvmgr.c (populate) and persist.c (consume).
 * key[] holds an owned deep copy populated by kvmgr_capture_keys() via strndup,
 * safe to read after the kvmgr write lock is released. */
typedef struct {
    int    index;                   /* shadow slot index */
    char   key[XCE_MAXNAME_LEN];    /* owned copy — not a borrowed pointer into kvtab */
    size_t key_len;                 /* length including null terminator */
} slot_key_t;

/* Capture key→index mappings from a kvmgr's kvtab into a flat slot_key_t[] array.
 * Called during checkpoint freeze (Part 3.8), AFTER vfreeze() returns, while the
 * kvmgr write lock is still held.  Allocates and populates keys_out[0..n_keys_out-1]
 * with one entry per live entity.  Each entry's key field is a deep-copied string
 * (strndup into slot_key_t.key[XCE_MAXNAME_LEN]) — safe to read after the write
 * lock is released during the lock-free serialization phase.
 * Caller must free(keys) after serialization completes.
 *
 * Returns 0 on success, -ENOMEM if allocation fails (in which case *keys_out is NULL
 * and *n_keys_out is 0 — caller must abort the checkpoint and call persist_checkpoint_abort()).
 *
 * NOTE: Supports both KVMGR_KEY_INT and KVMGR_KEY_STRING kvmgrs.  For integer-keyed
 * kvmgrs, the key is formatted as a decimal string before copying.
 * Caller must hold the kvmgr write lock for the entire duration of this call. */
int kvmgr_capture_keys(meta_obj_t* kvmgr, slot_key_t** keys_out, size_t* n_keys_out);

A.2 zarray COW Extensions

/* zarray.h — additions for COW snapshot support */

/* NOTE on struct layout: Part 3.4 defines xzarray_private_t (the RTTI private
 * section) as the home for shadow and shadow_caps fields, accessed via
 * RTTI_OBJ_TO_PRIV(). The struct definition below shows these fields as members
 * of xzarray_t for readability, but in the actual implementation they live in
 * the private section and are accessed as priv->shadow and priv->shadow_caps.
 * The RTTI type system allocates the private section alongside each instance.
 * This is the same pattern used by xzarray_private_t.lock (mylock_t). */

typedef struct {
    int             count;
    size_t          caps;
    size_t          delta;
    size_t          pos;     /* next scan position for vstore (was int — size_t to match shadow_caps) */
    xzarray_item_t* data;        /* Active array */
    destroy_fn_t    killer;
} xzarray_t;

/* Private section — allocated by makeInstance() alongside each xzarray_t instance.
 * Accessed via RTTI_OBJ_TO_PRIV(obj). See Part 3.4 for the design rationale. */
typedef struct {
    mylock_t          lock;
    xzarray_item_t*   shadow;      /* Frozen snapshot (NULL when not in COW) */
    size_t            shadow_caps; /* Shadow array capacity */
} xzarray_private_t;

/* Freeze: swap data → shadow, create new empty active.
   Caller must hold write lock. O(1) — pointer swap only.
   Returns 0 on success, -ENOMEM on allocation failure.
   Public API: vfreeze(var) — RTTI dispatches to xzarray_freeze(meta_obj_t*).
   Do not call xzarray_freeze(xzarray_t*) directly; it is a file-private
   static function that takes meta_obj_t*, not xzarray_t*. */
extern int vfreeze(var);

/* Thaw: merge shadow back into active, apply GC.
   should_retain(arg, item) returns true if entity should survive.
   Caller must hold write lock.
   Frees shadow when done. Returns 0 on success.
   Public API: vthaw(var, should_retain, arg) — RTTI dispatches to xzarray_thaw(meta_obj_t*, retain_fn_t, void*).
   Do not call xzarray_thaw(xzarray_t*, ...) directly; it is a file-private
   static function that takes meta_obj_t*, not xzarray_t*. */
/* retain_fn_t is defined in rtti/typeinfo.h (see Appendix A.9) */
extern int vthaw(var, retain_fn_t should_retain, void* arg);

A.3 Replication Module

/* repl.h — see Appendix A.10 for MAX_STANDBYS and other tunables in include/config.h */

#include "config.h"   /* MAX_STANDBYS, DEFAULT_SYNC_TIMEOUT_MS */

typedef struct {
    endpoint_t*  ep;           /* Full endpoint (SSL, FSM, outq) — uses iomgr/vtask pipeline */
    _Atomic uint64_t acked_lsn; /* C11 atomic — accessed locklessly by WAL trim (min across standbys)
                                 * and written by repl_on_ack() under ack_lock. Use
                                 * atomic_store_explicit(acquire)/atomic_load_explicit(release).
                                 * On LP64 platforms a plain uint64_t is atomically readable,
                                 * but using _Atomic documents the intent and ensures correctness
                                 * if compiled for ILP32. See Part 6.2 for full analysis. */
    int          lagging;
    char         host[XCE_MAXNAME_LEN];
    uint32_t     port;
    _Atomic int  in_catch_up;  /* Primary-side mirror of the standby's ha_state_t.in_catch_up.
                                 * Set to 1 by repl_add_standby() when applied_lsn < primary's
                                 * committed_lsn at join time (standby needs catch-up).
                                 * Cleared to 0 by repl_on_catchup_done() when OP_replCatchUpDone
                                 * is received from this standby.
                                 * Read by repl_wait_ack() to exclude catching-up standbys from
                                 * the "at least one ACK" predicate — standbys in catch-up never
                                 * send OP_replAck for live entries (per Part 6.4 ACK gate).
                                 * Use atomic_load/store_explicit with acquire/release. */
} standby_conn_t;

typedef struct {
    standby_conn_t  standbys[MAX_STANDBYS];
    int             nstandbys;
    mylock_t        lock;
    uint64_t        pending_lsn;
    pthread_cond_t  ack_cond;
    pthread_mutex_t ack_lock;     /* Must be pthread_mutex_t for pthread_cond_timedwait —
                                   * same rationale as checkpoint_mutex (Part 3.8). */
    uint64_t        last_disconnected_lsn[MAX_STANDBYS]; /* Per-slot: acked_lsn at disconnect
                                                            time, for one-interval trim delay */
    struct timespec last_disconnect_time[MAX_STANDBYS];   /* Per-slot: when the standby
                                                            disconnected, for trim delay */
    /* DISCONNECT CLEANUP REQUIREMENT: repl_remove_standby() MUST clear the
     * disconnected slot's fields (set ep=NULL, acked_lsn=0, lagging=0,
     * host[0]='\0', port=0) BEFORE decrementing nstandbys. This ensures
     * the WAL trim guard's min(acked_lsn) loop — which iterates
     * standbys[0..nstandbys-1] — never includes stale data from a
     * previously-disconnected slot. Similarly, repl_add_standby() MUST
     * fully initialize the new slot before incrementing nstandbys.
     * Both operations hold repl.lock, so the loop sees a consistent
     * snapshot. The last_disconnected_lsn[] and last_disconnect_time[]
     * arrays are NOT cleared on disconnect — they persist across
     * reconnections to support the one-interval trim delay.
     *
     * ACKED_LSN INITIALIZATION (CRITICAL): repl_add_standby() MUST initialize
     * the new slot's acked_lsn to persist->checkpoint_lsn (the last completed
     * checkpoint LSN), NOT to 0. Initializing to 0 would cause min(acked_lsn)
     * to equal 0, blocking ALL WAL trimming until the new standby fully catches
     * up from LSN 0 — potentially accumulating gigabytes of un-trimmable WAL.
     * checkpoint_lsn is the safe floor: WAL before checkpoint_lsn is covered
     * by the snapshot the standby will receive during catch-up, so it can be
     * trimmed immediately. WAL from checkpoint_lsn forward is what the standby
     * needs; min(acked_lsn) naturally preserves it until the standby ACKs those
     * entries. repl_add_standby() signature: accepts persist_t* for checkpoint_lsn
     * lookup, or the caller passes checkpoint_lsn directly as a uint64_t. */
} replicator_t;

/* Fan-out send to all standbys */
int  repl_broadcast(replicator_t* r, wal_entry_hdr_t* hdr, void* payload);

/* Wait for at least one standby to ACK the given LSN (sync mode).
 * Implementation MUST check the predicate before waiting to avoid a
 * signal-before-wait race: if repl_on_ack() signals ack_cond before
 * repl_wait_ack() parks on it, the signal is lost. Correct pattern:
 *   lock(ack_lock);
 *   while (r->acked_lsn < target_lsn && !timed_out)
 *       pthread_cond_timedwait(&r->ack_cond, &r->ack_lock, &deadline);
 *   unlock(ack_lock);
 * The while-loop predicate check must be inside the lock, before the first wait.
 *
 * CATCH-UP EXCLUSION: Standbys with in_catch_up != 0 (ha->in_catch_up set in the
 * ha_state_t of the standby's slot) do NOT send OP_replAck for live entries —
 * they gate ACKs until catch-up completes (Part 6.4). Therefore repl_wait_ack()
 * MUST exclude catching-up standbys from its "at least one ACK" predicate.
 * If the only connected standby is in catch-up, repl_wait_ack() falls back to
 * degraded mode immediately (does not wait for the full timeout_ms) and returns
 * REPL_DEGRADED. Callers treat REPL_DEGRADED the same as a timeout — the WAL
 * entry is durable on the primary's local WAL but not yet confirmed on any standby.
 * The per-standby in_catch_up flag is read from standby_conn_t.in_catch_up
 * (a copy of the remote standby's ha->in_catch_up, updated on each OP_replAck
 * that carries the standby's applied_lsn — if applied_lsn < primary's
 * committed_lsn at join time, in_catch_up is set; cleared on OP_replCatchUpDone). */
int  repl_wait_ack(replicator_t* r, uint64_t target_lsn, uint32_t timeout_ms);

/* Called when ACK received from a standby */
void repl_on_ack(replicator_t* r, uint64_t acked_lsn);

/* Add/remove standby connections (via endpoint/iomgr SSL pipeline) */
/* checkpoint_lsn: pass c->persist.checkpoint_lsn — the new slot's acked_lsn is
 * initialized to this value (not 0) to avoid blocking WAL trimming while the
 * standby catches up. See ACKED_LSN INITIALIZATION comment in replicator_t. */
int  repl_add_standby(replicator_t* r, endpoint_t* ep, const char* host, uint32_t port,
                      uint64_t checkpoint_lsn);
int  repl_remove_standby(replicator_t* r, endpoint_t* ep);

/* Check standby health and lag */
int  repl_health_check(replicator_t* r);

A.4 HA Module

/* ha.h */

typedef enum { CTRL_PRIMARY = 0, CTRL_STANDBY = 1 } ctrl_role_t;

typedef struct {
    _Atomic uint64_t term;   /* C11 atomic election term (fencing token). ALL accesses
                              * must use atomic_load/store_explicit with acquire/release.
                              * The term is the sole authoritative election term —
                              * persist_t.term is an alias populated from this field
                              * during promotion (Part 5.6 step 2) for convenience in
                              * the WAL write path. Standby receiver updates this field
                              * via atomic_store_explicit(release) — see Part 6.4. */
    _Atomic int32_t role;    /* Atomic — ALL reads must use atomic_load_explicit(acquire).
                              * Uses _Atomic int32_t (not _Atomic ctrl_role_t) because
                              * C11 atomic semantics on enum types are implementation-defined;
                              * some compilers silently ignore _Atomic on enums, falling back
                              * to non-atomic loads. The role-check window mitigation (Part 5.7)
                              * requires a proven lock-free atomic — _Atomic int32_t guarantees
                              * this on all supported platforms. Cast to ctrl_role_t at access:
                              *   (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire)
                              * Writes use atomic_store_explicit with release:
                              *   atomic_store_explicit(&c->ha.role, (int32_t)CTRL_PRIMARY, memory_order_release) */
    int             my_priority;      /* My index in candidates[] */
    replicator_t    repl;             /* Replicate to standbys (primary) */
    _Atomic uint64_t applied_lsn;     /* Last applied WAL entry (standby).
                                       * Must be _Atomic — written by iomgr thread, read by watchdog.
                                       * See Part 5.3 for full rationale. */
    endpoint_t*     primary_ep;       /* Connection to current primary via iomgr SSL pipeline (standby).
                                       * See Part 5.2 comment for full lifecycle spec. */
    char            primary_host[XCE_MAXNAME_LEN];
    uint32_t        primary_port;
    struct timespec last_heartbeat;   /* Last heartbeat from primary (CLOCK_MONOTONIC, standby) */
    _Atomic int     promotion_failed;      /* Non-zero if last promotion attempt failed — retry on next watchdog tick.
                                            * Must be _Atomic: written by ha_promote_to_primary() on the watchdog
                                            * thread, read by ha_check_heartbeat() on the same thread but also
                                            * visible to vcpu threads checking promotion status. Relaxed ordering. */
    _Atomic int     promotion_retry_count; /* Number of consecutive failed promotion attempts.
                                            * Must be _Atomic because it is incremented by the
                                            * watchdog thread and read by vcpu threads during
                                            * promotion status checks. Use atomic_load/store
                                            * with relaxed ordering. */
    _Atomic int     in_catch_up;           /* C11 atomic — non-zero while standby is processing historical
                                            * catch-up entries. See Part 5.3 for full lifecycle and
                                            * access pattern documentation. */
    int             hooks_registered;      /* Idempotency guard for persist hooks — prevents
                                            * double-registration on promote→demote→re-promote cycles.
                                            * ha_promote_to_primary() must check this flag before
                                            * calling xce_add_hook(). Reset to 0 on demote. */
    atomic_int      failover_requested;    /* Async-signal-safe flag set by SIGUSR1 handler.
                                            * Polled by HA watchdog thread which executes the
                                            * actual failover (Part 5.8). Never read/written
                                            * outside of SIGUSR1 handler and watchdog thread. */
    uint32_t        last_received_crc;     /* CRC of the last received WAL entry on the replication
                                            * stream — used for chained CRC validation (P0-C fix).
                                            * Mirrors persist_t.last_crc but on the standby side. */
    ha_config_t     config;           /* HA configuration (timeouts, intervals) */
    pthread_mutex_t lock;             /* Protects primary_ep and last_heartbeat against concurrent
                                       * iomgr NULL-then-free and watchdog reads. Initialized in
                                       * ha_init(). See Part 5.3 for full access protocol. */
} ha_state_t;

/* Check if this controller should be primary */
static int am_i_primary(cluster_t* c);

/* Heartbeat watchdog — called from the dedicated HA watchdog pthread (NOT delaymgr). See P1-3 */
static void ha_check_heartbeat(cluster_t* c);

/* Promote self from standby to primary */
int ha_promote_to_primary(cluster_t* c);

/* Demote self from primary to standby (admin-triggered failover) */
int ha_demote_to_standby(cluster_t* c);

A.5 Cap'n Proto Schema Additions

# Persistence configuration — embedded in ClusterConfig
# NOTE: Path fields (wal_path, snap_path) are REMOVED from this schema.
# Paths are derived from --workdir and -n CLI flags at controller startup:
#   effective_dir = workdir + "/" + name
#   wal_path      = effective_dir + "/data/wal"
#   snap_path     = effective_dir + "/data/snap"
struct PersistConfig $C.mapname("persist_config_t") {
    walSyncMs    @0 :UInt32 $C.mapname("wal_sync_ms") = 10;
    snapIntervalS@1 :UInt32 $C.mapname("snap_interval_s") = 60;
    snapWalSizeMb@2 :UInt32 $C.mapname("snap_wal_size_mb") = 64;
    finishedRetainS @3 :UInt32 $C.mapname("finished_retain_s") = 3600;
    workerRetainS  @4 :UInt32 $C.mapname("worker_retain_s") = 86400;
    syncTimeoutMs@5 :UInt32 $C.mapname("sync_timeout_ms") = 50;
    snapKeepN    @6 :UInt32 $C.mapname("snap_keep_n") = 2;   # last-known-good snapshot retention
    transientJobTimeoutS @7 :UInt32 $C.mapname("transient_job_timeout_s") = 300;  # 0 = no auto-termination
}

# WAL entry — used in Cap'n Proto schema for replication messages (OP_replEntry).
# On-disk WAL format uses a raw C struct (wal_entry_hdr_t, see Part 2.2), NOT Cap'n Proto.
# The WALEntry schema duplicates the C header fields for network transport only.
# Fields: term, lsn, timestamp, type, crc, prevCrc mirror wal_entry_hdr_t.
# Payload is the Cap'n Proto-serialized entity data (Job, QueueConf, Worker, etc.).
struct WALEntry $C.mapname("wal_entry_t") {
    term        @0 :UInt64;
    lsn         @1 :UInt64;
    timestamp   @2 :UInt64;
    type        @3 :UInt32;
    payload     @4 :Data;
    crc         @5 :UInt32;
    prevCrc     @6 :UInt32;
}

# Snapshot header — first record in a checkpoint file
struct SnapshotHeader $C.mapname("snapshot_hdr_t") {
    lsn             @0 :UInt64;
    timestamp       @1 :UInt64;
    term            @2 :UInt64;
    version         @7 :UInt32 = 1;   # WAL format version — bumped on incompatible schema changes
    nJobs           @3 :UInt32;
    nQueues         @4 :UInt32;
    nWorkers        @5 :UInt32;
    checksum        @6 :UInt32;  # CRC32 over entire serialized Snapshot body (excluding this header)
    headerChecksum  @8 :UInt32;  # CRC32 over header fields (lsn, timestamp, term, version, nJobs, nQueues, nWorkers) — protects security-critical term and lsn from tampering
}

# Full snapshot — checkpoint file contents
struct Snapshot $C.mapname("snapshot_t") {
    header      @0 :SnapshotHeader;
    jobs        @1 :List(Job);
    queues      @2 :List(QueueConf);
    workers     @3 :List(Worker);
    resources   @4 :List(Resource);
    configs     @5 :List(ConfigVar);
}

# Redirect response — standby tells client where the primary is
struct RedirectInfo $C.mapname("xce_redirect_t") {
    host @0 :Text $C.mapname("host");
    port @1 :UInt32 $C.mapname("port");
}

# In RequestReply union — add redirect variant:
# redirect @8 :RedirectInfo $C.mapname("redirect");

# New error code in common/
# #define E_XCE_REDIRECT (-100)  /* Not an error — redirect instruction */
# #define EXCE_ERR_REDIRECT_LOOP (-101)  /* Too many redirect hops (>= 5) */

# Entity terminal states for deferred cleanup
# Worker status uses bitmask flags on UInt32, not enum (see Part 4.4):
#   #define XCE_WORKER_UNAVAIL   (0x00010000)  — Worker daemon died / disconnected
#   #define XCE_WORKER_HIDDEN    (0x00200000)  — Admin explicitly removed entity
# Job status — add transient bitmask (NOTE: 0x200 is JOB_STAT_WAIT, use 0x400):
#   #define JOB_STAT_TRANSIENT   (0x400)       — Job state uncertain after failover
# Queue status — add hidden bitmask:
#   #define QUEUE_STAT_HIDDEN    (0x0100)      — Admin deleted queue

A.6 libxce Discovery Module

/* xce_cluster.h — libxce cluster discovery API */

typedef struct {
    char     host[XCE_MAXNAME_LEN];
    uint32_t port;
} xce_candidate_t;

typedef struct {
    char     ca_cert_path[PATH_MAX];  /* CA cert path for SSL_VERIFY_PEER (empty = skip) */
} xce_tls_config_t;

typedef struct {
    xce_candidate_t  candidates[MAX_HA_CANDIDATES];
    int              ncandidates;           /* Number of candidates configured */
    uint32_t         connect_timeout_ms;    /* Per-candidate connect timeout (default 2000) */
    uint32_t         retry_base_ms;         /* Backoff base (default 100) */
    uint32_t         retry_max_ms;          /* Backoff cap (default 5000) */
    int              max_retries;           /* Max reconnection attempts (default 10) */
    xce_tls_config_t tls;                  /* TLS options — CA cert path for SSL_VERIFY_PEER.
                                            * Populated from --ca-cert CLI flag (P2.5-5/P2.5-6). */
} xce_cluster_config_t;

/* Connect to the cluster — returns SSL* connected to primary, or NULL on failure.
 * The returned SSL* is owned by the caller and must be freed with
 * xce_disconnect_cluster() when no longer needed. */
SSL* xce_connect_cluster(xce_cluster_config_t* conf);

/* Disconnect and free an SSL connection returned by xce_connect_cluster */
void xce_disconnect_cluster(SSL* ssl);

/* Connect + join as specific role */
SSL* xce_join_cluster(xce_cluster_config_t* conf, xce_peerinfo_t* myinfo);

/* Handle disconnection — auto-reconnect with backoff + redirect following.
 * On success, *ssl is updated to the new connection. On failure, *ssl is NULL. */
int  xce_reconnect(SSL** ssl, xce_cluster_config_t* conf);

/* REMOVED: xce_load_cluster_config() — no YAML parsing in libxce.
 * xce_cluster_config_t is populated by caller from --peers CLI flag. */

A.7 New Files Summary

File	Purpose	Approximate Size
src/controller/persist_wal.c	WAL writer, segment management, fsync batching	~400 LOC
src/controller/persist_snap.c	Checkpoint, recovery, hook functions	~400 LOC
src/controller/include/persist.h	Persistence API and types	~80 LOC
src/controller/repl.c	Fan-out replication via endpoint/iomgr SSL pipeline	~500 LOC
src/controller/include/repl.h	Replication API and types	~60 LOC
src/controller/ha.c	LSF-style priority election, monitor thread	~300 LOC
src/controller/include/ha.h	HA API and types	~40 LOC
src/lib/cluster.c	Candidate selection, redirect following, backoff+jitter, reconnection	~400 LOC
src/lib/include/xce/cluster.h	Discovery API and types	~50 LOC
Total new code		~2230 LOC

A.8 Modified Files Summary

File	Change
src/lib/zarray.c + include/zarray.h	Merge zarray_t into xzarray_t (P0-0a); add shadow, shadow_caps fields; add xzarray_freeze(), xzarray_thaw(); modify vget, vput, iter_next for COW mode
src/controller/job.c	Remove dead NJOBS/jobid/load_jobs body (P0-0); move xce_reply() after AFTER_EVENT in handle_jobsubmit()
src/lib/kvmgr.c	Register persist_nop_freeze/persist_nop_thaw for the kvmgr hash table's own varray interface (kvmgr_type in kvmgr.c). The kvmgr has TWO varray interfaces: (1) the hash table's varray (kvmgr itself) — which does NOT support COW and uses no-op freeze/thaw; and (2) the data zarray (xzarray_type in zarray.c) — which IS the COW target and uses the real xzarray_freeze/xzarray_thaw implementations. An implementer must NOT accidentally register no-ops for the data zarray, or checkpoint COW will silently skip freezing the data array. The xzarray_type registration in zarray.c already includes xzarray_freeze/xzarray_thaw slots — it is NOT modified by P0-0b. P0-0b only adds no-op slots to the kvmgr hash table's own varray compound literal in kvmgr.c.
src/controller/cluster.c	Wire persist_init/recover/start/stop/fini and HA into lifecycle; wire cluster_load_jobs() to persistence loader; deferred endpoint cleanup (after zarray merge); use delaymgr for checkpoint timer; heartbeat watchdog uses dedicated pthread (see P1-3)
src/controller/include/persist.h	Add persist_nop_freeze/persist_nop_thaw static inline no-ops for varray types that don't support COW
src/controller/netio.c + src/include/xce/netio.h	No changes needed for HA — existing SSL pipeline reused as-is for controller-to-controller connections (trusted LAN model, Part 5.9)
src/controller/join.c	Add redirect logic in handle_join() — standby sends E_XCE_REDIRECT + RedirectInfo for worker/scheduler/apiserver join requests; worker join WRITE lock is acceptable — async replication only (never blocks on repl_wait_ack inside write lock)
src/controller/worker.c	Extend existing mark_worker_unavail() with w->host.dead_since = time(NULL); add WORKER_HIDDEN state path; handle_worker_leave() already calls mark_worker_unavail() and sets w->ep = NULL — no structural change needed; wire restore_worker() into recovery path
src/include/xce/cluster_def.h	Add persist_t persist; (P0-5) and ha_state_t ha; (P1-5) to cluster_t; add WORKER_UNAVAILABLE, WORKER_HIDDEN entity states; worker status uses bitmask flags (e.g., XCE_WORKER_HIDDEN = 0x00200000), not state enum
src/schema/controller.capnp	Add PersistConfig, WALEntry, SnapshotHeader, Snapshot, RedirectInfo schemas; add term to Request; add standby to HostRole; add replEntry/replAck/redirect/heartbeat opcodes; add redirect variant to RequestReply; add deadSince to Worker; worker status remains UInt32 (bitmask, not enum)
src/controller/myconf.c	Parse persistence and HA configuration from YAML
src/worker/main.c	Replace hardcoded controller address with xce_cluster_config_t; add reconnection loop
src/scheduler/	Same pattern as worker — use xce_cluster_config_t
src/frontend/apiserver/backend/clib.go	Use xce_cluster_config_t for controller discovery in RegisterToBackend()
rtti/typeinfo.h	Add vfreeze_fn_t/vthaw_fn_t types, grow intf_varray_t to 8 slots, add vfreeze()/vthaw() global API wrappers (see Appendix A.9)
rtti-0.1.0 (nix)	Bump version from 0.0.1, update nix derivation hash and XCE import path

A.9 rtti Library Enhancements

The rtti library (rtti-0.0.1) is maintained alongside XCE and requires the following additions to support the COW snapshot mechanism. Since there are no released versions of rtti, backward compatibility is not a constraint — these changes are made in-place and version bumped to rtti-0.1.0.

A.9.1 New Function Pointer Types

Add to rtti/typeinfo.h before the intf_varray_t struct definition, after the existing vclear_fn_t typedef (around line 172):

/* COW snapshot support — freeze/thaw for application-level copy-on-write */
typedef bool (*retain_fn_t)(void* arg, void* item);
typedef int  (*vfreeze_fn_t)(meta_obj_t* obj);
typedef int  (*vthaw_fn_t)(meta_obj_t* obj, retain_fn_t should_retain, void* arg);

retain_fn_t is the callback used by vthaw to decide whether an entity survives checkpoint GC. It returns true to retain, false to garbage-collect. Passing NULL for should_retain means "keep all shadow entities, no GC" — used when aborting a checkpoint (merge shadow back without discarding anything). This type must live in rtti (not in persist.h or zarray.h) because vthaw_fn_t references it in the intf_varray_t interface — callers that invoke vthaw(var, my_retain, arg) need the type visible through the rtti header.

A.9.2 Updated intf_varray_t

Add two new slots after the existing six:

typedef struct {
    vstore_fn_t   vstore;
    vget_fn_t     vget;
    vget_vec_fn_t vget_vec;
    vput_fn_t     vput;
    vput_vec_fn_t vput_vec;
    vclear_fn_t   vclear;
    vfreeze_fn_t  vfreeze;  /* NEW — freeze (O(1) ptr swap) */
    vthaw_fn_t    vthaw;    /* NEW — thaw + merge + GC */
} intf_varray_t;

Layout: The struct grows from 24 bytes (6 × 8 byte pointers on LP64) to 32 bytes (8 × 8 byte pointers). The interface_table_t.varray field is already a pointer to intf_varray_t, so the struct size change is isolated to the varray interface — no other interface structs are affected. All existing compound-literal initializers (like the one in kvmgr.c:288 and zarray.c:666) must be updated to supply the two new NULL initializers. The C99 designated-initializer pattern { .vstore = ..., .vget = ..., } is recommended to make this robust against future reordering.

A.9.3 New Global API Functions

Add extern declarations after the existing varray API block (around line 259 of typeinfo.h):

extern int vfreeze(var);                                          /* O(1) freeze — pointer swap */
extern int vthaw(var, retain_fn_t should_retain, void* arg);      /* merge shadow + optional GC; should_retain=NULL keeps all */

These dispatch through RTTI_OBJ_TO_TYPE(obj)->interfaces.varray->vfreeze / ->vthaw, following the same pattern as the existing vstore, vget, etc. wrappers.

A.9.4 No-op Implementations in XCE

librtti provides only the abstract interface — the no-op implementations belong in XCE, not in librtti. Registration sites for varray types that don't support COW supply their own no-ops:

/* In persist.h — shared no-ops for XCE types that don't support COW */
static inline int persist_nop_freeze(meta_obj_t* obj) { return 0; }
static inline int persist_nop_thaw(meta_obj_t* obj, retain_fn_t f, void* a) { return 0; }

Types like kvmgr that don't support COW register these no-ops in their compound literal:

/* kvmgr.c */
.vfreeze = (vfreeze_fn_t)persist_nop_freeze,
.vthaw   = (vthaw_fn_t)persist_nop_thaw,

The library dispatch wrappers (vfreeze(), vthaw()) return -EINVAL when the method pointer is NULL — this is a bug catcher, not a feature gate. Every registration site must supply all 8 function pointers; a NULL slot is a programming error.

A.9.5 xzarray Registration

In zarray.c, the typeinfo_t xzarray_type registration must update the varray interface compound literal to include the new slots:

.varray = &(intf_varray_t) {
    .vstore   = (vstore_fn_t)xzarray_store,
    .vget     = (vget_fn_t)xzarray_get_vec,    /* xzarray_get_vec handles single-get too */
    .vget_vec = (vget_vec_fn_t)xzarray_get_vec,
    .vput     = (vput_fn_t)xzarray_put,
    .vput_vec = (vput_vec_fn_t)xzarray_put_vec,
    .vclear   = (vclear_fn_t)xzarray_clear,
    .vfreeze  = (vfreeze_fn_t)xzarray_freeze,  /* NEW */
    .vthaw    = (vthaw_fn_t)xzarray_thaw,       /* NEW */
},

A.9.6 Build Integration

• Version: Bump rtti-0.0.1 → rtti-0.1.0 in the nix derivation (or equivalent build system metadata)
• Nix store path: The updated library gets a new hash; update the import path in XCE's meson.build or nix flake accordingly
• Compiler flag: No new dependencies; <stdatomic.h> (for the atomic_int election_done flag in Part 5.4) is C11 and should already be in the toolchain
• Test coverage: Add tests for xzarray_freeze/xzarray_thaw round-trip, COW-mode vstore linear scan, and shadow merge correctness with overlapping indices

A.10 System-Level Tunable Constants — include/config.h

All system-level tunable constants are consolidated into include/config.h so that operators and developers can find and adjust limits in one place. Module-specific headers (persist.h, repl.h, cluster_def.h) #include "config.h" and use these definitions — they do not define their own local copies.

/* include/config.h — System-level tunable constants for XCE controller
 * All #define values below are the system-wide defaults.
 * Runtime-overridable values (persist_config_t, ha_config_t) are NOT here —
 * they live in their respective config structs and are set from cluster.bin (controller)
 * or from --peers/--ca-cert CLI flags (workers/schedulers). */

/* --- HA limits --- */
#define MAX_HA_CANDIDATES       4       /* Structural max candidates from --peers flag.
                                          * Reduced from 16: design validated for ≤3. */
#define DEFAULT_HB_INTERVAL_MS  3000    /* Primary heartbeat interval (ms) */
#define DEFAULT_HB_TIMEOUT_MS   10000   /* Standby heartbeat timeout (ms) */
#define DEFAULT_ELECTION_TIMEOUT_MS 2000  /* Election probe deadline (ms) */
#define DEFAULT_SYNC_TIMEOUT_MS 500     /* Sync replication ACK timeout (ms) */
#define DEFAULT_MAX_PROMOTION_RETRIES 5 /* Max failed promotion attempts */
#define DEFAULT_HA_CERT_WARN_DAYS 30   /* Warn if cert expires within N days */
#define MAX_TERM_JUMP           10000   /* Maximum term increment accepted in a single replication
                                          * or OP_replEntry message. Prevents term overflow
                                          * from a corrupted or malicious packet forcing the
                                          * controller into permanent step-down mode.
                                          * Defined in config.h (not inline in Part 6.4) so
                                          * all callers share a single source of truth. */

/* --- Persistence limits --- */
#define WAL_FORMAT_VERSION      1       /* Bumped on incompatible schema changes.
                                          * When bumping, write a migration function
                                          * (e.g., migrate_snapshot_v1_to_v2) before
                                          * changing this value. Older snapshots
                                          * load only if all new fields are optional. */
#define WAL_SEG_SIZE            (64 * 1024 * 1024)  /* 64 MB pre-allocated WAL segment */
#define WAL_HDR_CRC_SIZE        32      /* sizeof(wal_entry_hdr_t) - 2*sizeof(uint32_t) */
#define MAX_WAL_PAYLOAD_BYTES   (1u << 20)  /* 1 MB per WAL entry */
#define WAL_ENTRY_TYPE_MAX     100     /* Highest valid entry type — bump when new types added */
#define MAX_SNAPSHOT_BYTES      (256 * 1024 * 1024)  /* 256 MB — reject larger snapshots */
#define MAX_AUTO_TERM_PER_CHECKPOINT 10240  /* Max JOB_STAT_TRANSIENT jobs auto-terminated in one
                                              * checkpoint cycle. Bounds the heap-allocated
                                              * auto_term[] array in persist_checkpoint().
                                              * Jobs beyond this cap survive to the next
                                              * checkpoint (bounded delay, not data loss). */

/* --- Persistence defaults (runtime-overridable via persist_config_t) --- */
#define DEFAULT_WAL_SYNC_MS     10      /* fsync interval (ms) */
#define DEFAULT_SNAP_INTERVAL_S 60      /* checkpoint interval (s) */
#define DEFAULT_SNAP_WAL_SIZE_MB 64     /* checkpoint when WAL exceeds this (MB) */
#define DEFAULT_FINISHED_RETAIN_S 3600  /* retain finished jobs (s) */
#define DEFAULT_WORKER_RETAIN_S  86400  /* retain unavailable workers (s) */

/* --- Derived --- */
#define MAX_STANDBYS  3  /* Standalone constant — NOT derived from MAX_HA_CANDIDATES.
                          * Deployment ceiling is ≤3 controllers (1 primary + 2 standbys).
                          * Defined independently so bumping MAX_HA_CANDIDATES for structural
                          * reasons does not silently increase MAX_STANDBYS beyond the
                          * validated deployment ceiling. */

Design rule: If a constant affects more than one module or is an operator-visible limit, it belongs in config.h. If it is a purely internal implementation detail of a single module (e.g., an internal struct size or a local retry count), it may stay in the module header. The test: if an operator would ask "what's the maximum?" or a developer would search for the constant when debugging, it goes in config.h.