- persist_write_checkpoint_marker(): writes WAL_CHECKPOINT header entry
with CRC, increments current_lsn; never updates last_replicated_crc
(checkpoint markers are local-only, never replicated)
- write_snapshot(): atomic tmp→rename write of P0 sentinel snapshot
("XCESNAP\0" + checkpoint_lsn); full Cap'n Proto body in Task 11
- trim_wal_segments(): removes seg-*.wal files whose start LSN is
strictly less than checkpoint_lsn
- rotate_snapshots(): collects .capnp files, sorts by name (LSN order),
deletes oldest beyond snap_keep_n
- persist_checkpoint(): EBUSY guard via atomic_exchange, sequences
marker→lsn record→snapshot→trim→rotate, clears flag on exit
- 5 TDD tests in test_persist_checkpoint.c covering: snapshot file
creation, checkpoint_lsn advance, WAL lsn advance, EBUSY guard,
and rotation keeping at most snap_keep_n files
- All 18 controller unit tests pass (0 failures)
Add kvtab_foreach walk primitive (P0-14) to kv.c/kv.h, then implement
kvmgr_capture_keys in kvmgr.c that deep-copies all key→index mappings
into a malloc'd slot_key_t[] array safe to read after the write lock is
released during checkpoint serialization (Part 3.8).
Key design points:
- Both kvtab_foreach and kvmgr_capture_keys take var (RTTI data pointer)
and use RTTI_DATA_TO_OBJ internally — the RTTI framework's makeInstance
returns the data region pointer, not the meta_obj_t header pointer
- String keys are copied via strncpy into slot_key_t.key[XCE_MAXNAME_LEN]
- Integer keys are formatted as decimal strings via snprintf
- Two-pass approach: count live entries first, allocate, then fill
- Tests cover string kvmgr (3 entries), empty kvmgr, and deep-copy safety
Extends src/schema/controller.capnp with persistence structures needed
for WAL replication and snapshots:
- PersistConfig (8 fields, @0-@7): capnp-serialisable copy of the
persistence configuration; mapped to capnp_persist_cfg_t to avoid
collision with the plain-C persist_config_t in persist.h
- SnapshotHeader (9 fields, @0-@8): snapshot manifest with LSN, term,
timestamp, record counts and checksums; mapped to capnp_snap_hdr_t
- ReplEnvelope (6 fields, @0-@5): replication wire envelope (NOT the
on-disk format); named to avoid collision with wal_entry_hdr_t and
the WALEntry_ internal symbol c-capnproto generates; mapped to
capnp_repl_envelope_t
Also extends existing structs:
- Job.transientSince @11 (UInt64=0, mapname transient_since): set when
job enters JOB_STAT_TRANSIENT; also adds uint64_t transient_since to
xce_job_t in types.h
- Worker.deadSince @6 (UInt64=0, mapname dead_since): mirrors existing
time_t dead_since field in xce_worker_t
OPCode enum extended with 7 replication opcodes @22-@28:
replEntry, replAck, redirect, heartbeat, steppingDown,
replSnapshot, replCatchUpDone
New C typedefs (capnp_persist_cfg_t, capnp_snap_hdr_t,
capnp_repl_envelope_t) added to typemap.h for the encode/decode shims.
TDD: test_persist_schema.c added (6 tests, all GREEN). Controller test
suite remains 1/1 PASS.
- In persist_wal_async(), check rotate_segment() return value and propagate -EIO on failure (MEDIUM)
- Change last_crc and last_replicated_crc fields from uint64_t to uint32_t to match CRC32 width (LOW)
Both fixes address code review issues in commit b0a5fbb.
- Expose persist_t struct concretely in persist.h (opaque pattern deferred)
- persist_set_workdir() derives wal/snap paths from workdir+name
- persist_init() creates WAL/snap directories, rejects NFS mounts
- persist_wal_async() writes WAL entries with chained CRC32 via zlib
- Dedicated fsync pthread fires every wal_sync_ms milliseconds
- persist_flush() forces fdatasync for tests and clean shutdown
- WAL_CHECKPOINT type rejected with -EINVAL (never written via async path)
- Stubs: persist_recover, persist_wal_sync_append, persist_checkpoint
- Add fsync_thread_running flag to guard persist_stop when thread not started
- Add zlib_dep to controller and unit test meson targets
- 3 new TDD tests: init_creates_directories, wal_async_writes_entry,
checkpoint_type_rejected — all GREEN
- Add in_cow flag + shadow/shadow_caps fields to xzarray_private_t
- Convert xzarray_item_t.inuse from bitfield to _Atomic _Bool; update
all read/write sites to use atomic_load/store_explicit
- Implement xzarray_freeze(): O(1) ptr swap, pre-grows active array
1.5x+16 to avoid realloc during COW, sets pos past shadow range
- Implement xzarray_thaw(): merges shadow into active (shadow wins only
for slots not updated during COW), supports optional retain_fn GC
- xzarray_get_vec(): falls back to shadow during COW so reads remain
valid between freeze and thaw
- Add COW guard assertion in xzarray_clear()
- Wire vfreeze/vthaw into xzarray_type interface
- Tests: 5 new COW tests all green; 4 pre-existing failures unchanged
- Rename `precision` field to `precision_ms` in delaymgr_t struct (delay.h)
- Replace `sleep(mgr->precision)` with `nanosleep()` for 10ms granularity (delay.c)
- Add `#include <time.h>` to delay.c (already in delay.h)
- Update delaymgr_init() to store precision_ms parameter
- Update all call sites: cluster.c (2→200ms), controller/testsuite/test.c (2→2000ms),
lib/testsuite/test_delay.c (1→1000ms)
- Add TDD test test_delaymgr_precision_ms_field verifying field exists and stores correctly
- delay_task() delay parameter remains in seconds; only sleep granularity changes
rtti header already present at rtti-0.0.2 with vfreeze_fn_t, vthaw_fn_t,
retain_fn_t, and the two new slots in intf_varray_t. No header modification
needed.
Convert intf_varray_t compound-literal initializers in zarray.c and kvmgr.c
to designated style with explicit .vfreeze = NULL and .vthaw = NULL slots.
kv.c has no intf_varray_t initializer.
Add test_zarray_cow.c with test_freeze_thaw_slots_exist which verifies the
struct layout has the slots accessible and that they are NULL until P0-2
wires the real implementations. Wire test into meson.build and test.c.
Full clean rebuild confirmed: 27/28 targets linked, no new test failures.
Only the 4 pre-existing failures remain.
Two correctness fixes found during final Phase 6 code review:
F1 (CRITICAL): GC permanently disabled — staging_merge_complete bootstrapping deadlock.
sched_begin_session reset staging_merge_complete to 0 but never set it back to 1,
so sched_end_session's guard (!staging_merge_complete) always fired, preventing GC
from ever running. select-node memory leaked indefinitely across scheduling sessions.
Fix: set staging_merge_complete=1 at the end of sched_begin_session after all
staging work (sched_account_jobs etc.) completes.
F2 (HIGH): recompact_node_ids() omitted from GC protocol in sched_end_session.
The protocol in resreq.h requires gc_sweep → recompact_node_ids. Without
recompaction, sel_node_next_id grows monotonically across sessions and
select_eval_ctx_t arrays (sized by max node_id) grow without bound.
Fix: call recompact_node_ids() after gc_sweep_select_nodes.
Also adds test_staging_merge_complete_gc_guard to document the bootstrap
invariant: sched_begin_session must leave staging_merge_complete=1 so the
GC guard in sched_end_session does not fire.
- simple_num/simple_str: guard h->numvals/strvals NULL and idx out-of-range before array access
- eval_select_simple SEL_RESNAME: guard host->numvals NULL and idx bounds before deref
- eval_select_inner SEL_RESNAME (memoizing path): same NULL/bounds guards for numvals and strvals
- build_host_eval_ctx: return valid empty context (num_res=0, NULL arrays) instead of NULL when num_res<=0
- sched_meet_host_selection: remove empty_ctx stack hack; always call build_host_eval_ctx which now handles the zero-resource case cleanly
Replace the always-return-1 stub with a real implementation that:
- Returns 1 immediately when s == NULL or s->root == NULL (no constraint)
- Builds a host_eval_ctx_t from w->resvals and calls eval_select_simple
- Uses a zero-resource stack context when resvals.caps == 0 so constant
sub-expressions (SEL_TRUE, SEL_FALSE, SEL_ANY) still evaluate correctly
- On OOM (build_host_eval_ctx returns NULL), conservatively passes the host
- Maps eval_select_simple result: >0 → 1 (match), <=0 → 0 (no-match)
Add #define UNIT_TEST 1 to test.c, remove redundant #ifdef guard, and
add three new TDD tests covering NULL select, SEL_TRUE, and SEL_FALSE.
All 6 tests pass.
- Add gc_clear_marks(): iterates intern table and zeroes all gc_mark bits
- Rename static gc_mark_tree → public gc_mark_select_tree (no attribute((unused)))
- Remove Phase 6 guard (live_jobs != NULL early return) from gc_sweep_select_nodes
- Remove the mark-clear loop from gc_sweep_select_nodes; add (void) casts for now-unused params
- Declare gc_clear_marks and gc_mark_select_tree in resreq.h with protocol comment
- Add TDD test test_gc_clear_marks_exported verifying the exported primitives
- Prefix all existing gc_sweep_select_nodes(NULL, 0) call sites in test_select.c with gc_clear_marks()
P0 fixes (9):
- R-22: COW vstore find-slot-and-write not atomic
- R-23: vput grow path without mutex during COW
- R-24: Post-growth slot double-claimed by concurrent threads
- R-25: Missing key_snapshot field for checkpoint serialization
- R-26: malloc(0) for WAL_CHECKPOINT entries
- R-27: Fallback to older snapshot with trimmed WAL
- R-28: hdr.term > ha->term + 1 too restrictive
- R-29: _Atomic on enum type (implementation-defined)
- R-30: handle_jobsubmit() ignores hook return value
- R-31: malloc NULL check missing in standby receiver
P1 fixes (5):
- R-32: No heartbeat during persist_recover() on new primary
- R-33: Catch-up from dead primary removed from promotion
- R-34: Added total_wal_bytes field to persist_t
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
