Add test-only function ha_trigger_manual_failover_for_test() to exercise manual
failover state machine (demote → promote cycle) without requiring live cluster_t.
Test coverage:
- test_manual_failover_transitions_role_and_increments_term: Validates role
transitions (PRIMARY → STANDBY → PRIMARY) and term monotonicity
- test_manual_failover_resets_hooks_registered: Confirms demotion resets the
hooks_registered idempotency guard
- test_manual_failover_monotonic_term_across_multiple_cycles: Validates term
remains monotonically increasing across 5 consecutive failover cycles
Implementation:
- Exposed only under #ifdef XCE_TESTING in ha.h
- Function body (6 lines): calls ha_do_demote() then ha_do_promote(); returns
the new term value via atomic_load_explicit()
- Enables design validation (P3-6) without test-cluster infrastructure
All tests pass. Build clean.
Add test_p2_integration.c with 7 integration tests covering the assembled P2
replication system: WAL_CHECKPOINT assert invariant, trim_lsn with active standby,
checkpoint must not update last_replicated_crc, sync degraded mode, full catch-up
lifecycle, needs_snapshot boundary conditions, and API completeness gate.
All 32 controller unit tests pass (0 failures).
Implement the catch-up state machine as testable pure-logic functions:
- repl_start_catchup(): marks standby in_catch_up=1, stores join_applied_lsn
- repl_complete_catchup(): clears in_catch_up=0, broadcasts ack_cond so any
blocked repl_wait_ack() re-checks the ACK condition immediately
- repl_needs_snapshot(): pure predicate — returns 1 when standby LSN predates
oldest surviving WAL segment (snapshot transfer required), 0 otherwise
Also fixes pre-existing linker error: repl.c was missing from controller_src
(xce_controller production binary), causing undefined references to repl_*
symbols introduced in P2-4.
7 new TDD tests in test_repl_catchup.c — all pass. Existing tests green.
- Move repl.h from controller/ to include/ (shared header) to break
persist.h ↔ repl.h circular include
- Extract wal_entry_hdr_t + wal_entry_type into include/wal_types.h so
repl.h can include only what it needs without pulling in all of persist.h
- Add replicator_t repl field to persist_t (embedded by value, P2-4)
- Wire repl_init/repl_fini into persist_init/persist_fini lifecycle
- Wire repl_broadcast into persist_wal_async after each WAL write;
last_replicated_crc updated here (not in persist_write_checkpoint_marker)
- Wire repl_wait_ack into persist_wal_sync_append; returns 0 on local
WAL success; sync_rc_out carries -ENODEV/-ETIMEDOUT for degraded mode
- Add 4 TDD tests in test_persist_repl.c: structural repl field check,
async no-standbys succeeds, sync degraded mode rc, checkpoint CRC invariant
Upgrade apply_wal_entry() from P0 static no-op stub to the real public
standby replay path:
- Reject oversized payloads (> MAX_WAL_PAYLOAD_BYTES) with -EMSGSIZE
- Skip WAL_CHECKPOINT (100) entries silently — local-only markers, never
replicated to standbys; current_lsn is NOT advanced on skip
- Skip unknown types (> WAL_ENTRY_TYPE_MAX) with WARN log and rc=0 for
forward compatibility with WAL written by newer controller versions
- Advance current_lsn to hdr->lsn+1 and update last_crc under p->lock
for all accepted known entries
- Gate restore_job()/worker_from_capnp() calls on p->cluster != NULL so
unit tests work without a full cluster wired in
- Add NULL guard returning -EINVAL for missing p or hdr
Add public declaration in persist.h with forward-declare stubs for
restore_job/worker_from_capnp in persist.c (avoids heavy cluster.h chain).
Test suite: 5 new Unity tests in test_repl_apply.c covering all branches.
Weak stubs for restore_job/worker_from_capnp in the test file satisfy the
linker without linking job.c/worker.c's heavy dependency graph.
All 5 new tests pass; full controller suite green.
Add replicator_t structure with per-standby connection slots (standby_conn_t),
lifecycle (repl_init/repl_fini), slot management (repl_add_standby,
repl_remove_standby, repl_expire_disconnect_slots), ACK tracking
(repl_on_ack), WAL trim guard (repl_trim_lsn), and stubs for
repl_broadcast/repl_wait_ack (wired in P2-3/P2-4).
Uses mylock_t (pthread_mutex_t alias) with mylock_lock/mylock_unlock.
snap_interval_s==0 in repl_expire_disconnect_slots forces immediate expiry
to support deterministic unit testing.
5 new Unity tests all pass; 115 pre-existing controller tests unchanged.
Add redirect helpers to ha.c/ha.h so a standby controller never accepts
join requests as if it were the primary:
- ha_should_redirect(): returns 1 when role is STANDBY and primary_host/port
are known — caller should send E_XCE_REDIRECT.
- ha_no_primary_known(): returns 1 when role is STANDBY but primary address
is not yet known — caller should send E_XCE_NOMASTER (retry later).
- Add E_XCE_NOMASTER and E_XCE_REDIRECT to errnox.h.
- Wire both guards into handle_join_() in join.c (before the role switch)
so all join types (worker, scheduler, apiserver, client) are protected.
- Full redirect response sends status=-E_XCE_REDIRECT with nodata body;
the primary host:port are logged so operators can trace misrouted clients.
Full Cap'n Proto RedirectInfo response body is deferred to a follow-up
(schema union would require regenerating capnp bindings).
5 new TDD tests — 110 total, 0 failures.
- Full ha.h: ctrl_role_t enum (CTRL_PRIMARY/CTRL_STANDBY), ha_config_t,
ha_state_t with _Atomic fields (term, role, applied_lsn, promotion
counters, watchdog_stop), pthread_mutex_t lock, and public API.
- endpoint_t* omitted from ha_state_t — including netio.h would pull
OpenSSL headers and risk circular deps; watchdog (P1-3) includes
netio.h directly where needed.
- ha_init validates: heartbeat_interval_ms >= 2000, heartbeat_timeout_ms
>= heartbeat_interval_ms, election_timeout_ms >= 500; returns -EINVAL
on bad config; zeros struct and sets initial role to CTRL_STANDBY.
- ha_fini destroys mutex; watchdog thread join deferred to P1-3.
- am_i_primary / ha_promote_to_primary / ha_demote_to_standby are
no-op stubs; cluster_t->ha field not yet wired (P1-5), so these avoid
incomplete-type dereferences until then.
- TDD: 5 new tests in test_ha_role.c confirmed RED before implementation,
all 5 GREEN after; all prior controller tests remain green.
Add terminal-state constants and GC retention predicates for the
checkpoint xzarray_thaw() GC pass:
- types.h: add JOB_STAT_TRANSIENT (0x400), XCE_WORKER_HIDDEN (0x00200000),
QUEUE_STAT_HIDDEN (0x0100)
- persist.c: implement job_is_terminal(), job_should_retain(),
worker_should_retain() — placed here so the unit test binary
(which links only persist.c) can access them without RTTI deps
- persist.h: declare the three GC predicates; include xce/types.h
- worker.c: set dead_since = time(NULL) in mark_worker_unavail()
when dead_since is not already set
- cluster.c: add TODO P0-8b marker on zarray_del endpoint cleanup
- meson.build: register test_gc_retention.c; raise test timeout 60→120s
- testsuite/test_gc_retention.c: 19 tests covering job_is_terminal,
job_should_retain (TRANSIENT TTL, terminal TTL, non-terminal always
retained), and worker_should_retain (HIDDEN/UNAVAIL/healthy/zero
dead_since); all pass (63 total, 0 failures)
Implements persist_job_hook_fun, persist_queue_hook_fun, and
persist_worker_hook_fun — AFTER_EVENT hooks that write WAL entries
whenever state mutations occur in the controller.
- detail_to_wal_type() maps all 15 HOOK_DETAIL_* values to WAL types;
unknown details return 0 and are silently skipped
- HOOK_DETAIL_JOB_NEW uses the sync replication path (WAL_JOB_SUBMIT);
all others use persist_wal_async
- P0 payload: 4-byte detail value; full Cap'n Proto serialization in Task 15
- Hook signature matches hook_fn_t exactly:
int fn(int type, int situation, int detail, void* cls, va_list ap)
- Added <stdarg.h> to persist.h for va_list; added hook.h and
xce/cluster_def.h includes to persist.c
- 18 new unit tests in test_persist_hooks.c covering all 15 detail→WAL
mappings, skip-on-unknown, and sequential LSN advancement
- cluster_def.h: add struct tag `cluster_s` to cluster_t to resolve
conflicting typedef with persist.h forward declaration
- cluster_def.h: add HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME enum
values after HOOK_DETAIL_JOB_RUSAGE_UPDATE
- job.c on_job_stop(): replace mark_job_update() call with direct
xce_run_hooks(BEFORE/AFTER, HOOK_DETAIL_JOB_STOP) so WAL writer
records stop events with the correct detail value
- job.c on_job_resume(): same pattern with HOOK_DETAIL_JOB_RESUME
- job.c do_clr_preason(): fix typo — second hook call was HOOK_BEFORE_EVENT
instead of HOOK_AFTER_EVENT (double-before, zero-after bug)
- join.c handle_worker_join(): move xce_reply() to after HOOK_AFTER_EVENT
so client ACK is not sent before WAL write completes (durability fix)
- testsuite/test_hooks.c: new TDD tests for JOB_STOP/RESUME enum values
- testsuite/controller_test.c + meson.build: register test_hooks suite
Reply ordering audit results:
job.c handle_jobsubmit: reply before AFTER_EVENT — intentional (jobid
returned to client before scheduler notification; WAL hook belongs in
BEFORE_EVENT for this path, handled in Task 13)
job.c do_set_preason: reply before set_preason() loop — intentional
fire-and-forget (no AFTER_EVENT hook at top level)
job.c do_clr_preason: reply before loop — same fire-and-forget pattern
job.c do_jobcontrol_op: no AFTER_EVENT hook, reply only — OK
join.c handle_worker_join: FIXED (reply moved after AFTER_EVENT)
worker.c on_worker_update/dead: reply after all hooks — OK
cluster.c: AFTER_EVENT hooks are internal (no reply) — OK
- persist_write_checkpoint_marker(): writes WAL_CHECKPOINT header entry
with CRC, increments current_lsn; never updates last_replicated_crc
(checkpoint markers are local-only, never replicated)
- write_snapshot(): atomic tmp→rename write of P0 sentinel snapshot
("XCESNAP\0" + checkpoint_lsn); full Cap'n Proto body in Task 11
- trim_wal_segments(): removes seg-*.wal files whose start LSN is
strictly less than checkpoint_lsn
- rotate_snapshots(): collects .capnp files, sorts by name (LSN order),
deletes oldest beyond snap_keep_n
- persist_checkpoint(): EBUSY guard via atomic_exchange, sequences
marker→lsn record→snapshot→trim→rotate, clears flag on exit
- 5 TDD tests in test_persist_checkpoint.c covering: snapshot file
creation, checkpoint_lsn advance, WAL lsn advance, EBUSY guard,
and rotation keeping at most snap_keep_n files
- All 18 controller unit tests pass (0 failures)
Extends src/schema/controller.capnp with persistence structures needed
for WAL replication and snapshots:
- PersistConfig (8 fields, @0-@7): capnp-serialisable copy of the
persistence configuration; mapped to capnp_persist_cfg_t to avoid
collision with the plain-C persist_config_t in persist.h
- SnapshotHeader (9 fields, @0-@8): snapshot manifest with LSN, term,
timestamp, record counts and checksums; mapped to capnp_snap_hdr_t
- ReplEnvelope (6 fields, @0-@5): replication wire envelope (NOT the
on-disk format); named to avoid collision with wal_entry_hdr_t and
the WALEntry_ internal symbol c-capnproto generates; mapped to
capnp_repl_envelope_t
Also extends existing structs:
- Job.transientSince @11 (UInt64=0, mapname transient_since): set when
job enters JOB_STAT_TRANSIENT; also adds uint64_t transient_since to
xce_job_t in types.h
- Worker.deadSince @6 (UInt64=0, mapname dead_since): mirrors existing
time_t dead_since field in xce_worker_t
OPCode enum extended with 7 replication opcodes @22-@28:
replEntry, replAck, redirect, heartbeat, steppingDown,
replSnapshot, replCatchUpDone
New C typedefs (capnp_persist_cfg_t, capnp_snap_hdr_t,
capnp_repl_envelope_t) added to typemap.h for the encode/decode shims.
TDD: test_persist_schema.c added (6 tests, all GREEN). Controller test
suite remains 1/1 PASS.
- Expose persist_t struct concretely in persist.h (opaque pattern deferred)
- persist_set_workdir() derives wal/snap paths from workdir+name
- persist_init() creates WAL/snap directories, rejects NFS mounts
- persist_wal_async() writes WAL entries with chained CRC32 via zlib
- Dedicated fsync pthread fires every wal_sync_ms milliseconds
- persist_flush() forces fdatasync for tests and clean shutdown
- WAL_CHECKPOINT type rejected with -EINVAL (never written via async path)
- Stubs: persist_recover, persist_wal_sync_append, persist_checkpoint
- Add fsync_thread_running flag to guard persist_stop when thread not started
- Add zlib_dep to controller and unit test meson targets
- 3 new TDD tests: init_creates_directories, wal_async_writes_entry,
checkpoint_type_rejected — all GREEN
