Add void (*on_connected)(endpoint_t *ep) field to endpoint_t with
endpoint_set_on_connected() helper. io_handler fires it exactly once
immediately after ssl_handshake_done=1 then nulls the pointer.
Enables ha_start_election_async to send OP_requestVote as soon as
the TLS handshake with a peer completes — no blocking SSL_connect.
6 new unit tests pass; controller suite 1/1 OK.
Add millisecond-granularity delay_task_ms() to delaymgr using
clock_gettime(CLOCK_MONOTONIC) for deadline computation. Extends
delay_task_t with nextrun_ts field; delay_worker distinguishes
ms-tasks from second-tasks via non-zero nextrun_ts. Backward
compatible with existing delay_task callers.
4 delay tests pass (2 new: tolerance and invalid-args).
Provides the timeout primitive for election and committing watchdogs.
xce_replconfig_reply_raw() - primary sends reply with real bytes via endpoint
xce_replconfig_req_ssl() - standby sends request over raw SSL socket
xce_replconfig_reply_ssl_decode() - standby decodes reply from raw SSL
xce_upgrade_config_req_extract() - documents architectural limitation; returns -ENOTSUP
xce_upgrade_config_req_extract_raw() - extract bytes from raw packet for OP_upgradeConfig
Same bypass pattern as existing xce_replconfigpush_req_raw(): build the
capnp message directly via capn_new_list8 + capn_setv8, sidestepping the
encode_Data_ptr path that routes raw bytes through xce_data_t (a
discriminated cluster-data union that cannot carry raw byte arrays).
Unblocks handle_config_request() and do_one_config_fetch() wiring.
Task A research finding: xce_upgrade_config_req_extract(xce_request_t*)
is unimplementable because xce_decode() calls capn_free() before any
handler runs, and decode_Data_ptr converts the wire-format List(UInt8)
into a zero-filled xce_data_t. Task D must capture the raw io_packet_t
bytes before xce_decode tears them down, then call the _raw variant.
Crash: encode_NameValue_list called with count=21, s=NULL on worker
rejoin, causing SIGSEGV in persist_worker_hook_fun.
Root cause: free_name_val_list(xce_name_val_t***, int) accepted count by
value. It zeroed *vals (resvals_to_sched = NULL) but could not zero
nresvals_to_sched. Any caller that freed the scheduler resource snapshot
(opget.c workerlist, reply.c clusterconf, dataex.c, sched.c dispatch)
left nresvals_to_sched stale. When the worker later rejoined and the
WAL HOOK fired, the WAL encoder read count=21 with a NULL pointer.
Fix: change signature to free_name_val_list(xce_name_val_t***, int*)
so both fields are zeroed atomically. Update all 7 call sites.
When a worker re-joins and already exists in workermgr, makeWorker_()
now overrides the existing slot instead of returning EEXIST. Admin-set
flags (LOCKEDU, CLOSED, HIDDEN, CLOSED_BY_ADMIN) survive the override.
Add XCE_WORKER_CLOSED_BY_ADMIN flag to distinguish admin-closed hosts
from workload-closed ones. worker_ctrl sets/clears it alongside CLOSED.
Refactor worker_init/worker_destroy into copy_worker/kill_worker helpers
shared by both fresh creation and override paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sel_key_cmp_fn compared keys with memcmp on the whole sel_node_key_t
struct, including the 4-byte padding between type and op1. The stack-
allocated search key in gc_sweep_select_nodes can have different padding
bytes than the heap-allocated copy stored by key_add, causing key_del
to miss the entry and leave a dangling pointer in the CSE intern table.
Replace memcmp with field-by-field comparison on type, op1, op2.
Also fix GC sweep tests that asserted pointer inequality after re-intern:
calloc may lawfully reuse the freed address, so pointer inequality does
not reliably indicate a successful sweep. Track the swept node's
node_id before freeing and assert the re-interned node gets a fresh id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
Four bugs fixed:
1. persist.c: array job WAL loop aborted after index [0] when replication
returned -ENODEV (no standbys). Loop guard now uses the return value of
persist_wal_sync_append() which is 0 on local-disk success regardless of
replication status.
2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
up the existing slot by name, updates its endpoint, and clears UNAVAIL so
the normal hook/reply/on_worker_join flow proceeds.
3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
connection dropped because no error handler was registered. Added
handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
checking_connection() to detect the failure and rejoin.
4. cluster.c: persist_recover() was called before cluster_build_resources()
populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
so resmgr is fully populated before WAL replay.
Three identical static copies (controller, worker, scheduler) replaced
with a single xce_parse_peers() in src/lib/cluster.c.
The libxce version is the canonical implementation: stores the logical
name from 'name=host:port' syntax so daemons on the same dev host can
identify themselves by name rather than IP address.
Worker (P2.5-5):
- Added parse_peers() static helper to src/worker/src/main.c
- Switched getopt to getopt_long to support --peers and --ca-cert long options
- When --peers is given: parses into xce_cluster_config_t, calls
xce_cluster_config_validate(), logs INFO with candidate count/identity/ca_cert
- If validation fails: prints clear error and exits(1)
- WARN logged when candidates_count > 3
- Backward compatible: no --peers falls back to existing single-controller path
Scheduler (P2.5-6):
- Added parse_peers() static helper to src/scheduler/src/main.c
- Extended argp struct with --peers (-P) and --ca-cert (-C) options
- Same validation/logging pattern as worker
- Backward compatible: no --peers falls back to existing single-controller path
Tests (new):
- src/lib/testsuite/test_cluster_parse.c: 8 unit tests for parse_peers logic
(single entry, multiple entries, no-name format, empty/null/missing-port errors,
name-stripped correctly, MAX_HA_CANDIDATES cap respected)
- Registered in test.c and meson.build
- All 8 tests GREEN; 3 pre-existing failures in unrelated GC/CSE tests unchanged
- hook_item_t.fn changed to _Atomic hook_fn_t (C11 stdatomic)
- xce_disable_hook(): atomically stores NULL into the slot fn field;
returns 0 on success, -ENOENT if fn not found, -EINVAL on bad args
- xce_add_hook(): scans for NULL (disabled) slots before allocating;
uses atomic_store_explicit(release) when writing fn pointer
- xce_run_hooks(): atomic_load_explicit(acquire) + NULL guard so
disabled slots are silently skipped without disrupting the loop
- Add src/lib/testsuite/test_hooks.c with 3 TDD tests:
test_disable_hook_stops_invocation
test_disable_hook_returns_enoent_if_not_found
test_add_hook_reuses_null_slot
- Register test_hooks in meson.build and test.c runner
Required by ha_demote_to_standby() to atomically disable persistence
hooks when the controller steps down from primary, preventing unbounded
hook list growth across promote/demote cycles (resolution A6-015,
A7-004, A7-016).
Add kvtab_foreach walk primitive (P0-14) to kv.c/kv.h, then implement
kvmgr_capture_keys in kvmgr.c that deep-copies all key→index mappings
into a malloc'd slot_key_t[] array safe to read after the write lock is
released during checkpoint serialization (Part 3.8).
Key design points:
- Both kvtab_foreach and kvmgr_capture_keys take var (RTTI data pointer)
and use RTTI_DATA_TO_OBJ internally — the RTTI framework's makeInstance
returns the data region pointer, not the meta_obj_t header pointer
- String keys are copied via strncpy into slot_key_t.key[XCE_MAXNAME_LEN]
- Integer keys are formatted as decimal strings via snprintf
- Two-pass approach: count live entries first, allocate, then fill
- Tests cover string kvmgr (3 entries), empty kvmgr, and deep-copy safety
- Add in_cow flag + shadow/shadow_caps fields to xzarray_private_t
- Convert xzarray_item_t.inuse from bitfield to _Atomic _Bool; update
all read/write sites to use atomic_load/store_explicit
- Implement xzarray_freeze(): O(1) ptr swap, pre-grows active array
1.5x+16 to avoid realloc during COW, sets pos past shadow range
- Implement xzarray_thaw(): merges shadow into active (shadow wins only
for slots not updated during COW), supports optional retain_fn GC
- xzarray_get_vec(): falls back to shadow during COW so reads remain
valid between freeze and thaw
- Add COW guard assertion in xzarray_clear()
- Wire vfreeze/vthaw into xzarray_type interface
- Tests: 5 new COW tests all green; 4 pre-existing failures unchanged
- Rename `precision` field to `precision_ms` in delaymgr_t struct (delay.h)
- Replace `sleep(mgr->precision)` with `nanosleep()` for 10ms granularity (delay.c)
- Add `#include <time.h>` to delay.c (already in delay.h)
- Update delaymgr_init() to store precision_ms parameter
- Update all call sites: cluster.c (2→200ms), controller/testsuite/test.c (2→2000ms),
lib/testsuite/test_delay.c (1→1000ms)
- Add TDD test test_delaymgr_precision_ms_field verifying field exists and stores correctly
- delay_task() delay parameter remains in seconds; only sleep granularity changes
rtti header already present at rtti-0.0.2 with vfreeze_fn_t, vthaw_fn_t,
retain_fn_t, and the two new slots in intf_varray_t. No header modification
needed.
Convert intf_varray_t compound-literal initializers in zarray.c and kvmgr.c
to designated style with explicit .vfreeze = NULL and .vthaw = NULL slots.
kv.c has no intf_varray_t initializer.
Add test_zarray_cow.c with test_freeze_thaw_slots_exist which verifies the
struct layout has the slots accessible and that they are NULL until P0-2
wires the real implementations. Wire test into meson.build and test.c.
Full clean rebuild confirmed: 27/28 targets linked, no new test failures.
Only the 4 pre-existing failures remain.
