sel_key_cmp_fn compared keys with memcmp on the whole sel_node_key_t
struct, including the 4-byte padding between type and op1. The stack-
allocated search key in gc_sweep_select_nodes can have different padding
bytes than the heap-allocated copy stored by key_add, causing key_del
to miss the entry and leave a dangling pointer in the CSE intern table.
Replace memcmp with field-by-field comparison on type, op1, op2.
Also fix GC sweep tests that asserted pointer inequality after re-intern:
calloc may lawfully reuse the freed address, so pointer inequality does
not reliably indicate a successful sweep. Track the swept node's
node_id before freeing and assert the re-interned node gets a fresh id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
Three fixes:
1. ExecHosts.UnmarshalJSON: JSON null (sent when no allocation exists)
was treated as a non-"-" string because json.Unmarshal(null, &str)
returns nil with str="". This created a phantom HostSlot{NumSlots:1,
HostName:""} producing "<1*>" for every PEND job. Now null and ""
are both treated as empty hosts.
2. page_print_jobs.go: added !startTime.IsZero() guard so a zero
timestamp ("0001-01-01T00:00:00Z") never produces a start line
even if the status check is bypassed.
3. Added PENDING_REASON field to JobFieldsMap so it is requested from
the API when -l is passed, and rendered in printJobsVerboseOutput
for PEND/PSUSP jobs before the start-time section.
Four bugs fixed:
1. persist.c: array job WAL loop aborted after index [0] when replication
returned -ENODEV (no standbys). Loop guard now uses the return value of
persist_wal_sync_append() which is 0 on local-disk success regardless of
replication status.
2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
up the existing slot by name, updates its endpoint, and clears UNAVAIL so
the normal hook/reply/on_worker_join flow proceeds.
3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
connection dropped because no error handler was registered. Added
handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
checking_connection() to detect the failure and rejoin.
4. cluster.c: persist_recover() was called before cluster_build_resources()
populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
so resmgr is fully populated before WAL replay.
All three persist hook functions (job/worker/queue) were ignoring the
situation argument, so they fired on both BEFORE and AFTER events.
BEFORE_EVENT hooks pass different va_list layouts than AFTER_EVENT:
- WORKER_JOIN BEFORE: (cluster_t*, char* name, int 0) vs AFTER: (cluster_t*, worker_t*)
- JOB_NEW BEFORE: (cluster_t*, xce_job_submit_t*) vs AFTER: (cluster_t*, xlist_t*)
Reading the wrong type from va_list caused a dereference crash.
Fix: return 0 immediately on HOOK_BEFORE_EVENT in all three hooks.
WAL payloads:
- Job state transitions (START/FINISH/FAIL/STOP/RESUME/UPDATE etc):
wal_jobstate_t {uint64_t jobid, uint32_t state} — restores exact state flags
- Queue state (QUEUE_CTRL): wal_queuehdr_t {uint32_t status} + name string —
re-applies admin open/close/active/inactive after config queues are built
- Worker CTRL: existing [uint32_t detail][capnp xce_worker_t] unchanged
New recovery functions:
- restore_job_state(c, jobid, state): looks up job, overwrites state flags
- restore_queue_state(c, name, status): looks up queue, overwrites status
apply_wal_entry: dispatches all WAL types including job state and queue
test_repl_apply.c: weak stubs for restore_job_state/restore_queue_state
so unit test binary links without job.c/queue.c
WAL hooks now write real Cap'n Proto payloads instead of stub uint32_t:
- WAL_JOB_SUBMIT: full xce_job_t serialized via encode_Job/write_Job
- WAL_WORKER_CTRL: [uint32_t detail][xce_worker_t capnp] via encode_Worker
- Other job events (START/FINISH/FAIL/STOP etc): uint64_t jobid only
restore_job(): deserializes Cap'n Proto xce_job_t, guards against duplicate
replay with key_find check, then calls create_job()
worker_from_capnp(): skips 4-byte detail prefix, deserializes xce_worker_t,
guards against duplicate replay, then calls restore_worker()
join.c: pass worker_t* (looked up by idx via vget) to WORKER_JOIN hook
so persist hook can serialize worker data without touching workermgr RTTI
Replace all fprintf(stderr,...) calls with INFO/WARN/ERROR/TRACE so
persist.c output appears in the controller's structured log rather than
mixing plain text on stderr with tlog-formatted output.
Three identical static copies (controller, worker, scheduler) replaced
with a single xce_parse_peers() in src/lib/cluster.c.
The libxce version is the canonical implementation: stores the logical
name from 'name=host:port' syntax so daemons on the same dev host can
identify themselves by name rather than IP address.
- xce_candidate_t gains name[] field for logical peer identity
- parse_peers stores name prefix before '=' (empty when omitted)
- cluster_init matches own identity by name first, hostname as fallback
enabling multiple controllers on the same dev host via
--peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7001
- myconf_t gains peers_cfg (xce_cluster_config_t) populated from --peers
- controller/main.c: parse --peers host:port,... via argp; ca_cert_path
auto-derived from workdir/certs/xce.crt (no --ca-cert flag needed)
- cluster_init: derive my_priority from position in peers list (index 0 =
primary); fall back to 0 when --peers not given (single-node mode)
- Add HA_DEFAULT_* constants to ha.h; use them in cluster_init()
- Strip stale task-label comments (P1-3a, P1-5, P1-6, P3-6 refs)
- Remove self-documenting 'Step N:' annotations from ha_do_promote/demote
- Rewrite stale forward-reference notes as current WHY comments
- Remove resolved TODO P0-8b from cluster.c
- parsePeer: use net.SplitHostPort (handles IPv6, stdlib convention)
- tryConnect: take Go strings, own CGO allocation internally
- RegisterToBackend: remove unreachable 'no HA candidates' error path
- Sync Peers field comment between utils and backend config structs
Worker (xcecd):
- Add xce_cluster_config_t cluster_cfg field to controller_connector_ctx_t
(controller_connector.h includes xce/cluster.h)
- Populate cc_ctx->cluster_cfg from parsed cluster_cfg after xcecd_start()
sets cc_ctx (main.c, removes the P3 placeholder)
- Replace xcecd_join_cluster() single-shot connect with a candidate-iterating
loop: tries each candidate, calls xcecli_stop() before the next attempt,
falls back to the original single-host path when candidates_count == 0
Scheduler (xce_scheduler):
- Add xce_cluster_config_t cluster_cfg field to xce_scheduler_t
(scheduler.h includes xce/cluster.h)
- Add sched_set_cluster_cfg() setter
- Replace the hardcoded xcecli_start/xcecli_join pair in sched_init() with
the same candidate-iterating pattern; original single-host path preserved
- main.c calls sched_set_cluster_cfg() before sched_init() when --peers given
(removes the P3 placeholder comment)
checking_connection() already calls xcecd_join_cluster() on retry, giving
the worker free reconnect-after-failover behaviour for no additional code.
Worker (P2.5-5):
- Added parse_peers() static helper to src/worker/src/main.c
- Switched getopt to getopt_long to support --peers and --ca-cert long options
- When --peers is given: parses into xce_cluster_config_t, calls
xce_cluster_config_validate(), logs INFO with candidate count/identity/ca_cert
- If validation fails: prints clear error and exits(1)
- WARN logged when candidates_count > 3
- Backward compatible: no --peers falls back to existing single-controller path
Scheduler (P2.5-6):
- Added parse_peers() static helper to src/scheduler/src/main.c
- Extended argp struct with --peers (-P) and --ca-cert (-C) options
- Same validation/logging pattern as worker
- Backward compatible: no --peers falls back to existing single-controller path
Tests (new):
- src/lib/testsuite/test_cluster_parse.c: 8 unit tests for parse_peers logic
(single entry, multiple entries, no-name format, empty/null/missing-port errors,
name-stripped correctly, MAX_HA_CANDIDATES cap respected)
- Registered in test.c and meson.build
- All 8 tests GREEN; 3 pre-existing failures in unrelated GC/CSE tests unchanged
Add test-only function ha_trigger_manual_failover_for_test() to exercise manual
failover state machine (demote → promote cycle) without requiring live cluster_t.
Test coverage:
- test_manual_failover_transitions_role_and_increments_term: Validates role
transitions (PRIMARY → STANDBY → PRIMARY) and term monotonicity
- test_manual_failover_resets_hooks_registered: Confirms demotion resets the
hooks_registered idempotency guard
- test_manual_failover_monotonic_term_across_multiple_cycles: Validates term
remains monotonically increasing across 5 consecutive failover cycles
Implementation:
- Exposed only under #ifdef XCE_TESTING in ha.h
- Function body (6 lines): calls ha_do_demote() then ha_do_promote(); returns
the new term value via atomic_load_explicit()
- Enables design validation (P3-6) without test-cluster infrastructure
All tests pass. Build clean.
Add test_p2_integration.c with 7 integration tests covering the assembled P2
replication system: WAL_CHECKPOINT assert invariant, trim_lsn with active standby,
checkpoint must not update last_replicated_crc, sync degraded mode, full catch-up
lifecycle, needs_snapshot boundary conditions, and API completeness gate.
All 32 controller unit tests pass (0 failures).
Implement the catch-up state machine as testable pure-logic functions:
- repl_start_catchup(): marks standby in_catch_up=1, stores join_applied_lsn
- repl_complete_catchup(): clears in_catch_up=0, broadcasts ack_cond so any
blocked repl_wait_ack() re-checks the ACK condition immediately
- repl_needs_snapshot(): pure predicate — returns 1 when standby LSN predates
oldest surviving WAL segment (snapshot transfer required), 0 otherwise
Also fixes pre-existing linker error: repl.c was missing from controller_src
(xce_controller production binary), causing undefined references to repl_*
symbols introduced in P2-4.
7 new TDD tests in test_repl_catchup.c — all pass. Existing tests green.
- Move repl.h from controller/ to include/ (shared header) to break
persist.h ↔ repl.h circular include
- Extract wal_entry_hdr_t + wal_entry_type into include/wal_types.h so
repl.h can include only what it needs without pulling in all of persist.h
- Add replicator_t repl field to persist_t (embedded by value, P2-4)
- Wire repl_init/repl_fini into persist_init/persist_fini lifecycle
- Wire repl_broadcast into persist_wal_async after each WAL write;
last_replicated_crc updated here (not in persist_write_checkpoint_marker)
- Wire repl_wait_ack into persist_wal_sync_append; returns 0 on local
WAL success; sync_rc_out carries -ENODEV/-ETIMEDOUT for degraded mode
- Add 4 TDD tests in test_persist_repl.c: structural repl field check,
async no-standbys succeeds, sync degraded mode rc, checkpoint CRC invariant
Implements sync replication wait: primary blocks until at least one non-catching-up
standby ACKs the target LSN, or times out. Uses pthread_cond_timedwait with
CLOCK_REALTIME deadline. Inner loop rechecks all standbys after each wakeup to
guard against spurious wakeups.
ACK deadlock prevention: repl_on_ack() is called exclusively from the io_worker
read callback (not vtaskmgr), so it only does atomic_store + pthread_cond_signal —
safe to call while vcpu threads block in repl_wait_ack().
4 new TDD tests (all green, 9 total in test_repl_unit):
- test_wait_ack_no_standbys_returns_enodev
- test_wait_ack_already_acked
- test_wait_ack_timeout (verifies >=80ms elapsed)
- test_wait_ack_woken_by_background_ack (pthread thread signals after 30ms)
Upgrade apply_wal_entry() from P0 static no-op stub to the real public
standby replay path:
- Reject oversized payloads (> MAX_WAL_PAYLOAD_BYTES) with -EMSGSIZE
- Skip WAL_CHECKPOINT (100) entries silently — local-only markers, never
replicated to standbys; current_lsn is NOT advanced on skip
- Skip unknown types (> WAL_ENTRY_TYPE_MAX) with WARN log and rc=0 for
forward compatibility with WAL written by newer controller versions
- Advance current_lsn to hdr->lsn+1 and update last_crc under p->lock
for all accepted known entries
- Gate restore_job()/worker_from_capnp() calls on p->cluster != NULL so
unit tests work without a full cluster wired in
- Add NULL guard returning -EINVAL for missing p or hdr
Add public declaration in persist.h with forward-declare stubs for
restore_job/worker_from_capnp in persist.c (avoids heavy cluster.h chain).
Test suite: 5 new Unity tests in test_repl_apply.c covering all branches.
Weak stubs for restore_job/worker_from_capnp in the test file satisfy the
linker without linking job.c/worker.c's heavy dependency graph.
All 5 new tests pass; full controller suite green.
Add replicator_t structure with per-standby connection slots (standby_conn_t),
lifecycle (repl_init/repl_fini), slot management (repl_add_standby,
repl_remove_standby, repl_expire_disconnect_slots), ACK tracking
(repl_on_ack), WAL trim guard (repl_trim_lsn), and stubs for
repl_broadcast/repl_wait_ack (wired in P2-3/P2-4).
Uses mylock_t (pthread_mutex_t alias) with mylock_lock/mylock_unlock.
snap_interval_s==0 in repl_expire_disconnect_slots forces immediate expiry
to support deterministic unit testing.
5 new Unity tests all pass; 115 pre-existing controller tests unchanged.
Add redirect helpers to ha.c/ha.h so a standby controller never accepts
join requests as if it were the primary:
- ha_should_redirect(): returns 1 when role is STANDBY and primary_host/port
are known — caller should send E_XCE_REDIRECT.
- ha_no_primary_known(): returns 1 when role is STANDBY but primary address
is not yet known — caller should send E_XCE_NOMASTER (retry later).
- Add E_XCE_NOMASTER and E_XCE_REDIRECT to errnox.h.
- Wire both guards into handle_join_() in join.c (before the role switch)
so all join types (worker, scheduler, apiserver, client) are protected.
- Full redirect response sends status=-E_XCE_REDIRECT with nodata body;
the primary host:port are logged so operators can trace misrouted clients.
Full Cap'n Proto RedirectInfo response body is deferred to a follow-up
(schema union would require regenerating capnp bindings).
5 new TDD tests — 110 total, 0 failures.
- ha_do_promote(): atomic_fetch_add term, set role=PRIMARY, reset
last_received_crc to WAL_CRC_NONE (COH-5 chain restart on promotion)
- ha_do_demote(): set role=STANDBY, reset hooks_registered idempotency guard
- ha_install_sigusr1(): sigaction handler stores only atomic int —
async-signal-safe per POSIX.1-2017 §2.4.3; global g_sigusr1_ha set
once before signals are enabled
- ha_promote_to_primary() / ha_demote_to_standby() remain (void)c stubs
pending P1-5 cluster wiring
Tests: 3 new cases in test_ha_role.c
test_promote_increments_term: term > term_before, role == PRIMARY
test_demote_sets_standby_role: role reverts to STANDBY after promote+demote
test_sigusr1_sets_failover_requested: raise(SIGUSR1) sets atomic flag
Result: 100 Tests 0 Failures 0 Ignored (controller suite)
- Add ha_start_watchdog() / ha_stop_watchdog() — dedicated pthread with
nanosleep(heartbeat_interval_ms) loop; stop via atomic watchdog_stop +
pthread_join (no pthread_cancel, safe for OpenSSL)
- Add ha_check_heartbeat_nocluster() — reads role with
atomic_load_explicit(memory_order_acquire), guards on primary_host[0]=='\0',
computes elapsed_ms from CLOCK_MONOTONIC, sets failover_requested on timeout
- Update ha_fini() to call ha_stop_watchdog() before mutex destroy
- 3 new tests (TDD RED→GREEN): watchdog starts/stops, no-op when no primary,
stale heartbeat no crash; all 97 controller tests green
- Full ha.h: ctrl_role_t enum (CTRL_PRIMARY/CTRL_STANDBY), ha_config_t,
ha_state_t with _Atomic fields (term, role, applied_lsn, promotion
counters, watchdog_stop), pthread_mutex_t lock, and public API.
- endpoint_t* omitted from ha_state_t — including netio.h would pull
OpenSSL headers and risk circular deps; watchdog (P1-3) includes
netio.h directly where needed.
- ha_init validates: heartbeat_interval_ms >= 2000, heartbeat_timeout_ms
>= heartbeat_interval_ms, election_timeout_ms >= 500; returns -EINVAL
on bad config; zeros struct and sets initial role to CTRL_STANDBY.
- ha_fini destroys mutex; watchdog thread join deferred to P1-3.
- am_i_primary / ha_promote_to_primary / ha_demote_to_standby are
no-op stubs; cluster_t->ha field not yet wired (P1-5), so these avoid
incomplete-type dereferences until then.
- TDD: 5 new tests in test_ha_role.c confirmed RED before implementation,
all 5 GREEN after; all prior controller tests remain green.
- hook_item_t.fn changed to _Atomic hook_fn_t (C11 stdatomic)
- xce_disable_hook(): atomically stores NULL into the slot fn field;
returns 0 on success, -ENOENT if fn not found, -EINVAL on bad args
- xce_add_hook(): scans for NULL (disabled) slots before allocating;
uses atomic_store_explicit(release) when writing fn pointer
- xce_run_hooks(): atomic_load_explicit(acquire) + NULL guard so
disabled slots are silently skipped without disrupting the loop
- Add src/lib/testsuite/test_hooks.c with 3 TDD tests:
test_disable_hook_stops_invocation
test_disable_hook_returns_enoent_if_not_found
test_add_hook_reuses_null_slot
- Register test_hooks in meson.build and test.c runner
Required by ha_demote_to_standby() to atomically disable persistence
hooks when the controller steps down from primary, preventing unbounded
hook list growth across promote/demote cycles (resolution A6-015,
A7-004, A7-016).
Add terminal-state constants and GC retention predicates for the
checkpoint xzarray_thaw() GC pass:
- types.h: add JOB_STAT_TRANSIENT (0x400), XCE_WORKER_HIDDEN (0x00200000),
QUEUE_STAT_HIDDEN (0x0100)
- persist.c: implement job_is_terminal(), job_should_retain(),
worker_should_retain() — placed here so the unit test binary
(which links only persist.c) can access them without RTTI deps
- persist.h: declare the three GC predicates; include xce/types.h
- worker.c: set dead_since = time(NULL) in mark_worker_unavail()
when dead_since is not already set
- cluster.c: add TODO P0-8b marker on zarray_del endpoint cleanup
- meson.build: register test_gc_retention.c; raise test timeout 60→120s
- testsuite/test_gc_retention.c: 19 tests covering job_is_terminal,
job_should_retain (TRANSIENT TTL, terminal TTL, non-terminal always
retained), and worker_should_retain (HIDDEN/UNAVAIL/healthy/zero
dead_since); all pass (63 total, 0 failures)
Implements persist_job_hook_fun, persist_queue_hook_fun, and
persist_worker_hook_fun — AFTER_EVENT hooks that write WAL entries
whenever state mutations occur in the controller.
- detail_to_wal_type() maps all 15 HOOK_DETAIL_* values to WAL types;
unknown details return 0 and are silently skipped
- HOOK_DETAIL_JOB_NEW uses the sync replication path (WAL_JOB_SUBMIT);
all others use persist_wal_async
- P0 payload: 4-byte detail value; full Cap'n Proto serialization in Task 15
- Hook signature matches hook_fn_t exactly:
int fn(int type, int situation, int detail, void* cls, va_list ap)
- Added <stdarg.h> to persist.h for va_list; added hook.h and
xce/cluster_def.h includes to persist.c
- 18 new unit tests in test_persist_hooks.c covering all 15 detail→WAL
mappings, skip-on-unknown, and sequential LSN advancement
- cluster_def.h: add struct tag `cluster_s` to cluster_t to resolve
conflicting typedef with persist.h forward declaration
- cluster_def.h: add HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME enum
values after HOOK_DETAIL_JOB_RUSAGE_UPDATE
- job.c on_job_stop(): replace mark_job_update() call with direct
xce_run_hooks(BEFORE/AFTER, HOOK_DETAIL_JOB_STOP) so WAL writer
records stop events with the correct detail value
- job.c on_job_resume(): same pattern with HOOK_DETAIL_JOB_RESUME
- job.c do_clr_preason(): fix typo — second hook call was HOOK_BEFORE_EVENT
instead of HOOK_AFTER_EVENT (double-before, zero-after bug)
- join.c handle_worker_join(): move xce_reply() to after HOOK_AFTER_EVENT
so client ACK is not sent before WAL write completes (durability fix)
- testsuite/test_hooks.c: new TDD tests for JOB_STOP/RESUME enum values
- testsuite/controller_test.c + meson.build: register test_hooks suite
Reply ordering audit results:
job.c handle_jobsubmit: reply before AFTER_EVENT — intentional (jobid
returned to client before scheduler notification; WAL hook belongs in
BEFORE_EVENT for this path, handled in Task 13)
job.c do_set_preason: reply before set_preason() loop — intentional
fire-and-forget (no AFTER_EVENT hook at top level)
job.c do_clr_preason: reply before loop — same fire-and-forget pattern
job.c do_jobcontrol_op: no AFTER_EVENT hook, reply only — OK
join.c handle_worker_join: FIXED (reply moved after AFTER_EVENT)
worker.c on_worker_update/dead: reply after all hooks — OK
cluster.c: AFTER_EVENT hooks are internal (no reply) — OK
