Replaces blocking pthread_join() per thread with a CLOCK_MONOTONIC
deadline + 5ms poll loop. CONF_PUSH_TIMEOUT_MS (500ms) now enforced.
Matches design §6.3 aggregate timeout guarantee.
- Add done field to push_ctx_t; push_thread sets it on exit
- Add static wait_all_with_timeout() before push_config_to_standbys()
- Wire push_config_to_standbys() to use wait_all_with_timeout()
- Add #include <stdatomic.h> to config_repl.c
- Guard push_ctx_t local typedef with #ifndef XCE_TESTING
- Expose push_ctx_t and wait_all_with_timeout_for_test() in header
under XCE_TESTING for unit test access
- Add test_wait_all_with_timeout_slow_threads (timeout fires, acked=0)
- Add test_wait_all_with_timeout_fast_threads (all complete, acked=3)
- Update design doc §6.3 with actual implementation signature and
5ms poll interval note
load_primary_config() reads from workdir/name/conf — conf_save_config()
was writing to workdir/conf. Latent bug: would have corrupted upgrade
restart cycle once Task 11 wired the upgrade path.
Update all conf_read_current_filename, conf_get_saved_tag, and
conf_check_stale calls in the test suite to use the corrected
effective_dir (workdir/name) path structure. Add new test
test_conf_save_config_path_consistency that explicitly verifies the
written path matches what load_primary_config() reads.
Crash: encode_NameValue_list called with count=21, s=NULL on worker
rejoin, causing SIGSEGV in persist_worker_hook_fun.
Root cause: free_name_val_list(xce_name_val_t***, int) accepted count by
value. It zeroed *vals (resvals_to_sched = NULL) but could not zero
nresvals_to_sched. Any caller that freed the scheduler resource snapshot
(opget.c workerlist, reply.c clusterconf, dataex.c, sched.c dispatch)
left nresvals_to_sched stale. When the worker later rejoined and the
WAL HOOK fired, the WAL encoder read count=21 with a NULL pointer.
Fix: change signature to free_name_val_list(xce_name_val_t***, int*)
so both fields are zeroed atomically. Update all 7 call sites.
Root cause: test_heartbeat_stale_no_crash_without_primary had
heartbeat_timeout_ms=100 < heartbeat_interval_ms=3000, causing ha_init
to return -EINVAL before memset. The test ignored the return value and
called ha_fini on garbage stack memory. The watchdog_thread field
happened to hold the value 3 (from a prior stack frame), so
ha_stop_watchdog called pthread_join(3, NULL) — crash at 0x2d3
(__pthread_clockjoin_ex accessing thread-struct field at offset 0x2d0).
Fix 1 (test): valid config (interval=2000, timeout=3000). Staleness is
injected via last_heartbeat -= 10s, so timeout value was irrelevant.
Added TEST_ASSERT_EQUAL_INT check on ha_init return.
Fix 2 (ha_init): move memset before validation so ha_fini is safe to
call even after a failed init — prevents the same bug class in callers
that check ha_init's return late.
Verified: test_ha_role 11/11 PASS in both non-ASAN and ASAN builds.
Replace monolithic add_worker with three focused functions:
- apply_join_state (static under production, exported under XCE_TESTING):
shared status/resource update called by both create and rejoin paths;
clears XCE_WORKER_UNAVAIL, applies hconfig overrides, updates resources
- init_worker: first-time join path via makeWorker + apply_join_state
- rejoin_worker: reconnect path with explicit admin-bit preservation
(XCE_WORKER_ADMIN_MASK = LOCKEDU|CLOSED|HIDDEN|CLOSED_BY_ADMIN)
Also makes find_matched_hostconfig non-static so join.c can call it
via the extern declaration already in worker.h.
Removes add_worker (no longer declared in worker.h, no callers).
All 7 tests in test_worker_join pass:
apply_join_state (clears UNAVAIL, hosttype override, resource values),
init_worker (creates entry, returns correct idx),
rejoin_worker (updates attrs/ep, preserves admin bits).
- create_lost_and_found(): provide hostattrs_t so makeWorker succeeds
- collect_queues/collect_workers (dataex.c): skip lost_and_found sentinel
so it is never pushed to apiserver or scheduler
- mark_worker_unavail(): reset hosttype/hostmodel to "unknown" and zero
all numeric attrs, then update resvals and slots — prevents stale
capacity from being visible while the host is down
- restore_worker(): apply the same attr reset on WAL recovery so
restored-but-unavailable workers advertise no capacity until reconnect;
also remove dead calloc() that was immediately overwritten by makeWorker
- Extract reset_hostattrs_unknown() helper to eliminate copy-paste
- Define XCE_LOST_AND_FOUND_NAME constant in config_repl.h; replace all
six raw "lost_and_found" string literals across config_repl.c and dataex.c
- handle_config_request(): primary-side handler sends up-to-date flag or
version_tag; SHA-256 integrity check guards against in-place file changes;
sha256/bytes fields left NULL pending Task 11 typemap shims
- fetch_config_from_primary(): standby fetch with exponential backoff retry
(20 attempts, 500ms..5s); falls back to saved config via load_primary_config();
do_one_config_fetch() stubbed (-ENOSYS) until Task 11 wires capnp over SSL
- handle_controller_join(): registers standby endpoint in ha.standby_endpoints
via zarray_store() for OP_upgradeConfig fan-out; sends success reply
- Moved load_primary_config() from myconf.c to config_repl.c so the unit test
binary (which links config_repl.c but not myconf.c) can resolve it
- myapp.c: capture am_i_primary() into is_primary (avoids redundant
second probe call); after load_primary_config() succeeds, record
CTRL_PRIMARY role via atomic_store_explicit so cluster_init_phase2()
sees the correct role (ha_init() defaults to CTRL_STANDBY)
- myapp.c: add #include "ha.h" for CTRL_PRIMARY, atomic_store_explicit,
and memory_order_release
- cluster.c: cluster_init() wrapper now calls cluster_fini(c) before
returning on create_ssl_context failure, hook failure, and
cluster_init_phase2 failure — prevents SSL ctx and phase1 resource leak
- load_primary_config(): resolves conf/current, reads raw bytes, computes
SHA-256 tag, decodes Cap'n Proto via load_cluster_config_from_file(),
checks reserved names, sets config_generation, checks staleness
- myconf_fini(): frees conffile and cluster_config (calls free_ClusterConfig)
- Remove static cf global from myconf.c; get_default_conf() now uses a
local struct cluster_configuration* via load_cluster_config()
- Add load_primary_config_for_test() under XCE_TESTING in config_repl.c
to test path resolution without triggering FATAL
- Add xce/cluster_def.h include to config_repl.c and test_config_repl.c
so cluster_t is complete when XCE_TESTING is active
- TDD: test_load_primary_config_missing_returns_error passes (18/18)
- Rename replConfigRequestMsg/replConfigPushRequestMsg/upgradeConfigRequestMsg
to replconfigrequestmsg/replconfigpushrequestmsg/upgradeconfigrequestmsg in
Request.body union to match existing lowercase convention
- Rename replConfigReplyMsg/replConfigPushReplyMsg/upgradeConfigReplyMsg
to replconfigreplymsg/replconfigpushreplymsg/upgradeconfigreplymsg in
RequestReply.data union
- Swap UpgradeConfigRequest fields so sha256@0/bytes@1 matches sibling
ReplConfigPushRequest pattern
- Move HeartbeatMsg out of the config replication section into its own
'# ── HA heartbeat ──' section above the replication structs
When a worker re-joins and already exists in workermgr, makeWorker_()
now overrides the existing slot instead of returning EEXIST. Admin-set
flags (LOCKEDU, CLOSED, HIDDEN, CLOSED_BY_ADMIN) survive the override.
Add XCE_WORKER_CLOSED_BY_ADMIN flag to distinguish admin-closed hosts
from workload-closed ones. worker_ctrl sets/clears it alongside CLOSED.
Refactor worker_init/worker_destroy into copy_worker/kill_worker helpers
shared by both fresh creation and override paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sel_key_cmp_fn compared keys with memcmp on the whole sel_node_key_t
struct, including the 4-byte padding between type and op1. The stack-
allocated search key in gc_sweep_select_nodes can have different padding
bytes than the heap-allocated copy stored by key_add, causing key_del
to miss the entry and leave a dangling pointer in the CSE intern table.
Replace memcmp with field-by-field comparison on type, op1, op2.
Also fix GC sweep tests that asserted pointer inequality after re-intern:
calloc may lawfully reuse the freed address, so pointer inequality does
not reliably indicate a successful sweep. Track the swept node's
node_id before freeing and assert the re-interned node gets a fresh id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
Three fixes:
1. ExecHosts.UnmarshalJSON: JSON null (sent when no allocation exists)
was treated as a non-"-" string because json.Unmarshal(null, &str)
returns nil with str="". This created a phantom HostSlot{NumSlots:1,
HostName:""} producing "<1*>" for every PEND job. Now null and ""
are both treated as empty hosts.
2. page_print_jobs.go: added !startTime.IsZero() guard so a zero
timestamp ("0001-01-01T00:00:00Z") never produces a start line
even if the status check is bypassed.
3. Added PENDING_REASON field to JobFieldsMap so it is requested from
the API when -l is passed, and rendered in printJobsVerboseOutput
for PEND/PSUSP jobs before the start-time section.
