- Add OPCode_probeConfigTag dispatch in cluster.c
- Implement probe_config_tag handler returning config tag via
standard xce_reply() → RequestReply envelope
- Populate redirectHost/redirectPort for follow_redirects support
- Wire probeConfigTag reply through typemap
- Add probeConfigTagReply struct to controller.capnp schema
handle_repl_config_push now sends a ReplConfigPushReply{accepted,term}
from every exit path instead of silently dropping on the four reject
conditions (malformed request, missing bytes/sha256, sha256 mismatch,
conf_save_config failure). Term is sourced live from c->ha.term.
-EAGAIN activation-defer still replies accepted=1, unchanged.
Adds an XCE_TESTING-only send-hook mirroring ha_wal_ack_send_hook so
tests can observe the reply payload without a live endpoint.
myconf->lock (mylock_init/fini wiring) had landed but load_primary_config()
itself was left in its RED-phase reproduction state (unguarded, decoding
directly into the shared myconf->cluster_config field), so the fix's own
regression test failed on resume: thread B raced through while thread A was
parked mid-call.
Wraps the whole free-decode-publish sequence in myconf->lock (acquired first,
released on every return path) and decodes into a local cluster_config_t*
first, publishing into myconf->cluster_config only after full decode and
validation succeed -- closing the UAF/double-free window a live 3-node ASan
soak caught (two concurrent handle_repl_config_push deliveries for the same
standby racing two load_primary_config calls).
Verified 4/4 clean full-suite runs (1036 tests, 0 failures) including the
specific race test.
Adds checkpointTerm (@6, append-only) to the SnapshotTransfer wire
message. The leader stamps it via persist_entry_term(p, checkpoint_lsn)
when sending a snapshot; the standby reconciles it into its own
per-LSN term index (persist_reconcile_checkpoint_term) right after
persist_load_snapshot succeeds, before applied_lsn advances. This
authoritatively overwrites any stale local term a divergent
pre-snapshot entry left at the checkpoint LSN, closing the gap task
3.1's RED test proved (Expected 7 Was 3 -> now passes). Transport
stays persist_t-free; the reconcile lives in persist.c per the
existing dependency direction. Decode is auto-generated via capnp
mapname; only the hand-rolled encode path (request.c/typemap.h, needed
for the raw snapshot payload) required a matching manual update.
test_ha_core_safety 25/0, test_config_repl 66/0, no ASAN reports.
Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.
Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
Wires do_upgrade_config_with_bytes to append WAL_CONFIG_MARKER through
raft_log_append (live term-stamped consensus record), demotes committed_config_tag
to a non-authoritative cache, and derives the committed config from the latest
committed marker. Preserves the derive across checkpoint compaction via an optional,
backward-compatible config-ref tail in the snapshot sentinel.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
- SEND_SNAPSHOT uses checkpoint_lsn (not committed_lsn) for file lookup
- ha_send_snapshot_to_standby stats file for actual total_size
- ha_compute_oldest_wal_lsn scans WAL dir for min segment LSN
- Removed hardcoded 131072 and oldest_wal_lsn=0 placeholders
ha_stream_wal_to_standby() reads WAL from committed_lsn range, accumulates
entries to 256/64KB per batch, enqueues all via xce_request() (non-blocking).
Partial final batch sent immediately. Wired into STREAM_WAL case.
3 tests: 500-entries/2-batches, partial batch, empty range.
Fixed stale task-number comments per review.
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
Pre-decode opcode peek in myreq_handler() captures raw packet bytes for
OP_upgradeConfig before xce_decode() frees the capnp segment. Passes
raw bytes to xce_upgrade_config_req_extract_raw() to extract sha256 and
config blob, then delegates to do_upgrade_config_with_bytes().
Removes Task 11 dependency. Phase 2 still gated by CONF_ENABLE_UPGRADE_CONFIG.
Replaces blocking pthread_join() per thread with a CLOCK_MONOTONIC
deadline + 5ms poll loop. CONF_PUSH_TIMEOUT_MS (500ms) now enforced.
Matches design §6.3 aggregate timeout guarantee.
- Add done field to push_ctx_t; push_thread sets it on exit
- Add static wait_all_with_timeout() before push_config_to_standbys()
- Wire push_config_to_standbys() to use wait_all_with_timeout()
- Add #include <stdatomic.h> to config_repl.c
- Guard push_ctx_t local typedef with #ifndef XCE_TESTING
- Expose push_ctx_t and wait_all_with_timeout_for_test() in header
under XCE_TESTING for unit test access
- Add test_wait_all_with_timeout_slow_threads (timeout fires, acked=0)
- Add test_wait_all_with_timeout_fast_threads (all complete, acked=3)
- Update design doc §6.3 with actual implementation signature and
5ms poll interval note
- create_lost_and_found(): provide hostattrs_t so makeWorker succeeds
- collect_queues/collect_workers (dataex.c): skip lost_and_found sentinel
so it is never pushed to apiserver or scheduler
- mark_worker_unavail(): reset hosttype/hostmodel to "unknown" and zero
all numeric attrs, then update resvals and slots — prevents stale
capacity from being visible while the host is down
- restore_worker(): apply the same attr reset on WAL recovery so
restored-but-unavailable workers advertise no capacity until reconnect;
also remove dead calloc() that was immediately overwritten by makeWorker
- Extract reset_hostattrs_unknown() helper to eliminate copy-paste
- Define XCE_LOST_AND_FOUND_NAME constant in config_repl.h; replace all
six raw "lost_and_found" string literals across config_repl.c and dataex.c
- load_primary_config(): resolves conf/current, reads raw bytes, computes
SHA-256 tag, decodes Cap'n Proto via load_cluster_config_from_file(),
checks reserved names, sets config_generation, checks staleness
- myconf_fini(): frees conffile and cluster_config (calls free_ClusterConfig)
- Remove static cf global from myconf.c; get_default_conf() now uses a
local struct cluster_configuration* via load_cluster_config()
- Add load_primary_config_for_test() under XCE_TESTING in config_repl.c
to test path resolution without triggering FATAL
- Add xce/cluster_def.h include to config_repl.c and test_config_repl.c
so cluster_t is complete when XCE_TESTING is active
- TDD: test_load_primary_config_missing_returns_error passes (18/18)
