Commit graph

14 commits

Author SHA1 Message Date
6b69a47c50 feat(ha-repl-hardening): batch WAL streaming from decision loop (task 3.4)
ha_stream_wal_to_standby() reads WAL from committed_lsn range, accumulates
entries to 256/64KB per batch, enqueues all via xce_request() (non-blocking).
Partial final batch sent immediately. Wired into STREAM_WAL case.
3 tests: 500-entries/2-batches, partial batch, empty range.
Fixed stale task-number comments per review.
2026-06-04 17:57:55 +00:00
3d076ccda6 fix(ha): fix config replication to standbys; add replConfigPush handler
- config_repl.c: add handle_repl_config_push — standby handler for
  OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
  conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
  populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
  instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
  real config bytes and tag read from disk (was NULL/0 — standbys got
  empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
  repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
  before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
  so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
  ha_find_and_connect_to_primary declaration.
2026-06-01 05:56:16 +00:00
dbb750ebd3 fix(election-iomgr-refactor): Raft election, config decode, and RDHUP chain
- ha.c: self-vote in ha_start_election_async (Raft invariant)
- ha.c: heartbeat_timeout_ms for hb_timeout_thread (not election_timeout_ms)
- ha.c: clear voted_for before RDHUP jitter (stale vote from dead primary)
- ha.c: same-term voted_for check in RDHUP jitter
- config_repl.c/h: use xce_decode for upgradeConfig (capnp extraction before capn_free)
- cluster.c: fix upgradeConfig handler direction (standbys apply, primaries skip)
- cluster.c: remove raw_pkt_copy hack; use req directly
- join.c: push config to newly-joined standbys
- typemap.h/request.h/request.c: raw_sha256/raw_bytes fields for decoded upgradeConfig
- test_ha_failover.c: update heartbeat timeout test to use g_fast_cfg
2026-05-31 03:54:30 +00:00
fd93997f78 fix(election-iomgr-refactor): config replication and RDHUP failover chain
- cluster.c: fix upgradeConfig handler (standbys apply, not primaries)
- cluster.c: fix raw_pkt_copy opcode peek (read scalar before capn_free)
- config_repl.h: enable CONF_ENABLE_UPGRADE_CONFIG=1
- config_repl.c: use client_ssl_ctx for outgoing config fetch connections
- config_repl.c: after config fetch, run cluster_init_phase2 + find_primary
- join.c: push config to newly-joined standbys on OPCode_join
- myapp.c: schedule fetch_config_from_primary after skipping own election
- ha.c: arm heartbeat timeout watchdog after find_and_connect_to_primary
2026-05-31 02:08:48 +00:00
27ea14cd9a feat(election-iomgr-refactor): remove old blocking election code (am_i_primary_quorum, vote_request_thread, bootstrap_config_if_needed) — task 5.3 2026-05-30 13:04:19 +00:00
94bf5d94b8 feat(ha-config-bootstrap): implement OP_ctrlHello state sync handler (task 9.3) 2026-05-29 05:22:17 +00:00
ab174c8b06 feat(ha-config-bootstrap): implement bootstrap_config_if_needed() (task 5.3)
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
  none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
2026-05-28 09:48:24 +00:00
5053c8d42f feat(config-repl): wire handle_upgrade_config() via raw packet interception
Pre-decode opcode peek in myreq_handler() captures raw packet bytes for
OP_upgradeConfig before xce_decode() frees the capnp segment. Passes
raw bytes to xce_upgrade_config_req_extract_raw() to extract sha256 and
config blob, then delegates to do_upgrade_config_with_bytes().

Removes Task 11 dependency. Phase 2 still gated by CONF_ENABLE_UPGRADE_CONFIG.
2026-05-21 13:51:52 +00:00
f3babec5bc feat(config-repl): implement wait_all_with_timeout() for push fan-out
Replaces blocking pthread_join() per thread with a CLOCK_MONOTONIC
deadline + 5ms poll loop. CONF_PUSH_TIMEOUT_MS (500ms) now enforced.
Matches design §6.3 aggregate timeout guarantee.

- Add done field to push_ctx_t; push_thread sets it on exit
- Add static wait_all_with_timeout() before push_config_to_standbys()
- Wire push_config_to_standbys() to use wait_all_with_timeout()
- Add #include <stdatomic.h> to config_repl.c
- Guard push_ctx_t local typedef with #ifndef XCE_TESTING
- Expose push_ctx_t and wait_all_with_timeout_for_test() in header
  under XCE_TESTING for unit test access
- Add test_wait_all_with_timeout_slow_threads (timeout fires, acked=0)
- Add test_wait_all_with_timeout_fast_threads (all complete, acked=3)
- Update design doc §6.3 with actual implementation signature and
  5ms poll interval note
2026-05-21 08:30:43 +00:00
1599d06bc4 feat: filter lost_and_found sentinel from pushes; reset unavail host attrs
- create_lost_and_found(): provide hostattrs_t so makeWorker succeeds
- collect_queues/collect_workers (dataex.c): skip lost_and_found sentinel
  so it is never pushed to apiserver or scheduler
- mark_worker_unavail(): reset hosttype/hostmodel to "unknown" and zero
  all numeric attrs, then update resvals and slots — prevents stale
  capacity from being visible while the host is down
- restore_worker(): apply the same attr reset on WAL recovery so
  restored-but-unavailable workers advertise no capacity until reconnect;
  also remove dead calloc() that was immediately overwritten by makeWorker
- Extract reset_hostattrs_unknown() helper to eliminate copy-paste
- Define XCE_LOST_AND_FOUND_NAME constant in config_repl.h; replace all
  six raw "lost_and_found" string literals across config_repl.c and dataex.c
2026-05-14 08:44:55 +00:00
5ef14a2717 feat: implement handle_upgrade_config() (gated), push_config_to_standbys(), create_lost_and_found() stub
- Add do_upgrade_config_with_bytes() helper with full validate/decode/
  re-encode/save/push/restart logic gated by CONF_ENABLE_UPGRADE_CONFIG
- Add push_config_to_standbys() stub: logs connected standbys and returns
  0; full parallel fan-out deferred to Task 11 (typemap byte shims needed)
- Add send_upgrade_error() private helper for consistent error replies
- Expand create_lost_and_found() stub with WARN log explaining Phase 2
  deferral; core config-replication feature works without it
- Declare push_config_to_standbys() in config_repl.h
2026-05-13 18:12:32 +00:00
6b06aa02d9 feat: split cluster_init() into phase1/phase2; update myapp_run() for two-phase startup with am_i_primary gating
- cluster_init_phase1(): hooks, HA init (priority from --peers), standby_endpoints zarray, cbuf
- cluster_init_phase2(): SSL ctx from ha.ssl_ctx, cluster_config guard, kvmgr sizing, iomgr, endpoint, persist, role promotion, create_lost_and_found, persist_recover
- cluster_init(): legacy wrapper calling both phases with inline SSL init for backward compat
- myapp_run(): phase1 → SSL init → am_i_primary → load_primary_config/fetch_config_from_primary → phase2 → start → run
- myreq_handler(): add OPCode_replConfig and OPCode_upgradeConfig dispatch
- handle_endpoint_error(): prune standby_endpoints on HostRole_controller disconnect
- cluster_run(): log restart_requested on exit
- config_repl.c: add create_lost_and_found, handle_config_request, handle_upgrade_config stubs
- config_repl.h: add declarations for new stubs; include netio.h + request.h
- cluster.h: declare cluster_init_phase1 and cluster_init_phase2
- myconf.c: fix FATAL → ERROR (FATAL macro undefined); add fetch_config_from_primary stub
2026-05-13 17:06:45 +00:00
cd02cf40d2 feat: add load_primary_config(), myconf_fini(); remove static cf global from myconf.c
- load_primary_config(): resolves conf/current, reads raw bytes, computes
  SHA-256 tag, decodes Cap'n Proto via load_cluster_config_from_file(),
  checks reserved names, sets config_generation, checks staleness
- myconf_fini(): frees conffile and cluster_config (calls free_ClusterConfig)
- Remove static cf global from myconf.c; get_default_conf() now uses a
  local struct cluster_configuration* via load_cluster_config()
- Add load_primary_config_for_test() under XCE_TESTING in config_repl.c
  to test path resolution without triggering FATAL
- Add xce/cluster_def.h include to config_repl.c and test_config_repl.c
  so cluster_t is complete when XCE_TESTING is active
- TDD: test_load_primary_config_missing_returns_error passes (18/18)
2026-05-13 16:06:18 +00:00
491c2550c2 feat: add config_repl Phase 1 — HEAD-file storage, SHA-256, trim, stale check, sanitise_cn
- config_repl.h/.c: atomic versioned file save, pointer file, trim, stale check
- test_config_repl.c: 17 Unity tests covering all Phase 1 storage functions
- typemap.h: add repl/heartbeat types + union members for xce_request_t/xce_reply_t
- clusterconfig.h: add config_generation field to cluster_config_t (capnp schema ref)
- meson.build: add config_repl.c and test_config_repl.c to build targets
- controller_test.c: register test_config_repl dispatch function
2026-05-13 15:45:28 +00:00