lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

build: remove nix dependency layer -- apt system libs + meson wrap subprojects

Browse source 
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).

Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
  libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
  meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
  subprojects under src/subprojects/ (source closure, ~500 files): the
  upstream trees plus packagefiles/ overlays and diff_files patches
  registering dependency/program overrides the nix .pc files used to
  provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
  Fresh-clone reproducibility verified: wiped trees + `meson subprojects
  download` reproduce the committed state exactly (fixed librtti.wrap's
  dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
  offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.

Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
  Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
  non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
  the dangling build RPATH makes ldd drop the very lines the naive filter
  needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
  (kernel does not parse quotes in shebangs), ships systemd unit samples +
  install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
  both bin/ wrapper and venv script.

Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
  loopback port so the CLI-error tests assert the no-controller failure
  path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
  sig_handler re-raises received signals via kill(0, sig) -- process GROUP
  wide (strace -k evidence) -- and the child previously shared the
  harness's pgrp, so an interrupted SSL handshake killed the whole suite
  (exit 130).

Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).

Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-20 18:04:16 +00:00
parent 1e5c85555a
commit 50cf424146
521 changed files with 106066 additions and 83 deletions
Download patch file Download diff file Expand all files Collapse all files

84
src/controller/testsuite/test_commit_barrier_rollout.c
View file

@ -1480,6 +1480,36 @@ static void cb5_arrange_upgrade_req(uint8_t *blob, size_t blen,
/* ── push_config_to_standbys: barrier result + push count (Req 4.2, 4.3) ── */
/* Concurrent-ACK injector: since repl_reset_config_acks (config-push
* quorum hardening) clears every standby's acked_config_lsn at the START
* of each push, a pre-ack installed before push_config_to_standbys no
* longer counts -- matching production, where a standby can only ever ack
* bytes it has already received. These tests therefore inject their ACKs
* from a helper thread while the main thread sits inside the barrier wait
* (same pattern as test_config_ack_wire_path_completes_barrier and the
* bug-1914 late-ACK test above). */
typedef struct {
cluster_t *c;
endpoint_t*eps[2];
uint64_t tag;
unsigned delay_ms;
} cb5_ack_inject_args_t;
static void *cb5_ack_inject_thread(void *arg)
{
cb5_ack_inject_args_t *a = (cb5_ack_inject_args_t *)arg;
usleep(a->delay_ms * 1000);
for (int i = 0; i < 2; i++) {
if (a->eps[i] != NULL)
repl_on_config_ack(&a->c->persist.repl, a->eps[i], a->tag);
}
return NULL;
}
/* statics survive Unity longjmp */
static cb5_ack_inject_args_t g_cb5_ack_inject;
/* ACKs withheld: the push barrier must report -ETIMEDOUT (return value),
* the transport push count via acked_out, and the tag must never commit.
* RED: the current return value IS the push count (2), and the internal
@ -1516,22 +1546,33 @@ static void test_push_config_withheld_acks_returns_etimedout(void)
/* Majority acks (self + 1 covering standby of n_peers=3): the push must
* return 0 and commit the tag. RED twice over: the return value is the
* push count (2 != 0), and the legacy internal wait (quorum 2 over
* n_active, no self-count) times out so the tag never advances. */
* n_active, no self-count) times out so the tag never advances.
* The covering ACK is injected from a helper thread mid-barrier: the
* push's own repl_reset_config_acks clears pre-installed acks (see the
* injector comment above). */
static void test_push_config_majority_acks_returns_zero_and_commits(void)
{
static const uint8_t blob[] = "cb5-push-blob";
uint8_t sha[32];
int pushed = -1;
int rc;
pthread_t th;
/* Arrange: one standby has already acked the in-flight tag */
/* Arrange: one standby acks the in-flight tag while the barrier waits */
cb5_setup();
SHA256(blob, sizeof(blob) - 1, sha);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[0], 0xAB12);
g_cb5_ack_inject.c = &g_cb5_c;
g_cb5_ack_inject.eps[0] = &g_cb5_sb_ep[0];
g_cb5_ack_inject.eps[1] = NULL;
g_cb5_ack_inject.tag = 0xAB12;
g_cb5_ack_inject.delay_ms = 700;
TEST_ASSERT_EQUAL_INT(0, pthread_create(&th, NULL, cb5_ack_inject_thread,
&g_cb5_ack_inject));
/* Act */
rc = push_config_to_standbys(&g_cb5_c, blob, sizeof(blob) - 1, sha,
0xAB12, &pushed);
pthread_join(th, NULL);
/* Assert */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
@ -1599,10 +1640,11 @@ static void test_upgrade_config_barrier_failure_error_reply_no_restart(void)
* restart_requested (Req 1.2/1.3/1.4/2.2/5.4). Materialization is now
* driven entirely by the marker-apply trigger (apply_committed_entry's
* WAL_CONFIG_MARKER case, persist.c, task 5), decoupled from this
* request-handling path. Both standbys pre-ack UINT64_MAX (covers any tag
* — the new tag is only known after conf_save_config hashes the re-encoded
* blob), so the case is green under both the legacy counting and the
* per-push arithmetic. */
* request-handling path. Both standbys ack UINT64_MAX from a helper thread
* mid-barrier (covers any tag — the new tag is only known after
* conf_save_config hashes the re-encoded blob, and the push's own
* repl_reset_config_acks clears pre-installed acks; see the injector
* comment above). */
static void test_upgrade_config_majority_ack_success_no_restart(void)
{
uint8_t *blob;
@ -1610,17 +1652,24 @@ static void test_upgrade_config_majority_ack_success_no_restart(void)
uint8_t sha[32];
int rc;
cb5_reply_obs_t obs = { 0 };
pthread_t th;
/* Arrange: full covering pre-ack from both standbys */
/* Arrange: full covering ACKs from both standbys, injected mid-barrier */
cb5_setup_upgrade();
blob = cb5_make_config_blob(&blen);
SHA256(blob, blen, sha);
cb5_arrange_upgrade_req(blob, blen, sha);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[0], UINT64_MAX);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[1], UINT64_MAX);
g_cb5_ack_inject.c = &g_cb5_c;
g_cb5_ack_inject.eps[0] = &g_cb5_sb_ep[0];
g_cb5_ack_inject.eps[1] = &g_cb5_sb_ep[1];
g_cb5_ack_inject.tag = UINT64_MAX;
g_cb5_ack_inject.delay_ms = 700;
TEST_ASSERT_EQUAL_INT(0, pthread_create(&th, NULL, cb5_ack_inject_thread,
&g_cb5_ack_inject));
/* Act */
rc = handle_upgrade_config(&g_cb5_c, &g_cb5_req_ep, &g_cb5_req);
pthread_join(th, NULL);
/* Assert: existing success reply unchanged; restart is NOT requested */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
@ -1682,16 +1731,22 @@ static void test_upgrade_config_marker_committed_hook_fires_once_after_commit(vo
uint8_t sha[32];
int rc;
cb5_reply_obs_t obs = { 0 };
pthread_t th;
extern int (*xce_upgrade_marker_committed_hook)(cluster_t *, uint64_t);
/* Arrange: full covering pre-ack from both standbys (same fixture as
* the no-restart success-path test) */
/* Arrange: full covering ACKs from both standbys injected mid-barrier
* (same fixture as the no-restart success-path test) */
cb5_setup_upgrade();
blob = cb5_make_config_blob(&blen);
SHA256(blob, blen, sha);
cb5_arrange_upgrade_req(blob, blen, sha);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[0], UINT64_MAX);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[1], UINT64_MAX);
g_cb5_ack_inject.c = &g_cb5_c;
g_cb5_ack_inject.eps[0] = &g_cb5_sb_ep[0];
g_cb5_ack_inject.eps[1] = &g_cb5_sb_ep[1];
g_cb5_ack_inject.tag = UINT64_MAX;
g_cb5_ack_inject.delay_ms = 700;
TEST_ASSERT_EQUAL_INT(0, pthread_create(&th, NULL, cb5_ack_inject_thread,
&g_cb5_ack_inject));
g_cb5_hook_calls = 0;
g_cb5_hook_last_tag = 0;
@ -1700,6 +1755,7 @@ static void test_upgrade_config_marker_committed_hook_fires_once_after_commit(vo
/* Act */
rc = handle_upgrade_config(&g_cb5_c, &g_cb5_req_ep, &g_cb5_req);
pthread_join(th, NULL);
/* Assert: fired exactly once, with the just-committed tag, and only
* after ha.committed_config_tag already reflects that same tag. */