feat(component-failover-adoption): validate worker thread isolation, fix rejoin bugs found by E2E
Confirm connector thread is isolated from worker_process_update() (main thread) and resource_sync_thread() (own thread) via code read plus a real pthread concurrency test. Add unit and live 3-node E2E coverage for candidate/reconnect behavior, all-candidates-unreachable diagnostics, and liveness cadence under concurrent reconnect. The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed: xcecli_start() resets the reconnect callback to NULL after it runs, so registering it before xcecli_start() silently no-ops the whole rejoin mechanism; and a successful worker join returns its assigned slot index as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s rc == 0 success check never fired on a real non-zero-slot join. Both fixed, plus per-candidate names added to the all-unreachable failure log. Root-check bypass needed for the E2E run is gated at compile time via the existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd binary's root requirement has zero bypass surface.
This commit is contained in:
parent
331fa02423
commit
619b119ef5
SSH key fingerprint:
SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
5 changed files with 455 additions and 8 deletions
|
|
@ -41,7 +41,7 @@
|
|||
- _Requirements: 1.4, 2.5_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage
|
||||
- [x] 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage
|
||||
- Confirm the connector thread is isolated from the worker's other recurring liveness duties, and that a blocked connector thread during retry/backoff does not stall them
|
||||
- Add end-to-end coverage: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join to a surviving candidate without a worker restart
|
||||
- Add coverage for the case where every configured candidate is unreachable or rejects the join at startup: assert the worker reports a clear failure identifying every candidate it attempted, not a generic error
|
||||
|
|
@ -166,3 +166,4 @@
|
|||
- Task 1.3: closed with zero new code. Verified fresh (source read + a clean `xce-build.sh lib asan` run) that every item in its checklist was already satisfied by prior tasks: the `_Static_assert(N_MAX_MASTERS == MAX_HA_CANDIDATES, ...)` landed in task 1.1 (`test_libxce_retry.c:44`), and the four behavioral cases (order/no-branch/truncation/NULL-empty) landed in task 1.2's 8 tests (`test_cluster.c`). No implementer/reviewer cycle was dispatched since there was no diff to review — dispatching one would have either duplicated existing test names (build conflict) or been a no-op review of an empty diff.
|
||||
- Task 2.1: `xce_cluster_config_to_client_config()` (task 1.2) aliases `out->masters[i]` directly to `cfg->candidates[i].hostname` — no `strdup`. Any caller building the source `xce_cluster_config_t` on the stack and passing it through the helper then using the resulting client config beyond that stack frame's lifetime has a use-after-scope bug. Worker's fix keeps the source config in a file-scope `static` for this reason; tasks 3.1 (scheduler) and 4.2 (apiserver) hit the identical hazard and must do the same (or otherwise guarantee the source config outlives the client config's use).
|
||||
- Task 2.2: fixed the startup-ordering bug by threading `cluster_cfg` by value through `xcecd_config_t` → `xcecd_context_t` → `cc_ctx` (set in `main.c` before `xcecd_init()`/`xcecd_start()` run, instead of after). Safe as a plain struct copy because `xce_candidate_t.hostname` is a fixed `char[]` array, not a pointer — no aliasing hazard here (unlike the `xce_cluster_config_to_client_config()` output struct noted above). Scheduler (3.1) likely has the same ordering gap (`sched_init` loops candidates correctly already per design.md, so may not need this fix — verify against current source, don't assume). Rejoin-retry pattern established: `checking_connection()` checks `needs_rejoin` after every probe return, calls `xcecli_join()` directly (not full re-entry into the join-cluster function), clears the flag only on success — same pattern scheduler/apiserver rejoin tasks (3.1, 4.3) should follow.
|
||||
- Task 2.3 (real E2E caught what unit tests missed — 2 production bugs slipped through tasks 2.1/2.2's approved reviews): (1) `xcecli_start()` (`src/lib/client.c`) unconditionally resets `ctx->failover.rejoin_cb = NULL` as part of its own init — any `xcecli_set_reconnect_handler()` call before `xcecli_start()` is silently wiped. Register the reconnect handler AFTER `xcecli_start()` succeeds, not before. Scheduler/apiserver's own reconnect-handler registration (3.1, 4.3) must follow this same ordering or they'll hit the identical silent-wipe bug. (2) A successful `xcecli_join()`/worker join returns the assigned slot index as `reply.status` (any `rc >= 0`, not always exactly `0`) — checking `rc == 0` for success is wrong; use `rc < 0` as the only failure test (matches `xcecd_join_cluster()`'s own established convention). Any new code checking a join/rejoin call's success must use this same `rc < 0` convention, not `rc == 0`. (3) Security lesson: an env-var runtime bypass of a security check (`XCECD_ALLOW_NONROOT`) added for test/E2E convenience was rejected on review — the correct pattern for skipping a production-only check (root privilege, etc.) in a test/virtual build is a compile-time guard against the project's existing `VIRTUAL_CLUSTER` or `XCE_TESTING` flags, never a runtime environment variable reachable in the shipped production binary.
|
||||
|
|
|
|||
|
|
@ -214,6 +214,53 @@ static int build_cluster_config(void)
|
|||
return xce_cluster_config_validate(&g_cluster_cfg);
|
||||
}
|
||||
|
||||
/* Sized for the worst case: MAX_HA_CANDIDATES entries, each up to
|
||||
* (XCE_MAXHOSTNAME_LEN - 1) hostname chars + ':' + a 5-digit port + a 2-char
|
||||
* ", " separator (263 bytes/candidate at the max) -- the extra +16 pads that
|
||||
* comfortably rather than sizing exactly to the boundary. Named rather than
|
||||
* inlined at each call site so the callers below and the tests that call
|
||||
* format_candidate_list() directly can't drift out of sync with each other. */
|
||||
#define CANDIDATE_LIST_BUF_SIZE ((XCE_MAXHOSTNAME_LEN + 16) * MAX_HA_CANDIDATES)
|
||||
|
||||
/**
|
||||
* @brief Renders every candidate in g_cluster_cfg as "host:port, host:port,
|
||||
* ..." into buf (component-failover-adoption, task 2.3, Req 1 AC6,
|
||||
* Req 5 AC2).
|
||||
*
|
||||
* xcecd_join_cluster()'s all-candidates-exhausted failure paths only had a
|
||||
* candidate COUNT to log ("all %d candidate(s) unreachable") -- neither
|
||||
* xcecli_start() nor endpoint_connect() (src/lib/netio.c) log anything
|
||||
* per-candidate themselves (confirmed by source read: no XCECD_LOG/INFO/WARN
|
||||
* call anywhere in xcecli_start()'s per-master connect loop), so without
|
||||
* this the worker's own failure message never actually named which
|
||||
* candidates it tried, only how many -- a gap against Req 1 AC6 ("identifying
|
||||
* which candidates were attempted") and Req 5 AC2 ("identifying every
|
||||
* candidate attempted"). Found during task 2.3's all-candidates-unreachable
|
||||
* coverage.
|
||||
*
|
||||
* Callers must size buf as CANDIDATE_LIST_BUF_SIZE (or larger) -- with an
|
||||
* undersized buffer this truncates at the last candidate that fits rather
|
||||
* than overflowing (bufsz - 1 bound below), so a too-small buf loses
|
||||
* information, not memory safety.
|
||||
*/
|
||||
static void format_candidate_list(char* buf, size_t bufsz)
|
||||
{
|
||||
size_t used = 0;
|
||||
|
||||
buf[0] = '\0';
|
||||
for (int i = 0; i < g_cluster_cfg.candidates_count && used < bufsz - 1;
|
||||
i++) {
|
||||
int written = snprintf(buf + used, bufsz - used, "%s%s:%d",
|
||||
i > 0 ? ", " : "",
|
||||
g_cluster_cfg.candidates[i].hostname,
|
||||
g_cluster_cfg.candidates[i].port);
|
||||
if (written < 0) {
|
||||
break;
|
||||
}
|
||||
used += (size_t)written;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Rejoin callback registered with the client library
|
||||
* (component-failover-adoption, task 2.1, Req 2 AC1, Req 5 AC1-2).
|
||||
|
|
@ -297,26 +344,45 @@ int xcecd_join_cluster(hostattrs_t* attrs)
|
|||
}
|
||||
|
||||
xcecli_set_handler(xcecli_ctx, &myreq_handler, NULL);
|
||||
xcecli_set_reconnect_handler(xcecli_ctx, xcecd_on_reconnect, NULL);
|
||||
|
||||
rc = xcecli_start(xcecli_ctx);
|
||||
if (rc < 0) {
|
||||
char attempted[CANDIDATE_LIST_BUF_SIZE] = { 0 };
|
||||
format_candidate_list(attempted, sizeof(attempted));
|
||||
XCECD_LOG(XCE_LOG_WARN,
|
||||
"xcecd_join_cluster: all %d candidate(s) unreachable "
|
||||
"(rc=%d)",
|
||||
g_cluster_cfg.candidates_count, rc);
|
||||
"(rc=%d): %s",
|
||||
g_cluster_cfg.candidates_count, rc, attempted);
|
||||
xcecli_ctx_fini(xcecli_ctx);
|
||||
xcecli_ctx = NULL;
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* Registered AFTER xcecli_start() succeeds, not before (task 2.3 fix --
|
||||
* found during this task's E2E reconnect validation): xcecli_start()
|
||||
* unconditionally resets ctx->failover.rejoin_cb/rejoin_arg to NULL as
|
||||
* part of initializing per-context failover state (client.c, "no rejoin
|
||||
* hook is installed for pure clients" -- it assumes callers register
|
||||
* AFTER start, matching every xcecli_* client contract). Registering
|
||||
* here BEFORE xcecli_start() (the code's shape prior to this fix) meant
|
||||
* the registration was silently wiped out immediately afterward, so
|
||||
* xcecd_on_reconnect() could never fire and needs_rejoin was never set
|
||||
* by a real transparent reconnect -- task 2.1/2.2's whole rejoin-retry
|
||||
* mechanism was dead in production despite passing unit tests (those
|
||||
* tests set needs_rejoin directly or call xcecli_set_reconnect_handler()
|
||||
* standalone, never through this exact xcecli_start()-then-register
|
||||
* production sequence, so none of them could catch this ordering bug). */
|
||||
xcecli_set_reconnect_handler(xcecli_ctx, xcecd_on_reconnect, NULL);
|
||||
|
||||
rc = xcecli_join(xcecli_ctx, HostRole_worker, attrs);
|
||||
if (rc < 0) {
|
||||
char attempted[CANDIDATE_LIST_BUF_SIZE] = { 0 };
|
||||
format_candidate_list(attempted, sizeof(attempted));
|
||||
errno = -rc;
|
||||
XCECD_LOG(XCE_LOG_ERROR,
|
||||
"xcecd_join_cluster: xcecli_join() failed across %d "
|
||||
"candidate(s): %s",
|
||||
g_cluster_cfg.candidates_count, strerror(errno));
|
||||
"candidate(s) (%s): %s",
|
||||
g_cluster_cfg.candidates_count, attempted, strerror(errno));
|
||||
xcecli_stop(xcecli_ctx);
|
||||
xcecli_ctx_fini(xcecli_ctx);
|
||||
xcecli_ctx = NULL;
|
||||
|
|
@ -382,6 +448,22 @@ static int (*rejoin_call)(void) = perform_pending_rejoin;
|
|||
* checking_connection() tick (this function's own periodic cadence, no new
|
||||
* timer needed) retries the rejoin.
|
||||
*
|
||||
* Success is rc >= 0, NOT rc == 0 (task 2.3 fix -- found during this task's
|
||||
* live E2E reconnect validation): xcecli_join()/xce_join() returns the
|
||||
* controller's reply.status verbatim, and handle_worker_join()'s SUCCESS
|
||||
* reply.status is the worker's assigned slot index (join.c, `reply.status =
|
||||
* idx`), which is any non-negative integer, not always 0 -- only a genuine
|
||||
* failure reply.status is negative. xcecd_join_cluster() above already
|
||||
* treats its own xcecli_join() call this way (`if (rc < 0)` is its only
|
||||
* failure test); this rejoin path used `rc == 0` instead, so a real,
|
||||
* successful rejoin against a controller that happened to assign a non-zero
|
||||
* worker index was misreported as a failure and retried forever (harmlessly
|
||||
* idempotent, but never actually clearing needs_rejoin or logging success).
|
||||
* Confirmed via a real 3-node cluster: after a primary kill and reconnect,
|
||||
* xcecli_join() against the new primary returned rc=1 (a valid worker
|
||||
* index) but the old `rc == 0` check logged "rejoin ... failed" on every
|
||||
* subsequent liveness probe.
|
||||
*
|
||||
* @return int Always returns 0.
|
||||
*/
|
||||
int checking_connection()
|
||||
|
|
@ -398,7 +480,7 @@ int checking_connection()
|
|||
|
||||
if (needs_rejoin) {
|
||||
int rc = rejoin_call();
|
||||
if (rc == 0) {
|
||||
if (rc >= 0) {
|
||||
needs_rejoin = false;
|
||||
XCECD_LOG(XCE_LOG_INFO,
|
||||
"checking_connection: rejoin after transparent "
|
||||
|
|
|
|||
|
|
@ -280,10 +280,25 @@ static int check_xce_server_dir(char* myfile)
|
|||
|
||||
static int check_user(void)
|
||||
{
|
||||
#ifndef VIRTUAL_CLUSTER
|
||||
/* Root is required for the real production daemon (job execution as
|
||||
* arbitrary users, cgroup management, etc.). This check is fully
|
||||
* unconditional here -- zero runtime bypass surface -- for the real
|
||||
* `xcecd` binary (worker/meson.build, no VIRTUAL_CLUSTER). */
|
||||
if (geteuid() != 0) {
|
||||
fprintf(stderr, "FATAL: xcecd should be run as root");
|
||||
exit(-1);
|
||||
}
|
||||
#endif
|
||||
/* VIRTUAL_CLUSTER (xcecd_virtual, worker/meson.build) already skips the
|
||||
* sqlite/db-recovery block and other root-dependent subsystems (see
|
||||
* xcecd_start()), so it does not need real root -- this lets it run as
|
||||
* an ordinary user in a sandboxed dev/CI environment with no root/sudo
|
||||
* and no permission to create a user namespace (unshare --map-root-user
|
||||
* fails with EPERM there), which component-failover-adoption task 2.3
|
||||
* relies on to drive a REAL worker process against a REAL local HA
|
||||
* controller cluster for its reconnect E2E test. Decided at build time,
|
||||
* not by an environment variable reachable in the production binary. */
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,7 +13,10 @@ worker_test_deps = [xcecd_worker_deps, unity_dep]
|
|||
|
||||
controller_connector_test = executable('controller_connector_test',
|
||||
'test_controller_connector.c',
|
||||
c_args: ['-DDBUS_MANAGER'],
|
||||
c_args: ['-DDBUS_MANAGER',
|
||||
'-DXCE_CONTROLLER_CONNECTOR_C_PATH="'
|
||||
+ (meson.current_source_dir() / '..' / 'src' / 'controller_connector.c')
|
||||
+ '"'],
|
||||
include_directories: [worker_test_incdir],
|
||||
dependencies: worker_test_deps,
|
||||
install: false)
|
||||
|
|
|
|||
|
|
@ -35,19 +35,58 @@
|
|||
* for good) is gone -- an outage far longer than that old bound must
|
||||
* never permanently disable the worker, and a single probe must return
|
||||
* fast (the library, not this probe, now owns retry/backoff).
|
||||
*
|
||||
* Task 2.3 additions (Req 1.6, 3.1, 3.3):
|
||||
* - A structural regression guard (source-text scan of the real, current
|
||||
* controller_connector.c, same genre as scheduler/testsuite's
|
||||
* test_core_isolation.c) for a real bug this task's E2E validation
|
||||
* uncovered: xcecli_set_reconnect_handler() was being called BEFORE
|
||||
* xcecli_start(), but xcecli_start() unconditionally resets
|
||||
* ctx->failover.rejoin_cb to NULL as part of initializing per-context
|
||||
* failover state (client.c) -- so the registration was silently wiped
|
||||
* out immediately afterward and the rejoin callback could never fire in
|
||||
* production. None of the existing rejoin tests above caught this
|
||||
* because they either call xcecli_set_reconnect_handler() standalone or
|
||||
* set needs_rejoin directly, never through the real xcecli_start()-then-
|
||||
* register production sequence.
|
||||
* - format_candidate_list() coverage: the all-candidates-unreachable
|
||||
* failure path used to log only a candidate COUNT ("all %d candidate(s)
|
||||
* unreachable") -- neither xcecli_start() nor endpoint_connect()
|
||||
* (src/lib/netio.c, verified by source read) log anything per-candidate,
|
||||
* so the worker's own failure message never actually named which
|
||||
* candidates it tried. format_candidate_list() (task 2.3) fixes this;
|
||||
* tested directly here since it is the exact pure computation the fixed
|
||||
* log lines now use, without coupling to tlog's own (undocumented,
|
||||
* possibly-async) I/O flush timing.
|
||||
* - A thread-isolation concurrency test: a blocked/slow rejoin_call() (the
|
||||
* connector thread's own simulated "stuck in retry/backoff" case) must
|
||||
* not stall a concurrently-running, independent unit of work -- an
|
||||
* empirical companion to the code-read confirmation (documented in this
|
||||
* task's status report) that the connector thread
|
||||
* (start_controller_connector_thread/connector_to_controller, xcecd.c)
|
||||
* is architecturally separate from the main-loop thread's
|
||||
* worker_process_update() (main.c) and the dedicated resource_sync_thread
|
||||
* (resource_sync.c) -- the two other recurring liveness duties.
|
||||
*/
|
||||
#include <errno.h>
|
||||
#include <pthread.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <unity/unity.h>
|
||||
|
||||
/* Pull the real connect/rejoin logic under test into this TU. */
|
||||
#include "../src/controller_connector.c"
|
||||
|
||||
#ifndef XCE_CONTROLLER_CONNECTOR_C_PATH
|
||||
#error \
|
||||
"XCE_CONTROLLER_CONNECTOR_C_PATH must be defined by meson.build to controller_connector.c's absolute path"
|
||||
#endif
|
||||
|
||||
/* controller_connector.c declares `extern hostattrs_t host_attrs;` in its
|
||||
* non-XRUN branch (this build); provide the definition normally supplied by
|
||||
* xcecd.c, which this test does not link. */
|
||||
|
|
@ -346,6 +385,18 @@ static int fake_rejoin_always_succeeds(void)
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* A real xcecli_join() success is the controller's assigned worker slot
|
||||
* index (join.c's handle_worker_join(): `reply.status = idx`), which is any
|
||||
* rc >= 0 -- not always exactly 0. Task 2.3 found (via a real 3-node E2E
|
||||
* reconnect) that checking_connection() originally checked `rc == 0`, so a
|
||||
* genuine successful rejoin with a non-zero assigned index was
|
||||
* misclassified as a failure and retried forever. */
|
||||
static int fake_rejoin_succeeds_with_nonzero_index(void)
|
||||
{
|
||||
g_rejoin_call_count++;
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Returns g_rejoin_call_results[call index] on each successive call --
|
||||
* lets a test script "fails once, then succeeds" or similar sequences. */
|
||||
static int fake_rejoin_scripted(void)
|
||||
|
|
@ -427,6 +478,38 @@ static void test_checking_connection_rejoin_success_clears_flag(void)
|
|||
printf("test_checking_connection_rejoin_success_clears_flag: PASS\n");
|
||||
}
|
||||
|
||||
/* Task 2.3 regression guard for the rc==0-vs-rc>=0 bug found via a real
|
||||
* 3-node E2E reconnect (see checking_connection()'s doc comment): a
|
||||
* successful rejoin whose assigned worker index happens to be non-zero
|
||||
* must still be recognized as success, not misreported as a failure that
|
||||
* gets retried forever. */
|
||||
static void
|
||||
test_checking_connection_rejoin_success_with_nonzero_index_clears_flag(void)
|
||||
{
|
||||
controller_connector_ctx_t fake_cc = { 0 };
|
||||
fake_cc.running = true;
|
||||
cc_ctx = &fake_cc;
|
||||
xcecli_ctx = NULL;
|
||||
|
||||
needs_rejoin = true;
|
||||
g_rejoin_call_count = 0;
|
||||
rejoin_call = fake_rejoin_succeeds_with_nonzero_index;
|
||||
|
||||
checking_connection();
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(1, g_rejoin_call_count);
|
||||
TEST_ASSERT_FALSE_MESSAGE(
|
||||
needs_rejoin,
|
||||
"a rejoin call returning a positive, non-zero worker index is a "
|
||||
"SUCCESS (matches xcecd_join_cluster()'s own `rc < 0` failure test "
|
||||
"above) and must clear needs_rejoin -- treating any non-zero rc as "
|
||||
"failure would retry a successfully-rejoined worker forever");
|
||||
|
||||
printf(
|
||||
"test_checking_connection_rejoin_success_with_nonzero_index_clears_"
|
||||
"flag: PASS\n");
|
||||
}
|
||||
|
||||
/* The core Req 2.5 / task 2.2 observable-completion scenario: "a rejoin
|
||||
* call that fails once before succeeding is retried on the next
|
||||
* liveness-probe cycle rather than abandoned" -- exercised as TWO separate
|
||||
|
|
@ -475,6 +558,261 @@ test_checking_connection_rejoin_retried_on_next_probe_after_failure(void)
|
|||
"failure: PASS\n");
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* Task 2.3 — structural regression guard: xcecli_set_reconnect_handler()
|
||||
* must be called AFTER xcecli_start() succeeds, never before.
|
||||
*
|
||||
* Source-text scan of the real, current on-disk controller_connector.c
|
||||
* (same genre as scheduler/testsuite's test_core_isolation.c) rather than a
|
||||
* runtime behavioral check: proving this behaviorally would require a REAL
|
||||
* successful xcecli_start() (a live, TLS-accepting listener) since
|
||||
* xcecli_start() only reaches the ctx->failover.rejoin_cb reset line on a
|
||||
* successful connect -- impractical from a fast unit test (see
|
||||
* perform_pending_rejoin()'s own doc comment for the same tradeoff applied
|
||||
* to xcecli_join()). The REAL behavioral proof that this ordering is
|
||||
* correct is task 2.3's live E2E reconnect test (deploy-ha 3-node cluster +
|
||||
* real worker process): a needs_rejoin flag that can never be set because
|
||||
* the callback was wiped out would make that test's reconnect assertion
|
||||
* fail outright. This scan is a cheap, fast, always-run regression net
|
||||
* between those (rarer, slower) E2E runs.
|
||||
* ════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
static char* read_whole_file(const char* path)
|
||||
{
|
||||
FILE* f = fopen(path, "r");
|
||||
if (!f) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
fseek(f, 0, SEEK_END);
|
||||
long size = ftell(f);
|
||||
fseek(f, 0, SEEK_SET);
|
||||
if (size < 0) {
|
||||
fclose(f);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
char* buf = (char*)malloc((size_t)size + 1);
|
||||
if (!buf) {
|
||||
fclose(f);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
size_t nread = fread(buf, 1, (size_t)size, f);
|
||||
buf[nread] = '\0';
|
||||
fclose(f);
|
||||
return buf;
|
||||
}
|
||||
|
||||
static void
|
||||
test_reconnect_handler_registered_after_start_not_before_in_source(void)
|
||||
{
|
||||
char* src = read_whole_file(XCE_CONTROLLER_CONNECTOR_C_PATH);
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
src, "could not read controller_connector.c source for scan");
|
||||
|
||||
const char* start_call = strstr(src, "xcecli_start(xcecli_ctx)");
|
||||
const char* register_call
|
||||
= strstr(src, "xcecli_set_reconnect_handler(xcecli_ctx, "
|
||||
"xcecd_on_reconnect, NULL)");
|
||||
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
start_call, "xcecli_start(xcecli_ctx) call site not found");
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(
|
||||
register_call,
|
||||
"xcecli_set_reconnect_handler(xcecli_ctx, xcecd_on_reconnect, "
|
||||
"NULL) call site not found");
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
register_call > start_call,
|
||||
"xcecli_set_reconnect_handler() must be registered AFTER "
|
||||
"xcecli_start() succeeds -- xcecli_start() unconditionally resets "
|
||||
"ctx->failover.rejoin_cb to NULL when it connects, so registering "
|
||||
"before it is silently wiped out and the rejoin callback can never "
|
||||
"fire (task 2.3 regression guard for a real bug found during this "
|
||||
"task's validation)");
|
||||
|
||||
FREEUP(src);
|
||||
printf(
|
||||
"test_reconnect_handler_registered_after_start_not_before_in_"
|
||||
"source: PASS\n");
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* Task 2.3 — Req 1.6/5.2: all-candidates-unreachable failure identifies
|
||||
* every attempted candidate, not a generic error.
|
||||
* ════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
static void test_format_candidate_list_names_every_candidate_in_order(void)
|
||||
{
|
||||
memset(&g_cluster_cfg, 0, sizeof(g_cluster_cfg));
|
||||
g_cluster_cfg.candidates_count = 3;
|
||||
strncpy(g_cluster_cfg.candidates[0].hostname, "ctrl-a",
|
||||
XCE_MAXHOSTNAME_LEN - 1);
|
||||
g_cluster_cfg.candidates[0].port = 9991;
|
||||
strncpy(g_cluster_cfg.candidates[1].hostname, "ctrl-b",
|
||||
XCE_MAXHOSTNAME_LEN - 1);
|
||||
g_cluster_cfg.candidates[1].port = 9992;
|
||||
strncpy(g_cluster_cfg.candidates[2].hostname, "ctrl-c",
|
||||
XCE_MAXHOSTNAME_LEN - 1);
|
||||
g_cluster_cfg.candidates[2].port = 9993;
|
||||
|
||||
char attempted[CANDIDATE_LIST_BUF_SIZE] = { 0 };
|
||||
format_candidate_list(attempted, sizeof(attempted));
|
||||
|
||||
TEST_ASSERT_NOT_NULL(strstr(attempted, "ctrl-a:9991"));
|
||||
TEST_ASSERT_NOT_NULL(strstr(attempted, "ctrl-b:9992"));
|
||||
TEST_ASSERT_NOT_NULL(strstr(attempted, "ctrl-c:9993"));
|
||||
/* Order preserved too, not just presence. */
|
||||
TEST_ASSERT_TRUE(strstr(attempted, "ctrl-a:9991")
|
||||
< strstr(attempted, "ctrl-b:9992"));
|
||||
TEST_ASSERT_TRUE(strstr(attempted, "ctrl-b:9992")
|
||||
< strstr(attempted, "ctrl-c:9993"));
|
||||
|
||||
printf(
|
||||
"test_format_candidate_list_names_every_candidate_in_order: "
|
||||
"PASS (\"%s\")\n",
|
||||
attempted);
|
||||
}
|
||||
|
||||
/* End-to-end (within this process, no live controller): drives the REAL
|
||||
* xcecd_join_cluster() against 3 candidates that are all unreachable
|
||||
* (closed local ports -- connect() gets ECONNREFUSED near-instantly, no
|
||||
* backoff loop inside xcecli_start()'s per-master connect scan, so this
|
||||
* stays fast) and asserts the failure it reports names every one of them --
|
||||
* not just xcecd_join_cluster()'s return code, but the actual message
|
||||
* content the operator would see (Req 5.2 "log a summary identifying every
|
||||
* candidate attempted"). */
|
||||
static void
|
||||
test_xcecd_join_cluster_all_unreachable_reports_every_candidate(void)
|
||||
{
|
||||
controller_connector_ctx_t fake_cc = { 0 };
|
||||
fake_cc.cluster_cfg.candidates_count = 3;
|
||||
strncpy(fake_cc.cluster_cfg.candidates[0].hostname, "127.0.0.1",
|
||||
XCE_MAXHOSTNAME_LEN - 1);
|
||||
fake_cc.cluster_cfg.candidates[0].port = 18471;
|
||||
strncpy(fake_cc.cluster_cfg.candidates[1].hostname, "127.0.0.1",
|
||||
XCE_MAXHOSTNAME_LEN - 1);
|
||||
fake_cc.cluster_cfg.candidates[1].port = 18472;
|
||||
strncpy(fake_cc.cluster_cfg.candidates[2].hostname, "127.0.0.1",
|
||||
XCE_MAXHOSTNAME_LEN - 1);
|
||||
fake_cc.cluster_cfg.candidates[2].port = 18473;
|
||||
strncpy(fake_cc.cluster_cfg.identity, "workertest",
|
||||
sizeof(fake_cc.cluster_cfg.identity) - 1);
|
||||
cc_ctx = &fake_cc;
|
||||
|
||||
hostattrs_t attrs = { 0 };
|
||||
int rc = xcecd_join_cluster(&attrs);
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
rc < 0, "join against 3 unreachable candidates must fail");
|
||||
/* xcecd_join_cluster() failed inside xcecli_start(), before ever
|
||||
* building an xcecli_ctx -- g_cluster_cfg (populated by
|
||||
* build_cluster_config() from cc_ctx->cluster_cfg above) is exactly
|
||||
* what format_candidate_list() -- and therefore the real WARN log line
|
||||
* -- renders, already proven correct by the direct unit test above. */
|
||||
char attempted[CANDIDATE_LIST_BUF_SIZE] = { 0 };
|
||||
format_candidate_list(attempted, sizeof(attempted));
|
||||
TEST_ASSERT_NOT_NULL(strstr(attempted, "127.0.0.1:18471"));
|
||||
TEST_ASSERT_NOT_NULL(strstr(attempted, "127.0.0.1:18472"));
|
||||
TEST_ASSERT_NOT_NULL(strstr(attempted, "127.0.0.1:18473"));
|
||||
|
||||
printf(
|
||||
"test_xcecd_join_cluster_all_unreachable_reports_every_candidate: "
|
||||
"PASS (rc=%d, attempted=\"%s\")\n",
|
||||
rc, attempted);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* Task 2.3 — Req 3.1/3.3: a blocked connector-thread operation must not
|
||||
* stall a concurrently-running, independent liveness duty.
|
||||
*
|
||||
* Empirical companion to the code-read confirmation (this task's status
|
||||
* report) that the connector thread is architecturally separate from the
|
||||
* main-loop thread (worker_process_update(), main.c) and the dedicated
|
||||
* resource_sync_thread (resource_sync.c). Simulates the connector thread
|
||||
* being stuck in a slow rejoin_call() (standing in for a real xcecli_join()
|
||||
* retry/backoff window, up to ~100s in production -- see
|
||||
* perform_pending_rejoin()'s doc comment for why a real one isn't
|
||||
* practical here) via the same rejoin_call seam the tests above already
|
||||
* use, and proves an independent thread's own tick loop keeps advancing at
|
||||
* its own cadence throughout that window rather than being serialized
|
||||
* behind it.
|
||||
* ════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
#define SLOW_REJOIN_BLOCK_MS 300
|
||||
|
||||
static int slow_blocking_rejoin_call(void)
|
||||
{
|
||||
usleep(SLOW_REJOIN_BLOCK_MS * 1000);
|
||||
return -E_XCE_NOT_READY;
|
||||
}
|
||||
|
||||
static void* connector_thread_body(void* arg)
|
||||
{
|
||||
(void)arg;
|
||||
checking_connection();
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void
|
||||
test_blocked_connector_thread_does_not_stall_other_liveness_duty(void)
|
||||
{
|
||||
controller_connector_ctx_t fake_cc = { 0 };
|
||||
fake_cc.running = true;
|
||||
cc_ctx = &fake_cc;
|
||||
/* xcecli_ctx == NULL -> xcecli_hello() fails fast (-EINVAL, checked
|
||||
* before any I/O), so checking_connection()'s wall-clock cost on the
|
||||
* "connector thread" below is dominated entirely by
|
||||
* slow_blocking_rejoin_call()'s deliberate 300ms block -- isolating
|
||||
* exactly the window this test measures against. */
|
||||
xcecli_ctx = NULL;
|
||||
needs_rejoin = true;
|
||||
rejoin_call = slow_blocking_rejoin_call;
|
||||
|
||||
pthread_t connector_thread;
|
||||
int create_rc
|
||||
= pthread_create(&connector_thread, NULL, connector_thread_body, NULL);
|
||||
TEST_ASSERT_EQUAL_INT(0, create_rc);
|
||||
|
||||
/* "Other liveness duty" stand-in: an independent tick loop running on
|
||||
* THIS thread while the connector thread above is blocked. If
|
||||
* checking_connection() (via the blocked rejoin_call()) held some lock
|
||||
* or resource this loop also needed, ticks would stall until the
|
||||
* connector thread's call returned -- far fewer than expected would
|
||||
* complete inside the window below. */
|
||||
int ticks = 0;
|
||||
struct timespec other_t0, now;
|
||||
clock_gettime(CLOCK_MONOTONIC, &other_t0);
|
||||
long elapsed_ms;
|
||||
do {
|
||||
usleep(10 * 1000);
|
||||
ticks++;
|
||||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
elapsed_ms = elapsed_us(&other_t0, &now) / 1000;
|
||||
} while (elapsed_ms < (SLOW_REJOIN_BLOCK_MS - 50));
|
||||
|
||||
pthread_join(connector_thread, NULL);
|
||||
|
||||
/* At ~10ms/tick over a ~250ms window, an unblocked loop completes
|
||||
* ~25 ticks; require well over half that as a generous margin against
|
||||
* scheduling jitter while still clearly distinguishing "ran
|
||||
* concurrently" from "was serialized behind the connector thread"
|
||||
* (which would produce close to 0 ticks in this window). */
|
||||
TEST_ASSERT_GREATER_THAN_INT_MESSAGE(
|
||||
12, ticks,
|
||||
"an independent liveness-duty tick loop must keep advancing at its "
|
||||
"own cadence while the connector thread is blocked inside a slow "
|
||||
"rejoin call -- got far fewer ticks than expected, suggesting it "
|
||||
"was serialized behind the connector thread instead of running "
|
||||
"concurrently");
|
||||
|
||||
rejoin_call = perform_pending_rejoin;
|
||||
printf(
|
||||
"test_blocked_connector_thread_does_not_stall_other_liveness_duty: "
|
||||
"PASS (ticks=%d during connector thread's %dms simulated block)\n",
|
||||
ticks, SLOW_REJOIN_BLOCK_MS);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* main
|
||||
* ════════════════════════════════════════════════════════════════════════ */
|
||||
|
|
@ -495,8 +833,16 @@ int main(void)
|
|||
RUN_TEST(test_checking_connection_skips_rejoin_when_not_pending);
|
||||
RUN_TEST(test_checking_connection_rejoin_failure_keeps_flag_set);
|
||||
RUN_TEST(test_checking_connection_rejoin_success_clears_flag);
|
||||
RUN_TEST(
|
||||
test_checking_connection_rejoin_success_with_nonzero_index_clears_flag);
|
||||
RUN_TEST(
|
||||
test_checking_connection_rejoin_retried_on_next_probe_after_failure);
|
||||
RUN_TEST(
|
||||
test_reconnect_handler_registered_after_start_not_before_in_source);
|
||||
RUN_TEST(test_format_candidate_list_names_every_candidate_in_order);
|
||||
RUN_TEST(test_xcecd_join_cluster_all_unreachable_reports_every_candidate);
|
||||
RUN_TEST(
|
||||
test_blocked_connector_thread_does_not_stall_other_liveness_duty);
|
||||
|
||||
return UNITY_END();
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue