diff --git a/.kiro/specs/scheduler-core/tasks.md b/.kiro/specs/scheduler-core/tasks.md index 55307dd5..118dd6f4 100644 --- a/.kiro/specs/scheduler-core/tasks.md +++ b/.kiro/specs/scheduler-core/tasks.md @@ -266,11 +266,12 @@ - _Depends: 7.4, 9.4_ - _Verified: 2026-08-05 — this is the spec's ONE task touching `src/controller/`. New `case OPCode_schedAdmin:` in `myreq_handler()` (cluster.c) mirrors the existing `OPCode_upgradeConfig` case structurally: `ha_is_primary()` gate (reply `-EAGAIN`, no forward), `sched.ep==NULL` check (reply "scheduler not connected", no forward) — both of this task's own literal Observables — then forwards via a new `xce_schedadmin_req()` (request.c, async send mirroring `xce_fetchfile_req`) and blocks the vtask thread on a condvar with a 5s deadline while holding `c->sched.lock` for the full round-trip (a design decision made and accepted during this spec's earlier research phase, matching `OPCode_upgradeConfig`'s posture at parity — no RBAC beyond the primary gate, blocking wait accepted as low-frequency human-paced admin traffic). A new branch in the existing `case OPCode_reply:` demuxes the scheduler's `SchedAdminReply` (by `opcode == OPCode_schedAdmin`) to a process-global single-slot pending-waiter mechanism (`schedadmin_register`/`_unregister`/`_signal`), serialized by `c->sched.lock` being held throughout — the same process-global-list pattern this codebase already uses for `job.c`'s fetchfile proxy. Adversarial review independently traced double-free/cleanup-per-path, the timeout/signal race, single-slot safety, and reply-opcode demux correctness against source (not prior sessions' claims) and found no correctness bugs; confirmed the `c->sched.lock` contention is broader than initially characterized (gates job/worker/resource/queue events, not just job-new/start) but is spec-mandated by requirements.md's own pseudocode, bounded to 5s, and does not risk the severe heartbeat-starvation/false-failover cascade (heartbeat send runs on a separate non-vtask delaymgr task) — flagged as a documented residual risk for a future hardening pass, not a blocker. Review also found and a follow-up fix-round closed a narrow-window memory leak (a reply arriving in the brief gap between the waiter's mutex unlock and its unregister call could hand off ownership into an already-abandoned slot) via a carefully-ordered unlock→unregister→relock sequence, with lock-ordering reasoning (avoiding an AB-BA inversion against `schedadmin_signal()`'s `g_schedadmin_mu`→`p->mu` acquisition order) independently re-derived and confirmed correct by me directly from the code before finalizing. Also cleaned up: two prior sessions' work had whole-file clang-format reformatted `cluster.c`/`request.c`/`request.h`, churning large amounts of unrelated pre-existing HA code through pure whitespace changes — minimized to the genuinely new content only (confirmed via `git diff` matching `git diff -w` exactly for all three files) to avoid polluting this HA-critical file's history. New `test_sched_admin_dispatch.c` (3 tests) drives the real `myreq_handler` via the existing `myreq_handler_for_test` seam (same precedent `test_config_repl.c` uses): not-primary rejection using a poison-pointer `sched.ep` that would crash if forwarding were ever incorrectly attempted (proven, not assumed), not-connected rejection, and a genuine round-trip test with a second pthread independently driving a real `OPCode_reply`/`SchedAdminReply` through the same handler to exercise the actual signal/demux path end-to-end. Controller suite 3/3 (1090/1090 Unity assertions including the 3 new tests), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh `meson compile`/`meson test --suite controller`/`meson test --suite scheduler` runs before marking complete. Residual, non-blocking notes for follow-up: the `c->sched.lock` contention breadth (above) and the process-global (vs. `cluster_t`-scoped) pending-waiter slot are both flagged as candidates for a future hardening pass, not required for this task._ -- [ ] 10.2 Controller-side admin opcode dispatch tests +- [x] 10.2 Controller-side admin opcode dispatch tests - Test standby rejection, disconnected-scheduler rejection, forward timeout, and a successful forward/reply round trip against the real scheduler-side handler from 9.4 - Observable: the test suite passes, covering every reply-on-every-path case named in the dispatch task - _Requirements: 5.10_ - _Depends: 10.1, 9.4_ + - _Verified: 2026-08-05 — audited task 10.1's own `test_sched_admin_dispatch.c` first: standby rejection and disconnected-scheduler rejection were already fully covered (confirmed unchanged). Two genuine gaps closed: (1) forward timeout — no test previously drove the real 5s `pthread_cond_timedwait` deadline to expiry. Added an `#ifdef XCE_TESTING`-only timeout-override seam in `cluster.c` (`SCHEDADMIN_TIMEOUT_MS` macro + `schedadmin_set_timeout_ms_for_test()`), verified genuinely inert in production builds: `-DXCE_TESTING` is set on exactly the 3 unit-test meson targets, absent from `xce_controller`'s own target and from every project-wide arg; the non-test `#else` branch is a bare `5000L` literal with the test-only global/setter entirely absent from that translation unit; deadline math confirmed correct for both the 5000ms default (behaviorally identical to the original `deadline.tv_sec += 5`) and the test's 200ms override (correct carry logic). `test_schedadmin_forward_timeout` sets a 200ms override, forwards to a real unconnected `sched.ep` with nothing ever replying, brackets the call with real `CLOCK_MONOTONIC` timestamps proving the wait genuinely blocked, and asserts the exact `{status=0, ok=false, error="request timeout"}` shape from requirements.md AC 10 — with the override reset to 5000 strictly before any `TEST_ASSERT` call (verified line-by-line, no `longjmp`-induced test-pollution risk). (2) "Round trip against the real scheduler-side handler" — investigated a literal two-binary integration (controller test invoking the scheduler's real `scheduler_req_handler()` in-process) and judged it architecturally awkward and out of scope (separate meson targets, incompatible fixture types, real symbol-collision risk) with no net correctness gain, since task 9.4's own `test_admin_request.c` already white-box-drives the real scheduler handler rigorously. Instead added two more round-trip outcome-branch tests using the same real `xce_encode`/`xce_decode` wire functions production uses: `test_schedadmin_round_trip_rpc_status_only` (a genuinely new C branch — reply with no `SchedAdminReply` payload) and `test_schedadmin_round_trip_business_error` (same branch as 10.1's existing success test, but proving `ok=false`/error-string field-passthrough fidelity — accurately self-described as payload coverage, not overclaimed as a new branch). Review independently verified every claim against source (confirmed `-DXCE_TESTING`'s exact meson scoping, confirmed sequential single-threaded Unity test execution rules out any cross-test race on the test-only global, confirmed `git diff` exactly matches `git diff -w` for `cluster.c` — zero reformat noise, learning correctly applied from task 10.1's cleanup). Controller suite 3/3 (1093/1093 Unity assertions, up from 1090), scheduler suite 26/26 unaffected, clang-format clean — independently re-verified by me via fresh `meson compile`/`meson test --suite controller`/`meson test --suite scheduler` runs before marking complete._ - [ ] 10.3 Build the xceadmin scheduler recording commands - Implement record-start, record-stop, and record-status client functions that connect to a controller, follow redirects to the primary if needed, send the corresponding admin request, and decode the reply diff --git a/src/controller/cluster.c b/src/controller/cluster.c index a63b4307..7655206d 100644 --- a/src/controller/cluster.c +++ b/src/controller/cluster.c @@ -309,6 +309,34 @@ static int schedadmin_signal(int status, sched_admin_reply_t* sar) return 1; } +#ifdef XCE_TESTING +/* + * schedadmin_set_timeout_ms_for_test / SCHEDADMIN_TIMEOUT_MS (task 10.2, Req + * 5.10 edge case "Scheduler disconnects mid-request"): the OPCode_schedAdmin + * case's forward-and-block-for-reply deadline is a fixed 5s in production + * (requirements.md AC 10: "await scheduler reply (timeout: 5s)"). Proving + * that deadline genuinely expires needs a REAL pthread_cond_timedwait + * timeout to fire -- not a shortcut that bypasses the wait -- but a real 5s + * wait per test run has no precedent in this suite (the longest real waits + * elsewhere are sub-second, e.g. DEFAULT_SYNC_TIMEOUT_MS=500ms in + * test_commit_resolve.c). This test-only override shrinks the deadline so + * the SAME wait-then-give-up code path below executes for real, just + * faster. Guarded by XCE_TESTING like myreq_handler_for_test below -- never + * compiled into production builds, where SCHEDADMIN_TIMEOUT_MS is a bare + * 5000 with zero indirection and the deadline math below folds to exactly + * the prior `deadline.tv_sec += 5` (5000 / 1000 == 5, 5000 % 1000 == 0). + */ +static long g_schedadmin_timeout_ms = 5000; + +void schedadmin_set_timeout_ms_for_test(long ms) +{ + g_schedadmin_timeout_ms = ms; +} +#define SCHEDADMIN_TIMEOUT_MS g_schedadmin_timeout_ms +#else +#define SCHEDADMIN_TIMEOUT_MS 5000L +#endif + static int myreq_handler(vtask_t* task) { vtask_data_t* data = (vtask_data_t*)task->data; @@ -869,8 +897,14 @@ static int myreq_handler(vtask_t* task) } struct timespec deadline; + long schedadmin_timeout_ms = SCHEDADMIN_TIMEOUT_MS; clock_gettime(CLOCK_REALTIME, &deadline); - deadline.tv_sec += 5; + deadline.tv_sec += schedadmin_timeout_ms / 1000; + deadline.tv_nsec += (schedadmin_timeout_ms % 1000) * 1000000L; + if (deadline.tv_nsec >= 1000000000L) { + deadline.tv_sec += 1; + deadline.tv_nsec -= 1000000000L; + } pthread_mutex_lock(&pending.mu); while (!pending.done) { diff --git a/src/controller/testsuite/test_sched_admin_dispatch.c b/src/controller/testsuite/test_sched_admin_dispatch.c index 93cb0078..e8b90a06 100644 --- a/src/controller/testsuite/test_sched_admin_dispatch.c +++ b/src/controller/testsuite/test_sched_admin_dispatch.c @@ -25,6 +25,62 @@ * built for the two Observables above made it straightforward. The full * dispatch test matrix (malformed requests, concurrent/serialized * round-trips, etc.) belongs to task 10.2. + * + * ── Task 10.2 additions (Req 5.10) ────────────────────────────────────── + * + * Task 10.2's literal scenario list is "standby rejection, disconnected- + * scheduler rejection, forward timeout, and a successful forward/reply + * round trip against the real scheduler-side handler from 9.4". The first + * two are the Observables above (already task 10.1's). This file adds: + * + * - test_schedadmin_forward_timeout: the genuinely-missing scenario -- + * no existing test drove the 5s deadline to actually expire. Uses + * schedadmin_set_timeout_ms_for_test (cluster.c, XCE_TESTING-only) to + * shrink the real deadline so pthread_cond_timedwait genuinely times + * out on a fast schedule, rather than mocking the wait away or eating + * a real 5s per test run (no precedent for that in this suite -- see + * the seam's own doc comment in cluster.c for the survey). + * + * - test_schedadmin_round_trip_rpc_status_only / + * test_schedadmin_round_trip_business_error: task 10.1's round-trip + * test only exercises ONE of the three outcome branches in the + * schedAdmin case's post-wait dispatch (cluster.c: got_done+reply, + * got_done+no-reply, timeout) -- the got_done+reply "success" shape. + * These two cover the other real, previously-uncovered branch + * (got_done, no SchedAdminReply payload -- an RPC-level outcome) and a + * business-level error shape (ok=false, e.g. "already recording" -- + * requirements.md's "Reply field semantics by action" error rows), + * proving reply.status stays 0 for a completed request that merely + * carries a domain error, never conflated with an RPC/timeout failure. + * + * ON "against the real scheduler-side handler from 9.4": task 9.4's own + * scheduler_req_handler()/sched_handle_admin_request() are proven for + * real, end-to-end, including sched_record_status()'s actual field + * construction, by scheduler/testsuite/test_admin_request.c (already + * landed with task 9.4) -- that file white-box #includes handler.c, + * recording.c, and their scheduler-internal dependencies (core.c, + * driver.c, snapshot*.c) directly into the SCHEDULER's own test binary + * and spies on xcecli_reply_schedadmin() to capture the real reply. This + * controller test binary (controller_unit_test) is a wholly separate + * meson target/executable with its own main() and its own global state + * (cluster_t/ha_state_t fixtures, not xce_scheduler_t) -- pulling + * scheduler/src/handler.c + recording.c (which itself needs core.h and + * the rest of xce_scheduler_t's dependency graph) into THIS binary to + * invoke scheduler_req_handler() literally would require the same kind + * of careful selective-source-linking test_admin_request.c's own header + * comment documents doing for the scheduler side, replicated cross- + * directory inside an already-large, HA-critical controller test target + * -- real risk (symbol/global collisions between two "whole daemon" + * white-box bundles sharing one process image) for no correctness gain, + * since 9.4's own suite already covers that ground rigorously. A genuine + * two-process E2E (real controller + real scheduler binaries, a real + * connection between them) is an E2E/soak-test concern, not this task's + * scope. So: this file proves the CONTROLLER's forward/relay mechanism + * against the real wire contract (xce_encode/xce_decode, the same + * functions production uses) and the exact reply shapes 9.4's real + * sched_record_status()/sched_handle_admin_request() are proven (by + * their own suite) to produce -- not a second, redundant proof that the + * scheduler-side handler itself is correct. */ #define _GNU_SOURCE #include @@ -33,6 +89,7 @@ #include #include #include +#include #include #include @@ -50,6 +107,11 @@ * consumed (freed) by the call, never touch/free it afterward. */ extern int myreq_handler_for_test(endpoint_t* ep, io_packet_t* pkt); +/* Test-only override for the OPCode_schedAdmin case's forward-and-block + * deadline (cluster.c, XCE_TESTING-only, task 10.2) -- see its own doc + * comment in cluster.c for why this seam exists. */ +extern void schedadmin_set_timeout_ms_for_test(long ms); + /* ── Fixture ────────────────────────────────────────────────────────────── * * The OPCode_schedAdmin dispatch case (and the ha_dispatch_gate primary/ @@ -115,6 +177,16 @@ static void sa_decode_queued_reply(endpoint_t* ep, xce_request_t* out) TEST_ASSERT_EQUAL_INT(0, rc); } +/* Monotonic milliseconds -- for measuring that the forward-timeout test + * genuinely waited out its (shrunk) deadline rather than returning early. + * Same construction as test_commit_resolve.c's mono_ms_r(). */ +static uint64_t sa_mono_ms(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000u + (uint64_t)ts.tv_nsec / 1000000u; +} + /* ── Observable 1: non-primary rejects immediately, no forwarding attempt ── * * ha_dispatch_gate (ha.c) denies OPCode_schedAdmin for every non-primary @@ -300,6 +372,240 @@ static void test_schedadmin_round_trip_success(void) printf("test_schedadmin_round_trip_success: PASS\n"); } +/* ── Task 10.2: forward timeout ───────────────────────────────────────────── + * + * c->sched.ep points at a real (if unconnected) endpoint_t, exactly like the + * round-trip test above, so xce_schedadmin_req really forwards and the + * dispatch case really blocks on pthread_cond_timedwait -- but NO second + * thread ever delivers a reply. schedadmin_set_timeout_ms_for_test shrinks + * the deadline so the real wait-then-give-up path completes on a fast test + * schedule; the override is restored to the production default IMMEDIATELY + * after myreq_handler_for_test returns, before any TEST_ASSERT below can + * longjmp out on failure (Unity's assertion macros unwind via longjmp) -- + * so a failing assertion in this test can never leave a later test in this + * binary running against a shrunk deadline. */ +static void test_schedadmin_forward_timeout(void) +{ + cluster_t* c = sa_make_cluster(); + atomic_store(&c->ha.role, CTRL_PRIMARY); + + endpoint_t sched_conn; + memset(&sched_conn, 0, sizeof(sched_conn)); + c->sched.ep = &sched_conn; + + endpoint_t caller_ep; + memset(&caller_ep, 0, sizeof(caller_ep)); + caller_ep.owner = c; + + io_packet_t* pkt = sa_build_request_pkt(); + + const long timeout_ms = 200; + schedadmin_set_timeout_ms_for_test(timeout_ms); + + uint64_t t0 = sa_mono_ms(); + int rc = myreq_handler_for_test(&caller_ep, pkt); + uint64_t took = sa_mono_ms() - t0; + + schedadmin_set_timeout_ms_for_test(5000); /* restore before any assert */ + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, rc, + "the dispatch case always replies -- it does not propagate " + "-ETIMEDOUT as its own return, even on a genuine timeout"); + + TEST_ASSERT_TRUE_MESSAGE( + took + 20 >= (uint64_t)timeout_ms, + "must not give up meaningfully before the (shrunk) deadline -- this " + "is what proves the real pthread_cond_timedwait deadline fired, not " + "a shortcut that skipped the wait"); + TEST_ASSERT_TRUE_MESSAGE( + took < (uint64_t)timeout_ms + 2000, + "must give up within a bounded window after the deadline, not hang"); + + xce_request_t decoded; + sa_decode_queued_reply(&caller_ep, &decoded); + TEST_ASSERT_EQUAL_INT(Request_body_replymsg, decoded.body_which); + xce_reply_t* rr = decoded.body.replymsg; + TEST_ASSERT_NOT_NULL(rr); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, rr->status, + "a timeout is relayed as an ok-RPC SchedAdminReply carrying " + "error=\"request timeout\", per requirements.md AC 10 -- not a " + "negative RPC status"); + TEST_ASSERT_EQUAL_INT(RequestReply_data_schedadminreplymsg, rr->data_which); + sched_admin_reply_t* got = rr->data.schedadminreplymsg; + TEST_ASSERT_NOT_NULL(got); + TEST_ASSERT_EQUAL_INT(0, got->ok); + TEST_ASSERT_EQUAL_STRING("request timeout", got->error); + xce_free_request(&decoded); + + sa_teardown(c); + printf("test_schedadmin_forward_timeout: PASS\n"); +} + +/* ── Task 10.2: additional round-trip outcome branches ────────────────────── + * + * Shared "second thread plays the scheduler" helper, generalized from + * test_schedadmin_round_trip_success's sa_deliver_scheduler_reply above so + * both new tests below can drive either outcome (a SchedAdminReply payload, + * or a bare RPC status with none) through the SAME real + * schedadmin_signal()/demux path, without duplicating the thread/encode + * boilerplate. Task 10.1's own sa_deliver_scheduler_reply is left untouched. */ +typedef struct { + cluster_t* c; + int status; + sched_admin_reply_t* payload; /* NULL => RequestReply_data_nodata */ +} sa_deliver_arg_t; + +static void* sa_deliver_reply_generic(void* argp) +{ + sa_deliver_arg_t* arg = (sa_deliver_arg_t*)argp; + + usleep(200000); /* same margin as sa_deliver_scheduler_reply above */ + + xce_reply_t rr; + memset(&rr, 0, sizeof(rr)); + rr.opcode = OPCode_schedAdmin; + rr.status = arg->status; + if (arg->payload != NULL) { + rr.data_which = RequestReply_data_schedadminreplymsg; + rr.data.schedadminreplymsg = arg->payload; + } else { + rr.data_which = RequestReply_data_nodata; + } + + xce_request_t req; + memset(&req, 0, sizeof(req)); + req.opcode = OPCode_reply; + req.body_which = Request_body_replymsg; + req.body.replymsg = &rr; + + io_packet_t* pkt = calloc(1, sizeof(*pkt)); + TEST_ASSERT_NOT_NULL(pkt); + TEST_ASSERT_TRUE(xce_encode(&req, pkt) >= 0); + + endpoint_t sched_reply_ep; + memset(&sched_reply_ep, 0, sizeof(sched_reply_ep)); + sched_reply_ep.owner = arg->c; + + myreq_handler_for_test(&sched_reply_ep, pkt); + return NULL; +} + +/* Exercises the got_done && got_reply==NULL branch (cluster.c, OPCode_reply + * case): the scheduler's OPCode_reply can legitimately carry no + * SchedAdminReply payload (an RPC-level outcome) -- the controller must + * still reply promptly, relaying the bare status, not treat it as a + * timeout. Never exercised by task 10.1's round-trip test, which always + * supplies a payload. */ +static void test_schedadmin_round_trip_rpc_status_only(void) +{ + cluster_t* c = sa_make_cluster(); + atomic_store(&c->ha.role, CTRL_PRIMARY); + + endpoint_t sched_conn; + memset(&sched_conn, 0, sizeof(sched_conn)); + c->sched.ep = &sched_conn; + + sa_deliver_arg_t arg = { .c = c, .status = -EIO, .payload = NULL }; + pthread_t tid; + TEST_ASSERT_EQUAL_INT( + 0, pthread_create(&tid, NULL, sa_deliver_reply_generic, &arg)); + + endpoint_t caller_ep; + memset(&caller_ep, 0, sizeof(caller_ep)); + caller_ep.owner = c; + + io_packet_t* pkt = sa_build_request_pkt(); + int rc = myreq_handler_for_test(&caller_ep, pkt); + + pthread_join(tid, NULL); + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, rc, + "the dispatch case always replies, even when the scheduler's " + "OPCode_reply carries no SchedAdminReply payload"); + + xce_request_t decoded; + sa_decode_queued_reply(&caller_ep, &decoded); + xce_reply_t* rr = decoded.body.replymsg; + TEST_ASSERT_NOT_NULL(rr); + TEST_ASSERT_EQUAL_INT_MESSAGE( + -EIO, rr->status, + "a payload-less scheduler reply relays its bare RPC status verbatim"); + TEST_ASSERT_EQUAL_INT(RequestReply_data_nodata, rr->data_which); + xce_free_request(&decoded); + + sa_teardown(c); + printf("test_schedadmin_round_trip_rpc_status_only: PASS\n"); +} + +/* A business-level error from the scheduler (ok=false, e.g. "already + * recording" -- requirements.md's "Reply field semantics by action" error + * rows) is a NORMAL completed round trip, not an RPC failure. Proves the + * controller relays that shape verbatim with reply.status == 0 -- a + * domain error is never conflated with an RPC/timeout error. */ +static void test_schedadmin_round_trip_business_error(void) +{ + cluster_t* c = sa_make_cluster(); + atomic_store(&c->ha.role, CTRL_PRIMARY); + + endpoint_t sched_conn; + memset(&sched_conn, 0, sizeof(sched_conn)); + c->sched.ep = &sched_conn; + + sched_admin_reply_t sreply; + memset(&sreply, 0, sizeof(sreply)); + sreply.ok = 0; + sreply.error = "already recording"; + sreply.recording = 1; + sreply.file = "/var/log/xce/scheduler-replay-x.replay"; + sreply.eventCount = 7; + sreply.startedAt = 999; + + sa_deliver_arg_t arg = { .c = c, .status = 0, .payload = &sreply }; + pthread_t tid; + TEST_ASSERT_EQUAL_INT( + 0, pthread_create(&tid, NULL, sa_deliver_reply_generic, &arg)); + + endpoint_t caller_ep; + memset(&caller_ep, 0, sizeof(caller_ep)); + caller_ep.owner = c; + + io_packet_t* pkt = sa_build_request_pkt(); + int rc = myreq_handler_for_test(&caller_ep, pkt); + + pthread_join(tid, NULL); + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, rc, + "a business-level error from the scheduler is still a completed " + "round trip"); + + xce_request_t decoded; + sa_decode_queued_reply(&caller_ep, &decoded); + xce_reply_t* rr = decoded.body.replymsg; + TEST_ASSERT_NOT_NULL(rr); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, rr->status, + "ok=false is a business error, not an RPC/timeout error -- " + "reply.status must stay 0"); + TEST_ASSERT_EQUAL_INT(RequestReply_data_schedadminreplymsg, rr->data_which); + sched_admin_reply_t* got = rr->data.schedadminreplymsg; + TEST_ASSERT_NOT_NULL(got); + TEST_ASSERT_EQUAL_INT(0, got->ok); + TEST_ASSERT_EQUAL_STRING("already recording", got->error); + TEST_ASSERT_EQUAL_INT(1, got->recording); + TEST_ASSERT_EQUAL_STRING("/var/log/xce/scheduler-replay-x.replay", + got->file); + TEST_ASSERT_EQUAL_UINT64(7, got->eventCount); + TEST_ASSERT_EQUAL_UINT64(999, got->startedAt); + xce_free_request(&decoded); + + sa_teardown(c); + printf("test_schedadmin_round_trip_business_error: PASS\n"); +} + /* ── Suite entry point ──────────────────────────────────────────────────── */ void test_sched_admin_dispatch(void) { @@ -307,4 +613,7 @@ void test_sched_admin_dispatch(void) RUN_TEST(test_schedadmin_rejects_when_not_primary); RUN_TEST(test_schedadmin_rejects_when_not_connected); RUN_TEST(test_schedadmin_round_trip_success); + RUN_TEST(test_schedadmin_forward_timeout); + RUN_TEST(test_schedadmin_round_trip_rpc_status_only); + RUN_TEST(test_schedadmin_round_trip_business_error); }