lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(libxce): bug-1425 -- populate jobsubmit's *jobid on idempotent-replay success

Browse source 
xcecli_jobsubmit() could return rc>=0 (success) while leaving its *jobid
output parameter at 0, reproducible under concurrent client load
immediately after a fresh single-node election (~1-3/9 clients per round
in the original repro).

Root cause: XCECLI_FAILOVER_CALL's idempotent-replay rule
(client_failover.c's xce_failover_classify) normalizes a retried resend
that hits the server's EEXIST/inflight-collision rejection into overall
success -- but that rejection reply carries no payload
(handle_jobsubmit's EEXIST branches reply with RequestReply_data_nodata,
job.c), so xce_jobsubmit() never populates *jobid on the winning
attempt, nor did it on the earlier failing attempt (raw status was
already negative).

Adds jobsubmit_finalize_jobid() (client_internal.h/client.c), called
from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets
*jobid = submit->jobid (the client-generated id already on the wire --
the controller commits/echoes this exact value on every genuine success
reply, bug-786/job.c:1439) whenever the overall call succeeded,
otherwise leaves it untouched. Placed at the jobsubmit-specific wrapper
layer, not the generic classify/decide layer, which stays fully generic.

3 new socket-free unit tests in test_libxce_retry.c (following this
codebase's direct-state-simulation style for I/O-adjacent logic).
Live-validated: full ASAN rebuild clean, libxce test suite 1/1 OK with
all 3 new tests passing, and a 5-round/45-client concurrent-submit
repro against a live single-node cluster showed 0 "no clean ack"
failures (vs. 1-3/9 per round pre-fix).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 10:47:09 +00:00
parent b33f8b68a6
commit 8a6c339a55
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
3 changed files with 95 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

44
src/lib/testsuite/test_libxce_retry.c
View file

@ -1027,6 +1027,47 @@ static void test_decide_dup_on_resent_write_is_success(void)
printf("test_decide_dup_on_resent_write_is_success: PASS\n");
}
/* bug-1425: the test above proves xcecli_retry_decide() can return rc == 0
* (success) on a DUP-replay WITHOUT any side channel for the underlying
* op's output data (e.g. a job's id) — the classify/decide layer is
* generic and knows nothing about jobid. jobsubmit_finalize_jobid() is the
* jobsubmit-specific repair xcecli_jobsubmit() applies right after
* XCECLI_FAILOVER_CALL returns, so that gap never reaches the caller. */
static void test_jobsubmit_finalize_populates_jobid_on_success(void)
{
/* Simulates the reported bug exactly: no attempt in the retry sequence
* ever wrote *jobid (it is still the caller's initial 0), yet the
* overall call succeeded (rc == 0, e.g. via the DUP-replay rule). */
uint64_t jobid = 0;
jobsubmit_finalize_jobid(&jobid, /*submitted_jobid=*/424242ULL,
/*overall_rc=*/0);
TEST_ASSERT_EQUAL_UINT64(424242ULL, jobid);
printf("test_jobsubmit_finalize_populates_jobid_on_success: PASS\n");
}
/* A genuine failure must never be masked by inventing a jobid — the caller's
* *jobid is left exactly as xce_jobsubmit() last set it (untouched on a
* negative overall rc, by xce_jobsubmit's own `if (status >= 0)` guard). */
static void test_jobsubmit_finalize_leaves_jobid_untouched_on_failure(void)
{
uint64_t jobid = 0;
jobsubmit_finalize_jobid(&jobid, 424242ULL, -ETIMEDOUT);
TEST_ASSERT_EQUAL_UINT64(0ULL, jobid);
printf("test_jobsubmit_finalize_leaves_jobid_untouched_on_failure: PASS\n");
}
/* xcecli_jobsubmit() is called with a NULL *jobid by no real caller today,
* but the finalize step must not assume otherwise -- it runs unconditionally
* after every call. */
static void test_jobsubmit_finalize_null_jobid_safe(void)
{
jobsubmit_finalize_jobid(NULL, 424242ULL, 0); /* must not crash */
printf("test_jobsubmit_finalize_null_jobid_safe: PASS\n");
}
/* Req 3.4 (D6): a transport-only cause that outlasts the budget keeps the LAST
* transport rc (not -E_XCE_NOT_READY — that mapping is reserved for no-primary
* exhaustion). Proves the two exhaustion causes are distinguished. */
@ -1737,6 +1778,9 @@ void test_libxce_retry(void)
RUN_TEST(test_decide_backoff_grows_and_is_bounded);
RUN_TEST(test_decide_terminal_surfaces_unchanged);
RUN_TEST(test_decide_dup_on_resent_write_is_success);
RUN_TEST(test_jobsubmit_finalize_populates_jobid_on_success);
RUN_TEST(test_jobsubmit_finalize_leaves_jobid_untouched_on_failure);
RUN_TEST(test_jobsubmit_finalize_null_jobid_safe);
RUN_TEST(test_decide_transport_exhaustion_keeps_last_rc);
RUN_TEST(test_decide_known_dead_ctx_retries_not_terminal);
RUN_TEST(test_decide_jitter_staggers_many_clients);