xcecli_jobsubmit() could return rc>=0 (success) while leaving its *jobid
output parameter at 0, reproducible under concurrent client load
immediately after a fresh single-node election (~1-3/9 clients per round
in the original repro).
Root cause: XCECLI_FAILOVER_CALL's idempotent-replay rule
(client_failover.c's xce_failover_classify) normalizes a retried resend
that hits the server's EEXIST/inflight-collision rejection into overall
success -- but that rejection reply carries no payload
(handle_jobsubmit's EEXIST branches reply with RequestReply_data_nodata,
job.c), so xce_jobsubmit() never populates *jobid on the winning
attempt, nor did it on the earlier failing attempt (raw status was
already negative).
Adds jobsubmit_finalize_jobid() (client_internal.h/client.c), called
from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets
*jobid = submit->jobid (the client-generated id already on the wire --
the controller commits/echoes this exact value on every genuine success
reply, bug-786/job.c:1439) whenever the overall call succeeded,
otherwise leaves it untouched. Placed at the jobsubmit-specific wrapper
layer, not the generic classify/decide layer, which stays fully generic.
3 new socket-free unit tests in test_libxce_retry.c (following this
codebase's direct-state-simulation style for I/O-adjacent logic).
Live-validated: full ASAN rebuild clean, libxce test suite 1/1 OK with
all 3 new tests passing, and a 5-round/45-client concurrent-submit
repro against a live single-node cluster showed 0 "no clean ack"
failures (vs. 1-3/9 per round pre-fix).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xcecli_joblist/reslist/workerlist/resval/jobinfo return a non-negative ROW
COUNT on success (rc = getreply.nrows), not a 0/status code. They were
wrapped in XCECLI_FAILOVER_CALL, which unconditionally classifies the raw
return value against fixed constants (E_XCE_* and POSIX errno magnitudes
accepted in either sign) — a row count that happens to equal one of those
constants (e.g. exactly 111 jobs == ECONNREFUSED) was silently
misclassified as a transport failure, forcing a spurious
xce_failover_rotate() on every call with that exact count. In
/xce-test-ha's arithmetic progression (3 + 3*round) this hit at round 36,
burning the retry budget on self-inflicted churn until it coincided with
a genuine transient and surfaced a real ECONNRESET to the caller.
Add XCECLI_FAILOVER_CALL_COUNT: only enters the retry/classify loop when
EXPR's raw result is negative; any non-negative result (0 or a genuine
row count of any magnitude) is unconditionally success. Switch the five
xce_get-family wrappers to it. Verified via 40-round sequential and
40-round chaotic HA soak past round 36, zero committed-job loss.
