docs(buglog): bug-1736 exposed to apiserver/xcecli; log bug-1921

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-06 13:54:53 +00:00

3.7 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1921: xce_test fake worker: startjob handler sync-RPC times out, reconnect drops its worker role

Description

Observed:

xce_test (worker session):
  WARN  xce_ctx_quiesce_ep: xce_ctx_reconnect: old connection still has 2 in-flight reference(s) after 2000 ms; reconnecting anyway
  INFO  xcecli_retry_next: xcecli failover: attempt 1 -> localhost:9991 (after status=-110)
controller:
  server sock 4 accepted 11          <- new connection, no join on it
  myreq_handler opcode update        <- the fake worker's job rusage report
  ERROR get_ep_worker: endpoint 0x7d1ff5c06ec0 is not a worker

In xce_test's worker context, a job dispatched to the fake worker is started by handle_startjob, which runs as a VTASK_IO task of the worker's own connection and makes a synchronous RPC (the start notify, answered with notifyAcked) on that same connection. The reply is queued as another task of that connection, behind the handler that waits for it, so the call times out (-110) and the client fails over through xce_ctx_reconnect. The fake worker never re-joins on the new connection, so the controller rejects all its later reports (update: job rusage) as coming from a non-worker endpoint.

Expected: the fake worker's start notify completes on its connection (the handler must not wait synchronously on a reply that can only be dispatched after it returns), and after any reconnect a worker-context session re-joins before sending worker requests. Impact: harness only: an xce_test fake worker loses its worker role ~10-12 s after its first job starts, so every job-usage report it sends afterwards is dropped (CPU time / memory / efficiency stay 0) and the job's later worker notifications are refused. Live run-usage checks need a manual second join. Real xcecd workers are not affected by this path (they do not start jobs from a connection vtask handler). Found by: test (bug-1736 live E2E, 2026-10-05)

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (xce.build)
Code: scheduler-framework @ e2472aa7-dirty
Deployment: single controller + scheduler + apiserver (run-resource-verify.sh --keep-running), one xce_test fake worker

Reproduction

bash .claude/skills/xce-test-resources/run-resource-verify.sh --keep-running --dir <D>
printf 'set masters localhost:9991\nset myname fakew\nworker\njoin\n!sleep 200\nquit\n' | xce_test --norc &
Submit a long job from another session (client, jobsub -u user1 -q normal -c "sleep 600"); close other hosts with hostctrl -h <host> -c -y so it lands on fakew.
/tmp/xce_test.log shows the -110 failover ~12 s after the start; the controller log shows get_ep_worker ... is not a worker for each later rusage report. Workaround: a second join in the worker session after the reconnect.

Frequency: deterministic, 3/3 runs, 2026-10-05
Evidence: not kept (session scratchpad logs)

Root cause

TODO: required for fixed/partial

Fix