docs(buglog): bug-1736 exposed to apiserver/xcecli; log bug-1921

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/bugs/bug-1921.md

@@ -0,0 +1,63 @@
+---
+id: "bug-1921"
+status: "open"
+status_updated: "2026-10-05"
+status_note: "newly logged"
+component: "harness"
+severity: "wrong-result"
+timestamp: "2026-10-05T05:53:10Z"
+last_seen: "2026-10-05T05:53:10Z"
+occurrences: 1
+files: ["src/harness/test.c", "src/lib/xcectx.c"]
+tags: ["harness", "worker", "reconnect", "rusage"]
+related: ["bug-1917", "bug-1736"]
+---
+
+# bug-1921: xce_test fake worker: startjob handler sync-RPC times out, reconnect drops its worker role
+
+## Description
+**Observed:**
+```
+xce_test (worker session):
+  WARN  xce_ctx_quiesce_ep: xce_ctx_reconnect: old connection still has 2 in-flight reference(s) after 2000 ms; reconnecting anyway
+  INFO  xcecli_retry_next: xcecli failover: attempt 1 -> localhost:9991 (after status=-110)
+controller:
+  server sock 4 accepted 11          <- new connection, no join on it
+  myreq_handler opcode update        <- the fake worker's job rusage report
+  ERROR get_ep_worker: endpoint 0x7d1ff5c06ec0 is not a worker
+```
+In `xce_test`'s worker context, a job dispatched to the fake worker is started by `handle_startjob`, which runs as a VTASK_IO task of the worker's own connection and makes a synchronous RPC (the start notify, answered with notifyAcked) on that same connection. The reply is queued as another task of that connection, behind the handler that waits for it, so the call times out (-110) and the client fails over through `xce_ctx_reconnect`. The fake worker never re-joins on the new connection, so the controller rejects all its later reports (`update`: job rusage) as coming from a non-worker endpoint.
+
+**Expected:** the fake worker's start notify completes on its connection (the handler must not wait synchronously on a reply that can only be dispatched after it returns), and after any reconnect a worker-context session re-joins before sending worker requests.
+**Impact:** harness only: an `xce_test` fake worker loses its worker role ~10-12 s after its first job starts, so every job-usage report it sends afterwards is dropped (CPU time / memory / efficiency stay 0) and the job's later worker notifications are refused. Live run-usage checks need a manual second `join`. Real xcecd workers are not affected by this path (they do not start jobs from a connection vtask handler).
+**Found by:** test (bug-1736 live E2E, 2026-10-05)
+
+## Environment
+- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
+- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (xce.build)
+- Code: scheduler-framework @ e2472aa7-dirty
+- Deployment: single controller + scheduler + apiserver (`run-resource-verify.sh --keep-running`), one `xce_test` fake worker
+
+## Reproduction
+1. `bash .claude/skills/xce-test-resources/run-resource-verify.sh --keep-running --dir <D>`
+2. `printf 'set masters localhost:9991\nset myname fakew\nworker\njoin\n!sleep 200\nquit\n' | xce_test --norc &`
+3. Submit a long job from another session (`client`, `jobsub -u user1 -q normal -c "sleep 600"`); close other hosts with `hostctrl -h <host> -c -y` so it lands on fakew.
+4. /tmp/xce_test.log shows the -110 failover ~12 s after the start; the controller log shows `get_ep_worker ... is not a worker` for each later rusage report. Workaround: a second `join` in the worker session after the reconnect.
+- Frequency: deterministic, 3/3 runs, 2026-10-05
+- Evidence: not kept (session scratchpad logs)
+
+## Root cause
+TODO: required for fixed/partial
+
+## Fix
+TODO: required for fixed/partial
+
+## Verification
+- Test: TODO: name + file; RED before the fix, GREEN after
+- Suites: TODO: suite results and build profile
+- Live: TODO: soak/deploy run, or n/a and why
+
+## References
+- Related: none