62 lines
4.6 KiB
Markdown
62 lines
4.6 KiB
Markdown
---
|
|
id: "bug-1922"
|
|
status: "fixed"
|
|
status_updated: "2026-10-08"
|
|
status_note: ""
|
|
component: "harness"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-08T02:32:58Z"
|
|
last_seen: "2026-10-08T02:32:58Z"
|
|
occurrences: 1
|
|
files: ["src/harness/test.c"]
|
|
tags: ["harness", "worker", "resync"]
|
|
related: ["bug-1921"]
|
|
fix_commits: ["c06798b2"]
|
|
gitea_issue: 346
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "862b3ca77f49"
|
|
---
|
|
|
|
# bug-1922: xce_test fake worker ignores jobStateQuery; each join waits out the 15 s reconciliation timeout
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
controller, after an xce_test fake worker joins:
|
|
WARN resync_on_timeout: resync: worker fakew5 reconciliation round 1 timed out after 15000ms -- resolving so dispatch is not withheld indefinitely (Req 5.3 ...
|
|
```
|
|
On every worker join the controller sends `OPCode_jobStateQuery` (worker-rejoin-resync) and withholds dispatch to that worker until the `OPCode_jobStateReport` answer arrives. `xce_test`'s worker context has no case for `OPCode_jobStateQuery` in its request switch, so it never answers and the controller resolves the round only by its 15 s timeout.
|
|
|
|
**Expected:** the fake worker answers the query like xcecd does (xcecd_resync.c): every queried job ID once, in order -- running if it is running one of its jobs, otherwise unrecognized -- plus an entry for each running job that was not queried.
|
|
**Impact:** harness only: every fake-worker join (and re-join, bug-1921) delays dispatch to that worker by 15 s, and the reconciliation never sees the jobs the fake worker is really running, so the controller's resync logic is untested with fake workers.
|
|
**Found by:** test (bug-1921 live verification, 2026-10-06)
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 6a42631b-dirty
|
|
- Deployment: single controller + scheduler + apiserver (`run-resource-verify.sh --keep-running`), one `xce_test` fake worker
|
|
|
|
## Reproduction
|
|
1. `bash .claude/skills/xce-test-resources/run-resource-verify.sh --keep-running --dir <D>`
|
|
2. `printf 'set masters localhost:9991\nset myname fakew\nworker\njoin\n!sleep 30\nquit\n' | xce_test --norc`
|
|
3. `<D>/logs/ctrl-1.log` shows "reconciliation round 1 timed out after 15000ms" for fakew ~15 s after the join.
|
|
- Frequency: deterministic, every fake-worker join, 2026-10-06
|
|
- Evidence: not kept (session scratchpad logs)
|
|
|
|
## Root cause
|
|
`xce_test`'s worker-context request switch (`handle_startjob`/`handle_notify`/`handle_jobcontrol`/`handle_runtasks` in src/harness/test.c) had no case for `OPCode_jobStateQuery`, the worker-rejoin-resync query the controller sends on every worker join. The query was silently ignored, so the controller could only resolve its reconciliation round by the 15 s timeout (Req 5.3), withholding dispatch to the fake worker until then.
|
|
|
|
## Fix
|
|
- New `src/harness/fake_resync.{c,h}`: `fr_add_page` reassembles a paginated query (an empty query is one complete page; a page that overflows the announced total resets the accumulator); `fr_build_report` builds the report entries like xcecd's `xcecd_resync_build_report` -- every queried ID once, in order, `running` when the fake worker runs it, otherwise `unrecognized`, then each running job that was not queried (`queried = 0`). The fake worker keeps no finished-job retention, so a finished job is reported unrecognized.
|
|
- `test.c`: `OPCode_jobStateQuery` -> `handle_jobstatequery`, which answers with `xce_job_state_report_req` (a non-blocking request with no reply to wait for, so it is safe in the handler context, unlike bug-1921's notifies).
|
|
|
|
## Verification
|
|
- Test: `test_fake_resync` (src/harness/testsuite/test_fake_resync.c, harness_unit_test): single-page and multi-page reassembly, empty query, overflowing page rejected, report = queried in order (running/unrecognized) + unqueried running extras. RED before (module absent: link failure), GREEN after, LSan clean.
|
|
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-07).
|
|
- Live: single controller + scheduler + apiserver (`run-resource-verify.sh --keep-running`), 2026-10-08: an `xce_test` fake worker joined, a job was dispatched to it, and the worker joined again (a round is armed only for a host with non-terminal jobs): the worker printed `job state query answered: 1 job(s)` and the controller logged no reconciliation timeout for it (before the fix: 'reconciliation round 1 timed out after 15000ms').
|
|
|
|
## References
|
|
- Related: none
|