feat(resource-value-verification): scheduler-side verification check (task 6.3)

poll_scheduler_log()/query_scheduler_resvals() tail the live scheduler
daemon's log (task 4.1's trace line) for each accepted injected resource,
with a bounded ~15s wait batched across all pending resources per poll
iteration (not N times the timeout). Reports MATCH / MISMATCH (both
values) / UNREACHABLE (timeout, distinct from mismatch), tracked in
SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS + scheduler-check-results.tsv.

Live-proven immune to bug-1579 (unlike the controller check): injecting a
host-attrs resource (cpuf) simultaneously produces a controller MISMATCH
and a scheduler MATCH, since this check reads task 4.1's trace log (written
once, at apply-time) rather than re-querying live state after the worker
disconnects.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-18 10:21:50 +00:00
parent cb5bbfcca2
commit 72b6fdea4a
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
2 changed files with 218 additions and 32 deletions

View file

@ -76,7 +76,7 @@
- _Depends: 6.1_
- _Boundary: xce-test-resources skill (controller check) — same file as 6.3/6.4 (`run-resource-verify.sh`), sequential, not parallel-safe_
- [ ] 6.3 Implement the scheduler-side verification check via trace-log tail
- [x] 6.3 Implement the scheduler-side verification check via trace-log tail
- Tail/grep the live scheduler daemon's log file (task 4.1's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no record, no replay, no REPL
- Compare the scheduler's observed value against the injected value, reported distinctly from the controller-side result; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch
- Observable completion: the check reports, per resource, whether the scheduler's observed value matches the injected value within the bounded wait, attributable specifically to the scheduler layer
@ -116,6 +116,8 @@
- _Depends: 7.1_
## Implementation Notes
- Task 6.3 confirmed (live, via a `cpuf` probe: controller MISMATCH + scheduler MATCH simultaneously) that the scheduler check is genuinely immune to bug-1579 — it reads task 4.1's trace log (written once, at apply-time, in the `RESVALS_DELTA` case only), never touched by a later worker-disconnect's `WORKERS_PUSH`-driven reset. **Real fix for bug-1579 identified, not yet applied** (see below) — the `!<shell-command>` escape in `xce_test` (`preprocess_and_run()`, `test.c` ~line 965) works in ANY context including `worker` mode, since it's handled in the single top-level REPL loop before context dispatch, not per-context. This means `inject_resources()` can stay in worker context (connection held open) after injecting, shell out via `!` to a separate `xce_test` client-mode process for the controller query, and only `quit` at the very end — fully closing bug-1579 (not just working around it in the default set) rather than needing a C-level `xce_test` change. Plan: retrofit this into `inject_resources()`/`query_controller_resvals()` as a follow-up before/alongside task 7.1, then restore a host-attrs resource (e.g. `cpuf`) to `DEFAULT_RESOURCE_ARGS` to prove it.
- Minor non-blocking follow-up from task 6.3's review: `poll_scheduler_log()`'s output isn't filtered to the caller's `pending` list (harmless today since the only caller already filters, but worth tightening).
- Task 6.2 (round 1 REJECTED, round 2 fixed) discovered **bug-1579**: any host-attrs-derived resource (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` — `worker.c`'s `N_HOST_ATTRS`) gets reset to baseline by the controller the instant the synthetic worker disconnects, and `inject_resources()` (6.1) always disconnects (`quit`) before each layer's separate check session runs — so a controller-layer check on any host-attrs name will ALWAYS false-MISMATCH under the current two-session structure. Fixed for the default set by swapping `DEFAULT_RESOURCE_ARGS`'s `cpuf` for `r1m` (periodic/dynamic, confirmed safe). **This risk is NOT fully closed** — a user-supplied custom `resource=value` argument using any host-attrs name will still hit this. A real fix requires either a C-level `xce_test` change (its REPL has no path from the `worker` context to the `client` context without a disconnect-equivalent `leave_ctx()`) or a bash coprocess restructuring to keep the worker connected across all layer checks — out of scope for tasks 6.1-6.4, worth a dedicated follow-up task if host-attrs resources ever need default-set coverage.
- Task 6.1 uncovered and got fixed a critical, unrelated, pre-existing controller bug (**bug-1578**, commit `b406dd7`, standalone — a dangling-pointer use-after-free in `sched_worker_change()`/`sched.c` that crashed the controller on worker disconnect once a scheduler had ever joined, 5 months old, unrelated to this feature). It's fixed now with regression coverage (`test_sched_worker_change.c`). If any later task sees a live controller die unexpectedly during a worker-disconnect sequence, confirm you're building from a commit that includes `b406dd7` before assuming it's a new bug. `inject_resources()` also fixed **bug-1577** (in-boundary): `XCE_WORKER_BIN=/bin/true` is exported before invoking `start-all.sh` to suppress the REAL `xcecd_virtual` worker daemon that `deploy-ha.sh`'s pre-existing worker branch would otherwise silently start (Requirement 2.1 violation) whenever the build dir happens to contain one.
- Task 6.1: injection results are stored in `INJECTED_VALUES`/`INJECTED_RESULTS` associative arrays plus `$TOP/logs/injection-results.tsv` (`name\tvalue\tresult`) for tasks 6.2-6.4's extension points, which are appended directly after `inject_resources()` in `main()` (explicit "Extension point" comments mark exactly where). `SKILL.md` is now stale (still says injection "not yet implemented", still illustrates `type` in the default set) — deliberately left unedited since task 6.1's CLI parsing shape didn't change; fix this in task 7.1 when SKILL.md is next touched.