feat(resource-value-verification): worker join and resource injection (task 6.1)
inject_resources() joins a synthetic worker through xce_test's worker
context and runs resupdate for the requested resource=value pairs
(default: Requirement 1's numeric gap set + mem/cpuf regression names,
deliberately excluding type/model due to bug-1574's live segfault risk).
Per-name accept/reject results captured into INJECTED_VALUES/
INJECTED_RESULTS plus injection-results.tsv for tasks 6.2-6.4.
Also suppresses the real xcecd_virtual worker daemon that deploy-ha.sh's
pre-existing worker branch would otherwise silently start whenever the
build dir contains one (XCE_WORKER_BIN=/bin/true), which would have
violated Requirement 2.1 - bug-1577.
Blocked pending a separate prerequisite fix (bug-1578, commit b406dd7):
live testing surfaced a pre-existing controller crash on worker disconnect,
unrelated to this feature, now fixed with regression coverage. Live-
reconfirmed post-fix: controller survives the full join+resupdate+quit flow
indefinitely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
b406dd762a
commit
75a0839d47
2 changed files with 198 additions and 29 deletions
|
|
@ -63,7 +63,7 @@
|
|||
- _Depends: 1.2, 3.1, 4.1, 4.2_
|
||||
|
||||
- [ ] 6. Core: resource injection and per-layer verification checks
|
||||
- [ ] 6.1 Implement worker join and resource injection
|
||||
- [x] 6.1 Implement worker join and resource injection
|
||||
- Join a synthetic worker through the `xce_test` worker context (no real worker daemon process), then run `resupdate` for one or more resource names/values
|
||||
- Observable completion: after this step runs, `resupdate` has been accepted for every requested resource name with zero real worker daemon processes running
|
||||
- _Requirements: 2.1, 2.2, 6.1, 6.4_
|
||||
|
|
@ -116,6 +116,8 @@
|
|||
- _Depends: 7.1_
|
||||
|
||||
## Implementation Notes
|
||||
- Task 6.1 uncovered and got fixed a critical, unrelated, pre-existing controller bug (**bug-1578**, commit `b406dd7`, standalone — a dangling-pointer use-after-free in `sched_worker_change()`/`sched.c` that crashed the controller on worker disconnect once a scheduler had ever joined, 5 months old, unrelated to this feature). It's fixed now with regression coverage (`test_sched_worker_change.c`). If any later task sees a live controller die unexpectedly during a worker-disconnect sequence, confirm you're building from a commit that includes `b406dd7` before assuming it's a new bug. `inject_resources()` also fixed **bug-1577** (in-boundary): `XCE_WORKER_BIN=/bin/true` is exported before invoking `start-all.sh` to suppress the REAL `xcecd_virtual` worker daemon that `deploy-ha.sh`'s pre-existing worker branch would otherwise silently start (Requirement 2.1 violation) whenever the build dir happens to contain one.
|
||||
- Task 6.1: injection results are stored in `INJECTED_VALUES`/`INJECTED_RESULTS` associative arrays plus `$TOP/logs/injection-results.tsv` (`name\tvalue\tresult`) for tasks 6.2-6.4's extension points, which are appended directly after `inject_resources()` in `main()` (explicit "Extension point" comments mark exactly where). `SKILL.md` is now stale (still says injection "not yet implemented", still illustrates `type` in the default set) — deliberately left unedited since task 6.1's CLI parsing shape didn't change; fix this in task 7.1 when SKILL.md is next touched.
|
||||
- Task 5.1: `run-resource-verify.sh` exists now (`.claude/skills/xce-test-resources/`), structured with explicit "Extension point" comments marking where task 6.1 (injection), 6.2-6.4 (per-layer checks), and 7.1 (verdict/cleanup-on-every-path) hook in — read those before starting 6.1. Trap is registered as the first statement in `main()`, before any process starts; `cleanup()` uses a single combined `pkill -9 -f` (alternation pattern) plus retry-until-`pgrep`-clean, not sequential per-process kills (a sequential version was tried first and found non-deterministically flaky under an async SIGTERM trap — bug-1575). Default deployment is single-node (not 3-node HA) — deliberate, since this skill verifies propagation not failover; override via `$XCE_RESVERIFY_PEERS`. Known non-blocking gap: `$XCE_RESVERIFY_SCHED_BIN`/`$XCE_RESVERIFY_APISERVER_BIN` only gate this script's own pre-flight check, NOT which binary `deploy-ha.sh` actually deploys (it hardcodes `/tmp/apiserver` and derives the scheduler path from `$BINARY`'s build dir) — don't rely on these two env vars to test against an alternate binary.
|
||||
- Task 4.2: the apiserver's trace log line (task 6.4 must grep for exactly this) is zap message `"resource-location stored"` with fields `resource` (string), `host` (string), `value` (float64 for numeric resources — the common case; string for `type`/`model`; bool for boolean-typed resources), e.g. `{"resource": "r15m", "host": "review-worker", "value": 1.75}`. Grep the log for `resource-location stored`, then parse `"resource": "<name>"` / `"host": "<host>"` / `"value": <val>` from the trailing field block.
|
||||
- **Task 4.2 discovered bug-1574, which will affect task 8.1 and possibly task 7.1's default resource set — read before implementing either**: `xce_test` can SEGFAULT on `resupdate -n type -v ...` / `resupdate -n model -v ...` (the two `Value_u_svalue`/string-typed resources). Root cause: `set_resval()`'s switch has no `Value_u_svalue` case, so `vals[idx].name` is left pointing at whatever garbage was already on `cmd_resupdate()`'s never-zero-initialized stack array — reading genuinely uninitialized stack memory, invisible to ASan (would need MemorySanitizer), and NON-DETERMINISTIC (task 2.2's reviewer saw `-n type -v foo` NOT crash under full ASAN; task 4.2's implementer saw `-n model -v CustomModelX` crash immediately — different argv/stack layout, same underlying UB). Task 1.2's `SKILL.md` scaffold already documents `type` as part of an illustrative default resource set "for string-path regression coverage" — **before task 7.1 finalizes the actual default resource set, either exclude `type`/`model` from any default until bug-1574 is fixed, or fix bug-1574 first** (it's a small, well-understood fix: zero-initialize `vals[]` in `cmd_resupdate()` and/or add the missing `Value_u_svalue` case to `set_resval()`'s value-population switch — out of boundary for tasks 4.1-4.2, in boundary for whichever task next touches `test.c`'s `cmd_resupdate()`/`set_resval()`, most likely as a prerequisite fix before task 8.1's live e2e run). A crash mid-run would make task 8.1's or the skill's own live verification flaky/non-deterministic, undermining the whole feature's purpose.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue