feat(resource-value-verification): wire deployment into skill via start-all.sh (task 5.1)
run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh scratch dir, confirms all three are genuinely up, and cleans up via a trap registered before any process starts - covering both normal completion and a mid-deployment abort. Logs always preserved regardless of --keep-running. Independently stress-tested with 8+ SIGTERM interrupts at varying deployment points and a genuine crashing-apiserver scenario - zero leftover processes in every case. Single-node by default (propagation testing, not HA failover), overridable via $XCE_RESVERIFY_PEERS. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
2ca25eebd0
commit
da41cce2ea
3 changed files with 361 additions and 31 deletions
|
|
@ -55,7 +55,7 @@
|
|||
- _Requirements: 5.1, 5.2, 5.4_
|
||||
|
||||
- [ ] 5. Integration: full-stack deployment orchestration
|
||||
- [ ] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh`
|
||||
- [x] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh`
|
||||
- Invoke `xce-deploy-ha`'s `deploy-ha.sh` (task 3.1) into a fresh scratch directory, then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already gates scheduler/apiserver startup on the controller health-check and handles the `-vvv`/debug-level logging needed for tasks 4.1/4.2's trace lines
|
||||
- Register a `trap`-based cleanup (or equivalent) before invoking `start-all.sh`, not just at the end of the script — this must cover a mid-deployment abort (e.g. apiserver fails to start after controller+scheduler are already up), not only a completed PASS/FAIL run; deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`) because Requirement 6.5 requires cleanup-by-default on every path unless the operator passed the "keep running" flag
|
||||
- Observable completion: a single invocation of the orchestration script leaves a running controller, live scheduler daemon (built with task 4.1's trace log, started at `-vvv`), and apiserver (built with task 4.2's trace log, debug-level logging enabled), all logging under the same scratch directory, with no manual steps; killing the script mid-deployment (e.g. before the apiserver step completes) still removes every process it had already started, unless "keep running" was passed
|
||||
|
|
@ -116,6 +116,7 @@
|
|||
- _Depends: 7.1_
|
||||
|
||||
## Implementation Notes
|
||||
- Task 5.1: `run-resource-verify.sh` exists now (`.claude/skills/xce-test-resources/`), structured with explicit "Extension point" comments marking where task 6.1 (injection), 6.2-6.4 (per-layer checks), and 7.1 (verdict/cleanup-on-every-path) hook in — read those before starting 6.1. Trap is registered as the first statement in `main()`, before any process starts; `cleanup()` uses a single combined `pkill -9 -f` (alternation pattern) plus retry-until-`pgrep`-clean, not sequential per-process kills (a sequential version was tried first and found non-deterministically flaky under an async SIGTERM trap — bug-1575). Default deployment is single-node (not 3-node HA) — deliberate, since this skill verifies propagation not failover; override via `$XCE_RESVERIFY_PEERS`. Known non-blocking gap: `$XCE_RESVERIFY_SCHED_BIN`/`$XCE_RESVERIFY_APISERVER_BIN` only gate this script's own pre-flight check, NOT which binary `deploy-ha.sh` actually deploys (it hardcodes `/tmp/apiserver` and derives the scheduler path from `$BINARY`'s build dir) — don't rely on these two env vars to test against an alternate binary.
|
||||
- Task 4.2: the apiserver's trace log line (task 6.4 must grep for exactly this) is zap message `"resource-location stored"` with fields `resource` (string), `host` (string), `value` (float64 for numeric resources — the common case; string for `type`/`model`; bool for boolean-typed resources), e.g. `{"resource": "r15m", "host": "review-worker", "value": 1.75}`. Grep the log for `resource-location stored`, then parse `"resource": "<name>"` / `"host": "<host>"` / `"value": <val>` from the trailing field block.
|
||||
- **Task 4.2 discovered bug-1574, which will affect task 8.1 and possibly task 7.1's default resource set — read before implementing either**: `xce_test` can SEGFAULT on `resupdate -n type -v ...` / `resupdate -n model -v ...` (the two `Value_u_svalue`/string-typed resources). Root cause: `set_resval()`'s switch has no `Value_u_svalue` case, so `vals[idx].name` is left pointing at whatever garbage was already on `cmd_resupdate()`'s never-zero-initialized stack array — reading genuinely uninitialized stack memory, invisible to ASan (would need MemorySanitizer), and NON-DETERMINISTIC (task 2.2's reviewer saw `-n type -v foo` NOT crash under full ASAN; task 4.2's implementer saw `-n model -v CustomModelX` crash immediately — different argv/stack layout, same underlying UB). Task 1.2's `SKILL.md` scaffold already documents `type` as part of an illustrative default resource set "for string-path regression coverage" — **before task 7.1 finalizes the actual default resource set, either exclude `type`/`model` from any default until bug-1574 is fixed, or fix bug-1574 first** (it's a small, well-understood fix: zero-initialize `vals[]` in `cmd_resupdate()` and/or add the missing `Value_u_svalue` case to `set_resval()`'s value-population switch — out of boundary for tasks 4.1-4.2, in boundary for whichever task next touches `test.c`'s `cmd_resupdate()`/`set_resval()`, most likely as a prerequisite fix before task 8.1's live e2e run). A crash mid-run would make task 8.1's or the skill's own live verification flaky/non-deterministic, undermining the whole feature's purpose.
|
||||
- Consolidated list of pre-existing (not-fixed-here) bugs surfaced by this feature's own live testing so far: bug-1571 (apiserver `RegisterToBackend` gate checks `Address` before `Peers` — worked around in generated config, not source, by task 3.1), bug-1572 (scheduler double-free/crash on a resource's 2nd periodic report — task 4.1), bug-1573 (see buglog.json, task 4.1), bug-1574 (`xce_test` segfault on string-typed `resupdate` — task 4.2, described above).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue