E2 Phase 2: external-license-resources 运维面 — stale 标记、last-updated 展示、xceadmin extres status
design.md 3.5 Phase 2(PEN D reason 专用名为可选项,复用 PREASON_NO_RESOURCE
已带资源名,跳过):
① esd stale 标记与 WARN
- esd_source_t 增 stale_limit/last_ok_time;esd_source_is_stale(): 从未有值
不算 stale、时钟超前钳到 0、严格大于(interval==limit 不抖动)、
ESD_STALE_LIMIT_DEFAULT 150;fail-open 保持旧值不变
- 采集失败首轮一次性 WARN,过限后每轮升级为 STALE WARN(带 age/limit/kept
值),恢复即静默
- status 文件 esd_status_write(): daemon/pid/started_at/generated_at/
pushes ok/fail/sources[{name,last_value,last_ok_time,stale}],tmp+rename
原子写;写失败仅状态转换时告警一次,绝不拖垮 poll 循环
② `xce resources` 展示 last-updated
- controller worker.c: EXTERNAL 共享值被接受时盖 last_report——同时盖
live resmgr 对象与 myconf 配置树副本(apiserver 定义 store 只由 join 时
config push 供给,串行的是配置树,缺后者 CLI 永远显示 never)
- scheduler core_events.c: RESOURCES_PUSH 原地更新拷贝 last_report(纯
metadata,resobj.c 比较本就排除)
- apiserver: model 带 last_report + 默认列(SelectFields 只投影默认列,
缺失即 API/CLI 全链不可见);convert cResourceToModel 透传
- apiserver handler.go: OnResourceLocationUpdated 对 host=="" 且定义带
EXTERNAL(0x10) 的推送刷新 store 内 LastReport——config push 快照之外
的活刷新;host 来源值与非 EXTERNAL 资源绝不触碰
- xcecli: LAST_UPDATED 列(-l verbose 表格/详情/JSON 三态),0 显示 "-"
④ `xceadmin extres status`
- 本机读 status 文件(远程读取留待 PR-2,需新 wire 协议;LAST_UPDATED
已覆盖远程新鲜度语义)——--status-file 显式路径或
/var/run/xce/esd-status.json 默认,无 controller 连接依赖
- XceExtresStatusError 本地文件类错误;click default=None 装饰期冻结坑
用调用期解析
TDD:test_extres_resvals +2(配置树盖戳/非 EXTERNAL 不盖戳)、test_esd_core
+4(stale 判式/状态文件)、apiserver models +2(默认列/json tag)、backend
integration +2(EXTERNAL 刷新/非 EXTERNAL 不刷新)、xceadmin +11(渲染/
本地读/容错)。全绿:controller 1360、scheduler 100、apiserver(cgo+nocgo)、
esd 11、xceadmin 385。
t1 E2E 实证:esd join 即推送→API last_report=join 时刻→CLI LAST_UPDATED
"Oct 27 13:52";破坏采集器后 60s 过限转 per-round STALE WARN + status
stale:true + fail-open 保值,恢复脚本即回 false;liccheck 作业提交→运行→
DONE,五 daemon 全健康。
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
parent
3eb1b9716d
commit
805288d747
20 changed files with 1179 additions and 47 deletions
275
src/xceadmin/tests/test_cli_extres.py
Normal file
275
src/xceadmin/tests/test_cli_extres.py
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
"""Tests for the ``extres`` command group (external-license-resources
|
||||
Phase 2 item 4, design.md 3.5 "``xceadmin extres status``").
|
||||
|
||||
``extres status`` is a LOCAL-file command -- the inverse slice of
|
||||
``scheduler dump``: where ``dump()`` proved xceadmin can be a local-file
|
||||
decoder with no network at all, ``extres status`` reads the status JSON
|
||||
``xceesd`` writes on its own host (``esd_status_write``,
|
||||
``src/esd/src/esd_status.c``) and formats it for the operator. No wire,
|
||||
no ``--controller``, no TLS -- the daemon's status file is deliberately
|
||||
the observability surface until the PR-2 proxy formalizes a status
|
||||
channel (design.md 3.6; a temporary remote protocol would be
|
||||
redesigned out from under us by PR-2's provider-lease model).
|
||||
|
||||
Split into its own file (rather than appended to ``test_cli.py``, 1187
|
||||
lines at the time of writing) to keep both files under the project's
|
||||
800-line guideline -- the same split precedent as
|
||||
``test_cli_scheduler_mock.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from xceadmin import cli as cli_module
|
||||
from xceadmin.cli import cli
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures: realistic status JSON in the exact shape esd_status_write()
|
||||
# emits (field order aside -- we parse, we never substring-match).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_NOW = 1_800_000_000 # a fixed "now" for deterministic age rendering
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def status_file(tmp_path: Path) -> Path:
|
||||
"""A status file exactly as esd_status_write() writes it for two
|
||||
sources: one fresh, one stale (last_ok_time 3600s in the past of
|
||||
_NOW against the daemon's 150s default limit).
|
||||
"""
|
||||
path = tmp_path / "esd-status.json"
|
||||
payload = {
|
||||
"daemon": "xceesd",
|
||||
"pid": 12345,
|
||||
"started_at": _NOW - 7200,
|
||||
"generated_at": _NOW,
|
||||
"pushes_ok": 142,
|
||||
"pushes_fail": 3,
|
||||
"sources": [
|
||||
{
|
||||
"name": "Verilog_XL",
|
||||
"last_value": 2.0,
|
||||
"last_ok_time": _NOW - 5,
|
||||
"stale": False,
|
||||
},
|
||||
{
|
||||
"name": "VCS",
|
||||
"last_value": 7.0,
|
||||
"last_ok_time": _NOW - 3600,
|
||||
"stale": True,
|
||||
},
|
||||
],
|
||||
}
|
||||
path.write_text(json.dumps(payload))
|
||||
return path
|
||||
|
||||
|
||||
def _freeze_now(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Pin time.time() to _NOW so age rendering is deterministic."""
|
||||
monkeypatch.setattr(cli_module.time, "time", lambda: _NOW)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formatting helper: pure functions tested directly (the _format_record_
|
||||
# result precedent in test_cli.py).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_format_status_renders_daemon_identity_and_counters() -> None:
|
||||
from xceadmin.cli import _format_extres_status
|
||||
|
||||
payload = json.loads(
|
||||
(Path(__file__).parent / "data" / "esd-status-sample.json").read_text()
|
||||
) if (Path(__file__).parent / "data" / "esd-status-sample.json").exists() else {
|
||||
"daemon": "xceesd",
|
||||
"pid": 12345,
|
||||
"started_at": _NOW - 7200,
|
||||
"generated_at": _NOW,
|
||||
"pushes_ok": 142,
|
||||
"pushes_fail": 3,
|
||||
"sources": [],
|
||||
}
|
||||
|
||||
text = _format_extres_status(payload, now=_NOW)
|
||||
|
||||
assert "xceesd" in text
|
||||
assert "12345" in text
|
||||
assert "142" in text
|
||||
assert "3" in text
|
||||
|
||||
|
||||
def test_format_status_marks_stale_source_with_age() -> None:
|
||||
from xceadmin.cli import _format_extres_status
|
||||
|
||||
payload = {
|
||||
"daemon": "xceesd",
|
||||
"pid": 1,
|
||||
"started_at": _NOW - 100,
|
||||
"generated_at": _NOW,
|
||||
"pushes_ok": 0,
|
||||
"pushes_fail": 0,
|
||||
"sources": [
|
||||
{
|
||||
"name": "VCS",
|
||||
"last_value": 7.0,
|
||||
"last_ok_time": _NOW - 3600,
|
||||
"stale": True,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
text = _format_extres_status(payload, now=_NOW)
|
||||
|
||||
assert "VCS" in text
|
||||
assert "STALE" in text
|
||||
assert "3600s" in text
|
||||
|
||||
|
||||
def test_format_status_fresh_source_shows_age_only() -> None:
|
||||
from xceadmin.cli import _format_extres_status
|
||||
|
||||
payload = {
|
||||
"daemon": "xceesd",
|
||||
"pid": 1,
|
||||
"started_at": _NOW - 100,
|
||||
"generated_at": _NOW,
|
||||
"pushes_ok": 0,
|
||||
"pushes_fail": 0,
|
||||
"sources": [
|
||||
{
|
||||
"name": "Verilog_XL",
|
||||
"last_value": 2.0,
|
||||
"last_ok_time": _NOW - 5,
|
||||
"stale": False,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
text = _format_extres_status(payload, now=_NOW)
|
||||
|
||||
assert "Verilog_XL" in text
|
||||
assert "STALE" not in text
|
||||
assert "5s" in text
|
||||
|
||||
|
||||
def test_format_status_source_with_no_value_yet() -> None:
|
||||
from xceadmin.cli import _format_extres_status
|
||||
|
||||
payload = {
|
||||
"daemon": "xceesd",
|
||||
"pid": 1,
|
||||
"started_at": _NOW - 100,
|
||||
"generated_at": _NOW,
|
||||
"pushes_ok": 0,
|
||||
"pushes_fail": 0,
|
||||
"sources": [
|
||||
{
|
||||
"name": "Verilog_XL",
|
||||
"last_value": None,
|
||||
"last_ok_time": 0,
|
||||
"stale": False,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
text = _format_extres_status(payload, now=_NOW)
|
||||
|
||||
assert "Verilog_XL" in text
|
||||
assert "no value yet" in text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI wiring: arg parsing, file reading, error translation.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cli_help_shows_extres_command() -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["--help"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "extres" in result.output
|
||||
|
||||
|
||||
def test_extres_status_reads_local_file(status_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_freeze_now(monkeypatch)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "xceesd" in result.output
|
||||
assert "Verilog_XL" in result.output
|
||||
assert "VCS" in result.output
|
||||
|
||||
|
||||
def test_extres_status_defaults_to_etc_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""With no --status-file, the default path is the deployed default
|
||||
(/var/run/xce/esd-status.json -- where esd.yaml's production
|
||||
status_file default points); a missing default file is a clean
|
||||
error, not a traceback.
|
||||
"""
|
||||
monkeypatch.setattr(
|
||||
cli_module, "DEFAULT_STATUS_FILE", str(tmp_path / "nonexistent.json")
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["extres", "status"])
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "nonexistent.json" in result.output
|
||||
|
||||
|
||||
def test_extres_status_missing_file_clean_error(tmp_path: Path) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli, ["extres", "status", "--status-file", str(tmp_path / "gone.json")]
|
||||
)
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "gone.json" in result.output
|
||||
assert "Traceback" not in result.output
|
||||
|
||||
|
||||
def test_extres_status_invalid_json_clean_error(tmp_path: Path) -> None:
|
||||
bad = tmp_path / "bad.json"
|
||||
bad.write_text("{not json")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)])
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "Traceback" not in result.output
|
||||
|
||||
|
||||
def test_extres_status_wrong_schema_clean_error(tmp_path: Path) -> None:
|
||||
"""JSON that parses but is not an esd status object (e.g. someone
|
||||
pointed --status-file at some other JSON) fails loud and clean.
|
||||
"""
|
||||
bad = tmp_path / "other.json"
|
||||
bad.write_text(json.dumps({"hello": "world"}))
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)])
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "Traceback" not in result.output
|
||||
|
||||
|
||||
def test_extres_status_no_controller_needed(monkeypatch: pytest.MonkeyPatch, status_file: Path) -> None:
|
||||
"""The command must not touch the controller: XCE_CONTROLLER_PEERS
|
||||
pointing at a dead endpoint changes nothing (the file is local).
|
||||
"""
|
||||
monkeypatch.setenv("XCE_CONTROLLER_PEERS", "127.0.0.1:1")
|
||||
monkeypatch.setenv("XCEADMIN_CONTROLLER", "127.0.0.1:1")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
Loading…
Add table
Add a link
Reference in a new issue