E2 Phase 2: external-license-resources 运维面 — stale 标记、last-updated 展示、xceadmin extres status

design.md 3.5 Phase 2(PEN D reason 专用名为可选项,复用 PREASON_NO_RESOURCE
已带资源名,跳过):

① esd stale 标记与 WARN
  - esd_source_t 增 stale_limit/last_ok_time;esd_source_is_stale(): 从未有值
    不算 stale、时钟超前钳到 0、严格大于(interval==limit 不抖动)、
    ESD_STALE_LIMIT_DEFAULT 150;fail-open 保持旧值不变
  - 采集失败首轮一次性 WARN,过限后每轮升级为 STALE WARN(带 age/limit/kept
    值),恢复即静默
  - status 文件 esd_status_write(): daemon/pid/started_at/generated_at/
    pushes ok/fail/sources[{name,last_value,last_ok_time,stale}],tmp+rename
    原子写;写失败仅状态转换时告警一次,绝不拖垮 poll 循环

② `xce resources` 展示 last-updated
  - controller worker.c: EXTERNAL 共享值被接受时盖 last_report——同时盖
    live resmgr 对象与 myconf 配置树副本(apiserver 定义 store 只由 join 时
    config push 供给,串行的是配置树,缺后者 CLI 永远显示 never)
  - scheduler core_events.c: RESOURCES_PUSH 原地更新拷贝 last_report(纯
    metadata,resobj.c 比较本就排除)
  - apiserver: model 带 last_report + 默认列(SelectFields 只投影默认列,
    缺失即 API/CLI 全链不可见);convert cResourceToModel 透传
  - apiserver handler.go: OnResourceLocationUpdated 对 host=="" 且定义带
    EXTERNAL(0x10) 的推送刷新 store 内 LastReport——config push 快照之外
    的活刷新;host 来源值与非 EXTERNAL 资源绝不触碰
  - xcecli: LAST_UPDATED 列(-l verbose 表格/详情/JSON 三态),0 显示 "-"

④ `xceadmin extres status`
  - 本机读 status 文件(远程读取留待 PR-2,需新 wire 协议;LAST_UPDATED
    已覆盖远程新鲜度语义)——--status-file 显式路径或
    /var/run/xce/esd-status.json 默认,无 controller 连接依赖
  - XceExtresStatusError 本地文件类错误;click default=None 装饰期冻结坑
    用调用期解析

TDD:test_extres_resvals +2(配置树盖戳/非 EXTERNAL 不盖戳)、test_esd_core
+4(stale 判式/状态文件)、apiserver models +2(默认列/json tag)、backend
integration +2(EXTERNAL 刷新/非 EXTERNAL 不刷新)、xceadmin +11(渲染/
本地读/容错)。全绿:controller 1360、scheduler 100、apiserver(cgo+nocgo)、
esd 11、xceadmin 385。

t1 E2E 实证:esd join 即推送→API last_report=join 时刻→CLI LAST_UPDATED
"Oct 27 13:52";破坏采集器后 60s 过限转 per-round STALE WARN + status
stale:true + fail-open 保值,恢复脚本即回 false;liccheck 作业提交→运行→
DONE,五 daemon 全健康。

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-27 14:00:33 +00:00
parent 3eb1b9716d
commit 805288d747
20 changed files with 1179 additions and 47 deletions

View file

@ -0,0 +1,275 @@
"""Tests for the ``extres`` command group (external-license-resources
Phase 2 item 4, design.md 3.5 "``xceadmin extres status``").
``extres status`` is a LOCAL-file command -- the inverse slice of
``scheduler dump``: where ``dump()`` proved xceadmin can be a local-file
decoder with no network at all, ``extres status`` reads the status JSON
``xceesd`` writes on its own host (``esd_status_write``,
``src/esd/src/esd_status.c``) and formats it for the operator. No wire,
no ``--controller``, no TLS -- the daemon's status file is deliberately
the observability surface until the PR-2 proxy formalizes a status
channel (design.md 3.6; a temporary remote protocol would be
redesigned out from under us by PR-2's provider-lease model).
Split into its own file (rather than appended to ``test_cli.py``, 1187
lines at the time of writing) to keep both files under the project's
800-line guideline -- the same split precedent as
``test_cli_scheduler_mock.py``.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from click.testing import CliRunner
from xceadmin import cli as cli_module
from xceadmin.cli import cli
# ---------------------------------------------------------------------------
# Fixtures: realistic status JSON in the exact shape esd_status_write()
# emits (field order aside -- we parse, we never substring-match).
# ---------------------------------------------------------------------------
_NOW = 1_800_000_000 # a fixed "now" for deterministic age rendering
@pytest.fixture
def status_file(tmp_path: Path) -> Path:
"""A status file exactly as esd_status_write() writes it for two
sources: one fresh, one stale (last_ok_time 3600s in the past of
_NOW against the daemon's 150s default limit).
"""
path = tmp_path / "esd-status.json"
payload = {
"daemon": "xceesd",
"pid": 12345,
"started_at": _NOW - 7200,
"generated_at": _NOW,
"pushes_ok": 142,
"pushes_fail": 3,
"sources": [
{
"name": "Verilog_XL",
"last_value": 2.0,
"last_ok_time": _NOW - 5,
"stale": False,
},
{
"name": "VCS",
"last_value": 7.0,
"last_ok_time": _NOW - 3600,
"stale": True,
},
],
}
path.write_text(json.dumps(payload))
return path
def _freeze_now(monkeypatch: pytest.MonkeyPatch) -> None:
"""Pin time.time() to _NOW so age rendering is deterministic."""
monkeypatch.setattr(cli_module.time, "time", lambda: _NOW)
# ---------------------------------------------------------------------------
# Formatting helper: pure functions tested directly (the _format_record_
# result precedent in test_cli.py).
# ---------------------------------------------------------------------------
def test_format_status_renders_daemon_identity_and_counters() -> None:
from xceadmin.cli import _format_extres_status
payload = json.loads(
(Path(__file__).parent / "data" / "esd-status-sample.json").read_text()
) if (Path(__file__).parent / "data" / "esd-status-sample.json").exists() else {
"daemon": "xceesd",
"pid": 12345,
"started_at": _NOW - 7200,
"generated_at": _NOW,
"pushes_ok": 142,
"pushes_fail": 3,
"sources": [],
}
text = _format_extres_status(payload, now=_NOW)
assert "xceesd" in text
assert "12345" in text
assert "142" in text
assert "3" in text
def test_format_status_marks_stale_source_with_age() -> None:
from xceadmin.cli import _format_extres_status
payload = {
"daemon": "xceesd",
"pid": 1,
"started_at": _NOW - 100,
"generated_at": _NOW,
"pushes_ok": 0,
"pushes_fail": 0,
"sources": [
{
"name": "VCS",
"last_value": 7.0,
"last_ok_time": _NOW - 3600,
"stale": True,
}
],
}
text = _format_extres_status(payload, now=_NOW)
assert "VCS" in text
assert "STALE" in text
assert "3600s" in text
def test_format_status_fresh_source_shows_age_only() -> None:
from xceadmin.cli import _format_extres_status
payload = {
"daemon": "xceesd",
"pid": 1,
"started_at": _NOW - 100,
"generated_at": _NOW,
"pushes_ok": 0,
"pushes_fail": 0,
"sources": [
{
"name": "Verilog_XL",
"last_value": 2.0,
"last_ok_time": _NOW - 5,
"stale": False,
}
],
}
text = _format_extres_status(payload, now=_NOW)
assert "Verilog_XL" in text
assert "STALE" not in text
assert "5s" in text
def test_format_status_source_with_no_value_yet() -> None:
from xceadmin.cli import _format_extres_status
payload = {
"daemon": "xceesd",
"pid": 1,
"started_at": _NOW - 100,
"generated_at": _NOW,
"pushes_ok": 0,
"pushes_fail": 0,
"sources": [
{
"name": "Verilog_XL",
"last_value": None,
"last_ok_time": 0,
"stale": False,
}
],
}
text = _format_extres_status(payload, now=_NOW)
assert "Verilog_XL" in text
assert "no value yet" in text
# ---------------------------------------------------------------------------
# CLI wiring: arg parsing, file reading, error translation.
# ---------------------------------------------------------------------------
def test_cli_help_shows_extres_command() -> None:
runner = CliRunner()
result = runner.invoke(cli, ["--help"])
assert result.exit_code == 0
assert "extres" in result.output
def test_extres_status_reads_local_file(status_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
_freeze_now(monkeypatch)
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)])
assert result.exit_code == 0, result.output
assert "xceesd" in result.output
assert "Verilog_XL" in result.output
assert "VCS" in result.output
def test_extres_status_defaults_to_etc_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""With no --status-file, the default path is the deployed default
(/var/run/xce/esd-status.json -- where esd.yaml's production
status_file default points); a missing default file is a clean
error, not a traceback.
"""
monkeypatch.setattr(
cli_module, "DEFAULT_STATUS_FILE", str(tmp_path / "nonexistent.json")
)
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status"])
assert result.exit_code != 0
assert "nonexistent.json" in result.output
def test_extres_status_missing_file_clean_error(tmp_path: Path) -> None:
runner = CliRunner()
result = runner.invoke(
cli, ["extres", "status", "--status-file", str(tmp_path / "gone.json")]
)
assert result.exit_code != 0
assert "gone.json" in result.output
assert "Traceback" not in result.output
def test_extres_status_invalid_json_clean_error(tmp_path: Path) -> None:
bad = tmp_path / "bad.json"
bad.write_text("{not json")
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)])
assert result.exit_code != 0
assert "Traceback" not in result.output
def test_extres_status_wrong_schema_clean_error(tmp_path: Path) -> None:
"""JSON that parses but is not an esd status object (e.g. someone
pointed --status-file at some other JSON) fails loud and clean.
"""
bad = tmp_path / "other.json"
bad.write_text(json.dumps({"hello": "world"}))
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)])
assert result.exit_code != 0
assert "Traceback" not in result.output
def test_extres_status_no_controller_needed(monkeypatch: pytest.MonkeyPatch, status_file: Path) -> None:
"""The command must not touch the controller: XCE_CONTROLLER_PEERS
pointing at a dead endpoint changes nothing (the file is local).
"""
monkeypatch.setenv("XCE_CONTROLLER_PEERS", "127.0.0.1:1")
monkeypatch.setenv("XCEADMIN_CONTROLLER", "127.0.0.1:1")
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)])
assert result.exit_code == 0, result.output