lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

merge feature/license-resources into working

Browse source 
E2 external-license-resources complete: Phase 1 (xceesd + extres role +
EXTERNAL admission), Phase 2 (ops plane: stale WARN, LAST_UPDATED,
xceadmin extres status), Phase 3 (per-project/per-queue quotas via
limit vres, bug-1976 allocs_shared fix), 3.3b elim collector, A-batch
admin-plane fixes, bugs 1972-1978 closed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-29 02:24:54 +00:00
commit 9b652d93e5
85 changed files with 6957 additions and 133 deletions
Download patch file Download diff file Expand all files Collapse all files

9
.kiro/specs/external-license-resources/brief.md
View file

@ -12,7 +12,7 @@ XCE 无法调度依赖外部 license server（FlexLM/lmstat 等）的 EDA 应用
XCE_RESF_EXTERNAL 是死 flag）。
## Desired Outcome
- **xceextresd**：独立采集 daemon（避免 lmstat 阻塞连坐 controller），
- **xceesd**：独立采集 daemon（避免 lmstat 阻塞连坐 controller），
以新 `HostRole` 成员 join controller，采集器插件：flexlm（lmstat）、
script（任意外部资源）。wire 零 schema 变更（复用 ResValList 管线），
唯一变更是 HostRole 加成员。
@ -38,7 +38,7 @@ share 机制）。
- **elastic-shared-resources**：同管线两种记账模式按资源分流
（EXTERNAL=server-authoritative，无 flag=elastic base+borrowed）——
opupd 守卫"SHARED 即收"预留其写入路径。即对其决策日志 P9 的回答。
- **PR-2 proxy**（resource-questions.md）：xceextresd 定位为该 proxy 的
- **PR-2 proxy**（resource-questions.md）：xceesd 定位为该 proxy 的
第一片（只读 report 能力），HostRole 只加一个成员。
- **project-config**：per-app 配额→per-project share（project tree 是
唯一并发上限处，limits.yaml 退役）。需要其扩展：集群级 pool location。
@ -48,7 +48,7 @@ share 机制）。
## Boundary Candidates
- HostRole_extres join/leave + opupd 准入守卫（controller）
- unsatisfy_ EXTERNAL 分支（scheduler）
- xceextresd daemon 与采集器插件（新 src/extresd/）
- xceesd daemon 与采集器插件（新 src/esd/）
- mapping.py `location: [external]` 路由
## Out of Boundary
@ -64,6 +64,7 @@ share 机制）。
provisioning-demand（headroom 需求源）、project-config（集群级池）
## Open（待用户/跨团队决策）
1. 命名：xceextresd（推荐）vs xcelsd。
1. ~~命名~~ 已定（2026-10-27）：**xceesd**（src/esd/，esd.yaml；HostRole
枚举成员 extres）。
2. 断联策略：fail-open（推荐）vs fail-closed。
3. P9 模式拆分与三个扩展点需与 Rongsong 确认（design §3.6）。

44
.kiro/specs/external-license-resources/design.md
View file

@ -110,17 +110,23 @@ license 服务器的"当前空闲数"是外部真值；本地只需补一个窗
### 3.1 架构决策：独立 daemon + 采集器插件
**xceextresd（外部资源守护进程）**，新 `src/extresd/`，C 实现，
照 scheduler 的 join 方式以新 `HostRole_extres` 角色接入 controller。
内部采集器插件接口：
**xceesd（外部资源守护进程；XCE External-resources Daemon）**，新
`src/esd/`，C 实现，照 scheduler 的 join 方式以新 `HostRole_extres`
角色接入 controller。内部采集器插件接口：
```
xceextresd
xceesd
├─ collector/flexlm.c — lmstat -c port@host -f feature 解析（EDA license 主路径）
├─ collector/script.c — 用户脚本输出 key=value/JSON（任意外部资源：云配额、网络带宽…）
└─ collector/http.c —（预留）
```
> **命名**（2026-10-27 定）：daemon 名 `xceesd`，目录 `src/esd/`，二进制
> `xceesd`，配置 `esd.yaml`，代码标识符 `esd`。wire 层 HostRole 枚举成员
> 仍用 `extres`（角色语义，与 reporter/provisioner 讨论对齐）。
> 注意 `esd` 是高频子串（closed/parsed…），全文检索认准全名。
> 旧稿中的 `xceextresd`/`extresd` 均指本 daemon（改名前写法）。
- **为什么不是 controller 内插件**：lmstat 是秒级阻塞调用，违反 controller
async I/O 红线（delaymgr/vtask 不得阻塞，HA/心跳会被连坐）；license server
故障必须与 controller 故障域隔离。
@ -133,7 +139,7 @@ xceextresd
### 3.2 数据流（wire 零 schema 变更，除 HostRole 加成员）
```
xceextresd (join: HostRole_extres)
xceesd (join: HostRole_extres)
│ OPCode_update / Update_data_resvals（现 wire：ResourceVal{name, host:"", val}）
▼
opupd.c do_update_resvals ── 放行 extres 角色；flag 守卫只许 SHARED 资源（§3.6）
@ -185,7 +191,7 @@ EXTERNAL：F ≥ uncomit + wanted（unsatisfy_ 循环内一个 flag 分支；
与 Slurm `_handle_consumed` 的 deficit 算法同构（都是"避开外部消费者份额"），
但更简：不需要 LastConsumed/external 推导，因为 lmstat 直接给 free。
### 3.3b elim（LSF load index）兼容性 —— 已盘点，归 xcecd 不归 extresd
### 3.3b elim（LSF load index）兼容性 —— 已盘点，归 xcecd 不归 esd
**LSF elim 模型**：用户在主机上部署脚本，周期 echo `资源名 数量`，lim 收集后
变成可调度资源。per-host 语义。License Scheduler 历史上正是从 elim 派生——
@ -217,15 +223,15 @@ posix_spawn 路径；读 stdout `name value` 行 elim 原格式；调现有 regi
API 进 registry；脚本超时看门狗 kill）。MAX_LOAD_INDICES=64 够用。
工作量 D 类（中型），与 E2 独立可并行。
**归属决策：elim 支持放 xcecd，不放 xceextresd**。理由：
**归属决策：elim 支持放 xcecd，不放 xceesd**。理由：
1. 作用域——elim 值 per-host（"这台机器有多少"），license 值集群全局
（"全池还剩多少"）；worker 本来就站在每台主机上，让 extresd 管主机值
（"全池还剩多少"）；worker 本来就站在每台主机上，让 esd 管主机值
是数据绕远路。
2. 故障域——elim 脚本挂只影响一台主机上报；license server 断联影响全集群
准入。xcecd 采集器继承前者的隔离性。
3. LSF 自己也是分开的（lim vs lsld/blcollect），复刻这个分界。
4. XCE 架构两条腿天然对齐：worker→per-worker resvals→host 判式；
extresd→sharedinst→全局判式。同一资源模型（定义/rusage/pend reason/
esd→sharedinst→全局判式。同一资源模型（定义/rusage/pend reason/
CLI 展示）。
**统一记账规则（两条腿共享，同一 flag 位决定）**：EXTERNAL 资源的满足性
@ -252,13 +258,13 @@ count) 聚合——无法判定 A=2/B=3 还是 A=3/B=2。LSF License Scheduler
daemon 每轮对账：Σ(派发中) vs 服务器 used，差值 = 外部消费者 + 泄漏，
异常 alert。漂移有界且方向安全：少拿→配额保守不超卖；多拿→服务器
total 暴露。
3. **报告层 = 证据分级 + 残差桶**（matcher 为 extresd 插件，不猜、显式
3. **报告层 = 证据分级 + 残差桶**（matcher 为 esd 插件，不猜、显式
化"不知道"）：
- Tier 0：派发记账（app→授予数），精确，永远可用；
- Tier 1：vendor checkout 属性（如 Synopsys LM_PROJECT 类，lmstat 可见；
XCE 作业 env 已注入 XCE_JOBID，vendor 走 env 即可盖入），精确；
- Tier 1'：PID 匹配（部分 vendor daemon 报客户端 PID）——jobPidInfo
已有 jobid→pid 映射（3545aa0d），extresd 从 controller 拉即可，精确；
已有 jobid→pid 映射（3545aa0d），esd 从 controller 拉即可，精确；
- Tier 2：(user, host) + checkout start 时间窗 匹配作业记录（LSF 基线
法升级：两个独立 XCE 作业通常可按 start 时间拆开），user+host 相同时
只能给出"这批作业合计持有 K"的界；
@ -287,7 +293,7 @@ resources:
```
```yaml
# extresd.yaml（daemon 侧：操作）
# esd.yaml（daemon 侧：操作）
poll_interval: 30s
resources:
- name: Verilog_XL # 必须与 resources.yaml 同名（控制器侧 unknown-resource 拒）
@ -300,7 +306,7 @@ resources:
stale_limit: 150s # 连续取不到 → WARN 保旧值（fail-open，可配 fail-closed 清零）
```
职责分界：resources.yaml 管声明（作业能否 rusage），extresd.yaml 管采集
职责分界：resources.yaml 管声明（作业能否 rusage），esd.yaml 管采集
（值从哪来）。`seed_sharedinst_from_config` 对 EXTERNAL 资源**不播种**
（值只来自 daemon；daemon 未上线时 get_resvals 读 0 → PEND NO_RESOURCE
detail 带资源名）。
@ -317,7 +323,7 @@ detail 带资源名）。
3. resource.c unsatisfy_：EXTERNAL 跳过 inuse 项（+trace 行）
4. cluster.c seed：EXTERNAL 不播种
5. mapping.py：`location: [external]` 路由
6. xceextresd 骨架：join/rejoin/failover 重连即全量重推（worker 状态补报
6. xceesd 骨架：join/rejoin/failover 重连即全量重推（worker 状态补报
9d7628e 同款语义）+ script/flexlm 采集器 + 轮询去抖
7. TDD：testsuite 红→绿（opupd 守卫、unsatisfy_ 分支、采集器解析）；
t1 E2E 用 script 采集器 + 假 license 脚本（不必有真 FlexLM）
@ -335,7 +341,7 @@ PEND detail 含 lic 名）；daemon kill/重启后值全量补推；controller
daemon 重新 join 新主并补推；`--sim` 录制含 daemon 推送事件且回放一致。
**elim 归属（与 E2 并行的独立 D 类项）**：LSF elim（per-host 外部资源脚本）
归 xcecd（worker）不归 xceextresd——详见 §3.3b。license（全局）与 elim
归 xcecd（worker）不归 xceesd——详见 §3.3b。license（全局）与 elim
（主机）是两条腿、同一资源模型、同一记账规则（EXTERNAL 跳 inuse）。
### 3.6 与 Rongsong 统一资源模型 spec 群的合并性审查（2026-10-26）
@ -379,13 +385,13 @@ daemon 推 free 直接覆盖 sharedInst。两套各写各的 → 同一资源 sh
唯一并发上限处，limits.yaml 退役"（332f86ea）；§3.3c Tier-0 配额写成
project share/member_quota 由 limit plugin 编译，不发明新机制。
- staleness/interval 进**资源描述符**（其 Phase 1 descriptor 字段
interval/last_report 已预留），不放 extresd.yaml；daemon 只执行。
interval/last_report 已预留），不放 esd.yaml；daemon 只执行。
- 持久化分级：daemon 推的 F 是 dynamic reported value → 非 WAL fact、
进 scheduler snapshot + daemon 重连全量补推即安全；将来 elastic 模式
的 total 才是 WAL fact。与 persistence.md 一致。
- 角色收敛：PR-2 proxy（leaning）统一 reporter+provisioner 为一个 daemon
role；**HostRole 只加一个成员**（避免将来再出现第二个同义 enum），
xceextresd 定位为 PR-2 proxy 的第一片（只读 report 能力）。
xceesd 定位为 PR-2 proxy 的第一片（只读 report 能力）。
- 归因 Tier 1' PID 匹配引用其 jobPidInfo（我们已实施，3545aa0d）。
**我们的设计对其 spec 的三个扩展点**（他目前装不下，需带回讨论）：
@ -407,8 +413,8 @@ daemon 推 free 直接覆盖 sharedInst。两套各写各的 → 同一资源 sh
"or reuse the provisioner role"）。
### 3.7 提请决策（未决）
1. daemon 命名：`xceextresd`（通用外部资源，推荐）vs `xcelsd`（贴 LSF
license 语义）；定位为 PR-2 proxy 第一片（见 §3.6）。
1. ~~daemon 命名~~ 已定（2026-10-27）：`xceesd` / `src/esd/` / `esd.yaml`；
HostRole 枚举成员仍为 `extres`；定位为 PR-2 proxy 第一片（见 §3.6）。
2. 断联默认策略：fail-open（保旧值+stale WARN，推荐）vs fail-closed（清零停派）。
3. P9 模式拆分与三个扩展点需与 Rongsong 确认（§3.6）。

194
.wolf/buglog.json
View file

@ -6,7 +6,7 @@
"id": "bug-1971",
"status": "fixed",
"status_updated": "2026-10-22",
"status_note": "",
"status_note": "two-choke archive in xce-build.sh; test harness green",
"component": "build",
"severity": "wrong-result",
"timestamp": "2026-10-22T07:50:00Z",
@ -27,6 +27,198 @@
"68c3ccd7"
],
"record": ".wolf/bugs/bug-1971.md"
},
{
"id": "bug-1972",
"status": "fixed",
"status_updated": "2026-10-26",
"status_note": "",
"component": "controller",
"severity": "crash",
"timestamp": "2026-10-26T09:46:58Z",
"last_seen": "2026-10-26T09:46:58Z",
"occurrences": 4,
"error_message": "test_job_live_gc asserted destruction by reading freed memory\n\n**Observed:**\n`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`\n(test_job_live_gc.c:169):\n\n```\n==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290\nREAD of size 8 at 0x5110000da290 thread T0\n #0 job_get_refcount ../lib/jobj.c:715\n #1 test_sweep_purges_expired_done_job test_job_live_gc.c:169\nfreed by:\n #3 job_dereference ../lib/jobj.c:705\n #5 job_live_gc_sweep ../controller/job.c:4777\n```\n\n**Expected:** a correct sweep dereferences the purged job down to refcount 0\nand destroys it; the test must verify destruction without touching the freed\nobject.\n\n**Impact:** controller suite aborts (killed by SIGABRT); the whole\n`controller test` meson entry fails. Test-only — the production sweep\n(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)\nis correct.\n\n**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier\nruns all predate that commit).",
"file": "src/controller/testsuite/test_job_live_gc.c",
"root_cause": "The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs\nAFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the\nentire RTTI object (header + private + data in one calloc block). Reading the\nrefcount through the dangling `job_t*` is itself the use-after-free: the\nstronger the sweep does its job, the more surely the test crashes. The same\npattern is absent from test_apply_pump.c's refcount test because there the\njob is never destroyed.",
"fix": "Move the refcount probe BEFORE the sweep and invert its meaning: assert\nrefcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,\nthen verify destruction by absence — `jlg_find_job(81001) == NULL` plus the\nidempotent second sweep (both already present). `expired` is never dereferenced\nafter the sweep; a comment marks the last safe point.\n\nVerification:\n- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic\n ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero\n ASAN reports).\n- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);\n sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,\n bug-free fixture staleness from fc33a5da's schema change, handled in the same\n working tree).\n- Live: n/a (test-only defect; production sweep unchanged).",
"tags": [
"test-only",
"uaf",
"gc"
],
"related_bugs": [],
"fix_commits": [
"f780a166"
],
"record": ".wolf/bugs/bug-1972.md"
},
{
"id": "bug-1973",
"status": "fixed",
"status_updated": "2026-10-27",
"status_note": "",
"component": "controller",
"severity": "wrong-result",
"timestamp": "2026-10-27T03:27:47Z",
"last_seen": "2026-10-27T03:27:47Z",
"occurrences": 1,
"error_message": "cluster_build_resources aborted at first EEXIST: new resource defs never registered\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).",
"file": "src/controller/cluster.c, src/controller/testsuite/test_config_resource_reconcile.c",
"root_cause": "`cluster_build_resources()` looped over the decoded config's resource list\ncalling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)\nreturns NULL with errno=EEXIST when the name is already registered — which is\nthe COMMON case at activation (boot already registered some definitions). The\nold loop treated that rc as fatal and aborted the whole loop, so every\ndefinition AFTER the first already-registered one was never registered.",
"fix": "Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under\none write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;\nRES_APPLY_CHANGED updates the registered object's definition fields\n(valtype/ordtype/flags/interval) IN PLACE — none of them touches the\nregistration slot (idx) or accumulated accounting; mirrors\nreconcile_config_queues' in-place refresh precedent (second live bug, see\nbug-1974 — kept here as one loop). Negative rc still propagates.\n\nVerification:\n- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +\n `test_reconcile_resources_hot_applies_changed_definition`\n (src/controller/testsuite/test_config_resource_reconcile.c); RED before the\n fix (lic1 unregistered / flags stayed 0x00), GREEN after.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in\n sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at\n free=1; release re-admitted it.",
"tags": [
"config",
"reconcile",
"E2",
"external-license"
],
"related_bugs": [
"bug-1854"
],
"fix_commits": [
"2fcacac2"
],
"record": ".wolf/bugs/bug-1973.md"
},
{
"id": "bug-1974",
"status": "fixed",
"status_updated": "2026-10-27",
"status_note": "",
"component": "controller",
"severity": "wrong-result",
"timestamp": "2026-10-27T03:35:12Z",
"last_seen": "2026-10-27T03:35:12Z",
"occurrences": 1,
"error_message": "config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM\n\n**Observed:**\nController log on t1 after re-pushing a corrected cluster.bin (extlic mapped\nwith `location: [external]` → flags SHARED|GLOBAL|EXTERNAL):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted\n```\n\n**Expected:** update_val_res' SHARED branch routes a host==\"\" resval for a\nSHARED-flagged resource into c->sharedinst; with correct flags the push lands.\n\n**Impact:** the extres daemon's pushes were permanently refused on a controller\nwhose registration predated the flag fix — the stale shape was frozen for the\nprocess lifetime, so re-pushing a corrected config did not heal it.\n\n**Found by:** E2 Phase 1 live E2E on t1.",
"file": "src/controller/cluster.c, src/controller/worker.c, src/xceadmin/src/xcecommon/mapping.py",
"root_cause": "Two compounding factors:\n1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/\n xceadmin-venv) carries a stale mapping.py with NO resourcemap handling\n (83-line diff vs tree) — cluster bins it builds carry flags=0 for every\n resource. Ops issue, fixed by building with the tree venv\n (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs\n redeployment.\n2. **Controller**: even with a corrected bin, `cluster_build_resources()`\n treated an already-registered name as \"nothing to do\" (bug-1854's\n RES_APPLY_CHANGED was reported but never applied), so the stale flags=0\n shape survived every activation.",
"fix": "`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the\nregistered object's definition fields (valtype/ordtype/flags/interval) in\nplace under the resmgr write lock — none of them touches the registration\nslot (idx) or accumulated accounting; mirrors reconcile_config_queues'\n\"updated by configuration activation\" in-place refresh precedent. (Same loop\nas bug-1973's fix.) Producer side tracked separately as ops follow-up:\nredeploy the xceadmin venv.\n\nVerification:\n- Test: `test_reconcile_resources_hot_applies_changed_definition`\n (src/controller/testsuite/test_config_resource_reconcile.c); RED before\n (flags stayed 0x00), GREEN after (0x1C, idx preserved).\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push\n landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN.",
"tags": [
"config",
"flags",
"E2",
"external-license"
],
"related_bugs": [
"bug-1854",
"bug-1973"
],
"fix_commits": [
"2fcacac2"
],
"record": ".wolf/bugs/bug-1974.md"
},
{
"id": "bug-1975",
"status": "fixed",
"status_updated": "2026-10-27",
"status_note": "",
"component": "controller",
"severity": "wrong-result",
"timestamp": "2026-10-27T03:40:05Z",
"last_seen": "2026-10-27T03:40:05Z",
"occurrences": 1,
"error_message": "xceesd idle on dead connection after controller restart (stable value: no re-push)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).",
"file": "src/esd/src/esd.c, src/lib/client.c, src/lib/client_internal.h",
"root_cause": "Three mechanisms conspired:\n1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip\n push) — with a stable value it pushes NOTHING.\n2. A reconnect inside the client library only fires while INSIDE a push's\n failover call — no pushes, no reconnect.\n3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;\n with no push attempt it never runs.",
"fix": "`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`\n(bug-1760's pure predicate — the transport's RDHUP handler sets\nXCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.\nThe write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the\nfull re-push contract for the failover path.\n\nVerification:\n- Test: covered by the live contract (client-side predicate already unit\n tested by bug-1760's suite); esd-side loop is integration-shaped (the\n trigger is a transport callback), so live E2E is the proof.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — controller killed and restarted; esd logged\n `connection broken -- forcing a full re-push to reconnect`, rotated to the\n new primary, re-pushed; `xcecli resources distribution list` shows extlic=4\n and a fresh rusage[extlic=3] job RUNS.",
"tags": [
"esd",
"external-license",
"reconnect",
"E2"
],
"related_bugs": [
"bug-1760",
"bug-1973",
"bug-1974"
],
"fix_commits": [
"2fcacac2"
],
"record": ".wolf/bugs/bug-1975.md"
},
{
"id": "bug-1976",
"status": "fixed",
"status_updated": "2026-10-28",
"status_note": "",
"component": "controller",
"severity": "wrong-result",
"timestamp": "2026-10-28T05:37:09Z",
"last_seen": "2026-10-28T05:26:00Z",
"occurrences": 1,
"error_message": "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced\n\n**Observed:**\nE2 Phase 3 t1 E2E: 4 projA jobs with `-R \"rusage[extlic=1]\"` against a\nper-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran\ninstead of the 4th PENDing with reason limit_global. Live gdb on the\nscheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with\n3 projA jobs RUN: the vres charge never landed.\n\n**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job\nre-admits it; projB unaffected (own per-project instance).\n\n**Impact:** every scheduler-side global limit (the limit vres mechanism)\nwas advisory only — no quota was ever enforced, for ANY shared resource\ncarried via allocs_shared, since dispatch-time adoption overwrote the\nscheduler's own charged accounts with a shared-stripped copy.\n\n**Found by:** spec task (external-license-resources E2 Phase 3 E2E).",
"file": "src/controller/job.c, src/controller/testsuite/test_dispatch_commit.c",
"root_cause": "The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)\nrebuilt `j->job.allocs` from the scheduler's dispatch payload but\nmigrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,\nallocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`\nwas neither moved nor freed (a leak on top). That rebuilt record is\nwhat `sched_job_update_fun()` re-sends to the scheduler as ground\ntruth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,\ncore_events.c) rebuilds `job->accounts` from exactly that wire copy —\nso `accounts_shared == NULL`, `sched_account_jobs` charges nothing to\nthe vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's\nown commit (`sched_commit_decision`, scheduler.c:1806-1812) built the\naccounts correctly — the controller's \"ground truth\" clobbered it.",
"fix": "job.c dispatch-time commit: migrate the shared records alongside the\nhost records — `MOVE_PTR(j->job.allocs->allocs_shared,\nalloc->allocs_shared); j->job.allocs->nres_shared =\nalloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the\ncaller frees the payload; also closes the leak).\n\nVerification:\n- Test: `test_dc_success_migrates_shared_allocs`\n (src/controller/testsuite/test_dispatch_commit.c). RED before the fix\n (\"Expected 2 Was 0\" on nres_shared), GREEN after.\n- Suites: full meson test 136/136 OK (build.dbg), including controller\n unit suite 1361/0.\n- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill\n one RUN job → pending projA re-admitted, external free pool\n (fake_lic) enforced independently; all daemons healthy throughout.",
"tags": [
"external-license-resources",
"limits",
"shared-resources",
"dispatch",
"e2-phase3"
],
"related_bugs": [
"bug-1860"
],
"fix_commits": [
"76ddd75f"
],
"record": ".wolf/bugs/bug-1976.md"
},
{
"id": "bug-1977",
"status": "fixed",
"status_updated": "2026-10-28",
"status_note": "",
"component": "controller",
"severity": "crash",
"timestamp": "2026-10-28T11:56:58Z",
"last_seen": "2026-10-28T11:56:58Z",
"occurrences": 1,
"error_message": "crr_build_config callers' stack arrays smaller than builder -- ASAN stack overflow\n\n**Observed:**\n```\nERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc...\nWRITE of size 8 at 0x7ffc... thread T0\n #0 crr_build_config testsuite/test_config_resource_reconcile.c:126 (dists[3] = &d4)\n #1 test_reconcile_resources_registers_and_seeds_shared_pools testsuite/test_config_resource_reconcile.c:161\n```\nEvery test in the file that calls `crr_build_config` crashes the same way;\nthe full controller suite reports 137/137 FAIL (meson reports each fixture\ncrash as its own failure).\n\n**Expected:** all four callers pass arrays large enough for the builder's\nfixtures; the suite stays green.\n\n**Impact:** the whole controller testsuite was unrunnable under ASAN since the\nexternal-license-resources test fixtures landed (commit 2fcacac2) -- any\nregression in the suite was invisible behind the deterministic crash.\n\n**Found by:** ASAN -- full regression build for the license-resource feature\n(`src/tools/xce-build.sh controller`, profile=asan).",
"file": "src/controller/testsuite/test_config_resource_reconcile.c",
"root_cause": "`crr_build_config` (test_config_resource_reconcile.c:70) registers FIVE\nresources (lic1/hostres/dyn1/extlic/slots -> writes `resources[4]`) and FOUR\ndistributions (writes `dists[3]`) and FIVE vals (writes `vals[4]`). The four\ncaller test functions declared only\n`resources[4] / dists[3] / vals[4]`, so the builder's writes to index 4 / 3 /\n4 overflowed the callers' stack arrays. When the extlic E2 tests were added to\nthe builder's fixture set (2fcacac2), none of the caller declarations were\nresized. (Earlier builds of this branch compiled only a worker-focused subset\n-- `test_config_resource_reconcile` was not linked -- so the overflow went\nunnoticed until the full controller suite was built.)",
"fix": "Resize all four caller arrays to match the builder: `resources[5]`,\n`dists[4]`, `vals[5]`, each with a bug-1977 comment pointing at the sizing\ncontract. Files: src/controller/testsuite/test_config_resource_reconcile.c\n(test_reconcile_resources_registers_and_seeds_shared_pools,\ntest_reconcile_resources_registers_new_defs_past_eexist,\ntest_reconcile_resources_hot_applies_changed_definition,\ntest_reconcile_resources_is_idempotent_and_hot_applies).\n\nVerification:\n- Test: whole-file suite `controller - xce:controller test` (meson): RED\n 137/137 before the fix, GREEN after.\n- Suites: `src/tools/xce-build.sh controller` profile=asan: 6/6 OK\n (sched push, sched worker change, apiserver push, host runusage, no-fork\n rule, controller 91s).\n- Live: n/a (testsuite-only defect; no production code touched).",
"tags": [
"testsuite",
"asan"
],
"related_bugs": [],
"fix_commits": [
"8c1995c2"
],
"record": ".wolf/bugs/bug-1977.md"
},
{
"id": "bug-1978",
"status": "fixed",
"status_updated": "2026-10-28",
"status_note": "",
"component": "controller",
"severity": "wrong-result",
"timestamp": "2026-10-28T12:47:59Z",
"last_seen": "2026-10-28T12:47:59Z",
"occurrences": 3,
"error_message": "esd reconnect re-pushes without re-join -- pushes -EPERM forever\n\n**Observed:**\n```\nt2 /tmp/xceesd-d.log:\n[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)\n[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s)\n[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)\n... (every 10s poll, forever)\n```\nAfter any controller restart the daemon's pushes fail rc=-1 (-EPERM) on\nevery attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,\nthe poll loop runs, the collector script fires every 10s, the status file\nrefreshes with `stale: false` -- only pushes_fail climbs.\n\n**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and\npushes resume (pushes_ok advances, `xce resources distribution list` shows\nextlic).\n\n**Impact:** the entire external-license value chain dies at the first\ncontroller restart: extlic never reaches c->sharedinst, the scheduler never\nhears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever\n(admission sees free=0). Found during the license-resource regression: the\nregression config push (gen 19) succeeded but `resources distribution list`\nshowed NO extlic row -- daemon健康、值链断。\n\n**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline\nevidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller\nrestarts in /opt/xce/logs/xce_controller.log, zero extres joins after\n10:58, and esd strace showing zero network sends (debounce suppressed -- no\nrotate ever triggered again).",
"file": "src/esd/src/esd.c",
"root_cause": "esd.c's reconnect contract was \"reconnect = full re-PUSH\" (esd_on_reconnect\nsets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a\nre-JOIN: the fresh controller endpoint is created by handle_connection()\n(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets\npriv->role = HostRole_extres (join.c handle_join_). The transparent\nreconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket\nbut never re-runs the role join, and the client library's rejoin hook\n(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the\nun-joined endpoint's resval update falls through BOTH branches of\ndo_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and\nreturns rc = -EPERM == -1 forever. (The worker solved this exact shape with\nthe needs_rejoin contract, controller_connector.c; the scheduler likewise.\nThe esd -- written later -- re-used the flag machinery but dropped the\nre-join itself.)",
"fix": "src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and\ng_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing\n`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread\n(outside the library's failover lock -- the xcecd_on_reconnect no-I/O\nconstraint), and only then does the full re-push. Join failure keeps the\nflag set (next tick retries -- the worker's self-healing cadence).\n\nVerification:\n- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push\n + test_reconnect_flags_recover_from_any_previous_state\n (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED\n with the old esd.c (compile error: g_needs_rejoin did not exist -- the\n re-join state was not represented at all), GREEN with the fix: 13/13.\n- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build\n rc=0.\n- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2\n (restart controller, observe \"esd: re-joined after reconnect\" + pushes_ok\n advancing + extlic row in `resources distribution list`).",
"tags": [
"esd",
"reconnect",
"failover"
],
"related_bugs": [
"bug-1760"
],
"fix_commits": [
"72301fd5"
],
"record": ".wolf/bugs/bug-1978.md"
}
]
}

86
.wolf/bugs/bug-1972.md Normal file
View file

@ -0,0 +1,86 @@
---
id: "bug-1972"
status: "fixed"
status_updated: "2026-10-26"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-10-26T09:46:58Z"
last_seen: "2026-10-26T09:46:58Z"
occurrences: 4
files: ["src/controller/testsuite/test_job_live_gc.c"]
tags: ["test-only", "uaf", "gc"]
related: []
fix_commits: ["f780a166"]
---
# bug-1972: test_job_live_gc asserted destruction by reading freed memory
## Description
**Observed:**
`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`
(test_job_live_gc.c:169):
```
==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290
READ of size 8 at 0x5110000da290 thread T0
#0 job_get_refcount ../lib/jobj.c:715
#1 test_sweep_purges_expired_done_job test_job_live_gc.c:169
freed by:
#3 job_dereference ../lib/jobj.c:705
#5 job_live_gc_sweep ../controller/job.c:4777
```
**Expected:** a correct sweep dereferences the purged job down to refcount 0
and destroys it; the test must verify destruction without touching the freed
object.
**Impact:** controller suite aborts (killed by SIGABRT); the whole
`controller test` meson entry fails. Test-only — the production sweep
(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)
is correct.
**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier
runs all predate that commit).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.39 (host ldd)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: feature/license-resources @ 68eabd8a-dirty (introduced in d028c8c3)
- Deployment: n/a (unit test)
## Reproduction
1. `cd src/xce.build && ASAN_OPTIONS=detect_leaks=0 ./controller/controller_unit_test`
- Frequency: deterministic (4/4 runs, 2026-10-26)
- Evidence: /home/lchen/code/xce/src/testlogs.archive/xce.build/this-20261026-082318-rc1
## Root cause
The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs
AFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the
entire RTTI object (header + private + data in one calloc block). Reading the
refcount through the dangling `job_t*` is itself the use-after-free: the
stronger the sweep does its job, the more surely the test crashes. The same
pattern is absent from test_apply_pump.c's refcount test because there the
job is never destroyed.
## Fix
Move the refcount probe BEFORE the sweep and invert its meaning: assert
refcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,
then verify destruction by absence — `jlg_find_job(81001) == NULL` plus the
idempotent second sweep (both already present). `expired` is never dereferenced
after the sweep; a comment marks the last safe point.
## Verification
- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic
ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero
ASAN reports).
- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);
sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,
bug-free fixture staleness from fc33a5da's schema change, handled in the same
working tree).
- Live: n/a (test-only defect; production sweep unchanged).
## References
- Related: none

86
.wolf/bugs/bug-1973.md Normal file
View file

@ -0,0 +1,86 @@
---
id: "bug-1973"
status: "fixed"
status_updated: "2026-10-27"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-27T03:27:47Z"
last_seen: "2026-10-27T03:27:47Z"
occurrences: 1
files: ["src/controller/cluster.c", "src/controller/testsuite/test_config_resource_reconcile.c"]
tags: ["config", "reconcile", "E2", "external-license"]
related: ["bug-1854"]
fix_commits: ["2fcacac2"]
summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered"
---
# bug-1973: cluster_build_resources aborted at first EEXIST: new resource defs never registered
## Description
**Observed:**
Controller log on t1, after pushing a cluster.bin whose resources section ADDED
the new `extlic` definition (external-license-resources E2 Phase 1):
```
[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory
```
The xceesd daemon joined and pushed extlic values, but the controller had
no `extlic` registration at all (update_resval key_find → -ENOENT).
**Expected:** a config activation that adds a new resource definition must
register it in the running controller's resmgr, so daemon pushes land.
**Impact:** every newly-added resource definition was silently inert until
the controller process was restarted (boot-time registration path did run).
For E2 this meant the whole extres-daemon chain was dead on a live controller.
**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.42 (de-nix deployment closure)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: debug (build.dbg), deployed with patchelf interpreter+RUNPATH
- Code: feature/license-resources @ ba42fdd8-dirty
- Deployment: t1 single-node (controller+scheduler+apiserver+worker), xceesd extres daemon
## Reproduction
1. Controller running with a cluster.bin containing resources {slots, ...}
2. `xceadmin config build && xceadmin config push` with a bin that ADDS `extlic`
3. Start xceesd; it joins and pushes extlic values
4. Controller: `update_resval 'extlic': No such file or directory` (-ENOENT)
- Frequency: deterministic
- Evidence: controller log lines quoted above (t1, 2026-10-26 17:40)
## Root cause
`cluster_build_resources()` looped over the decoded config's resource list
calling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)
returns NULL with errno=EEXIST when the name is already registered — which is
the COMMON case at activation (boot already registered some definitions). The
old loop treated that rc as fatal and aborted the whole loop, so every
definition AFTER the first already-registered one was never registered.
## Fix
Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under
one write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;
RES_APPLY_CHANGED updates the registered object's definition fields
(valtype/ordtype/flags/interval) IN PLACE — none of them touches the
registration slot (idx) or accumulated accounting; mirrors
reconcile_config_queues' in-place refresh precedent (second live bug, see
bug-1974 — kept here as one loop). Negative rc still propagates.
## Verification
- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +
`test_reconcile_resources_hot_applies_changed_definition`
(src/controller/testsuite/test_config_resource_reconcile.c); RED before the
fix (lic1 unregistered / flags stayed 0x00), GREEN after.
- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in
sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at
free=1; release re-admitted it.
## References
- Related: bug-1854 (resourceApplyDef change-detection contract this loop is
built on), E2 spec .kiro/specs/external-license-resources/design.md §3.2

85
.wolf/bugs/bug-1974.md Normal file
View file

@ -0,0 +1,85 @@
---
id: "bug-1974"
status: "fixed"
status_updated: "2026-10-27"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-27T03:35:12Z"
last_seen: "2026-10-27T03:35:12Z"
occurrences: 1
files: ["src/controller/cluster.c", "src/controller/worker.c", "src/xceadmin/src/xcecommon/mapping.py"]
tags: ["config", "flags", "E2", "external-license"]
related: ["bug-1854", "bug-1973"]
fix_commits: ["2fcacac2"]
---
# bug-1974: config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM
## Description
**Observed:**
Controller log on t1 after re-pushing a corrected cluster.bin (extlic mapped
with `location: [external]` → flags SHARED|GLOBAL|EXTERNAL):
```
[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted
```
**Expected:** update_val_res' SHARED branch routes a host=="" resval for a
SHARED-flagged resource into c->sharedinst; with correct flags the push lands.
**Impact:** the extres daemon's pushes were permanently refused on a controller
whose registration predated the flag fix — the stale shape was frozen for the
process lifetime, so re-pushing a corrected config did not heal it.
**Found by:** E2 Phase 1 live E2E on t1.
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.42 (de-nix deployment closure)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: debug (build.dbg)
- Code: feature/license-resources @ ba42fdd8-dirty
- Deployment: t1 single-node; xceesd extres daemon
## Reproduction
1. Register extlic with flags=0 (see contributing factor below)
2. Push a corrected cluster.bin where extlic has SHARED|GLOBAL|EXTERNAL
3. xceesd pushes extlic=4 (host=="")
4. Controller: `update_resval 'extlic': Operation not permitted` (-EPERM)
- Frequency: deterministic (persists until controller restart or the bug-1973 fix)
- Evidence: controller log lines quoted above (t1, 2026-10-26 18:40/18:45)
## Root cause
Two compounding factors:
1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/
xceadmin-venv) carries a stale mapping.py with NO resourcemap handling
(83-line diff vs tree) — cluster bins it builds carry flags=0 for every
resource. Ops issue, fixed by building with the tree venv
(src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs
redeployment.
2. **Controller**: even with a corrected bin, `cluster_build_resources()`
treated an already-registered name as "nothing to do" (bug-1854's
RES_APPLY_CHANGED was reported but never applied), so the stale flags=0
shape survived every activation.
## Fix
`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the
registered object's definition fields (valtype/ordtype/flags/interval) in
place under the resmgr write lock — none of them touches the registration
slot (idx) or accumulated accounting; mirrors reconcile_config_queues'
"updated by configuration activation" in-place refresh precedent. (Same loop
as bug-1973's fix.) Producer side tracked separately as ops follow-up:
redeploy the xceadmin venv.
## Verification
- Test: `test_reconcile_resources_hot_applies_changed_definition`
(src/controller/testsuite/test_config_resource_reconcile.c); RED before
(flags stayed 0x00), GREEN after (0x1C, idx preserved).
- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push
landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN.
## References
- Related: bug-1854 (resourceApplyDef change-detection), bug-1973 (same loop's
EEXIST abort), worker.c update_val_res guard (the -EPERM surface)

89
.wolf/bugs/bug-1975.md Normal file
View file

@ -0,0 +1,89 @@
---
id: "bug-1975"
status: "fixed"
status_updated: "2026-10-27"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-27T03:40:05Z"
last_seen: "2026-10-27T03:40:05Z"
occurrences: 1
files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"]
tags: ["esd", "external-license", "reconnect", "E2"]
related: ["bug-1760", "bug-1973", "bug-1974"]
fix_commits: ["2fcacac2"]
summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
title: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
---
# bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)
## Description
**Observed:**
Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after
the controller was restarted:
```
(no socket() attempts; strace showed only collector forks + epoll_wait;
/proc/net socket table had no trace of the controller connection)
```
Last log line was the boot-time `esd full re-push: 0/1 source(s)` — no
further activity.
**Expected:** within one poll cycle of the controller's restart the daemon
must re-establish the connection and re-push its value (the new primary has
no record of the daemon or its value).
**Impact:** after any controller restart/failover with a stable license
value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL
by design) and no daemon re-push — every rusage[extlic] job PENDs forever
with NO_RESOURCE until the license value happens to change.
**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.42 (de-nix deployment closure)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: debug (build.dbg)
- Code: feature/license-resources @ ba42fdd8-dirty
- Deployment: t1 single-node; xceesd extres daemon
## Reproduction
1. xceesd running, pushing a value the collector keeps returning unchanged
2. Restart (or fail over) the controller
3. Observe the daemon: no reconnect attempts, pushes never resume
- Frequency: deterministic whenever the value is stable across the restart
- Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above
## Root cause
Three mechanisms conspired:
1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip
push) — with a stable value it pushes NOTHING.
2. A reconnect inside the client library only fires while INSIDE a push's
failover call — no pushes, no reconnect.
3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;
with no push attempt it never runs.
## Fix
`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`
(bug-1760's pure predicate — the transport's RDHUP handler sets
XCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.
The write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the
full re-push contract for the failover path.
## Verification
- Test: covered by the live contract (client-side predicate already unit
tested by bug-1760's suite); esd-side loop is integration-shaped (the
trigger is a transport callback), so live E2E is the proof.
- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
- Live: t1 — controller killed and restarted; esd logged
`connection broken -- forcing a full re-push to reconnect`, rotated to the
new primary, re-pushed; `xcecli resources distribution list` shows extlic=4
and a fresh rusage[extlic=3] job RUNS.
## References
- Related: bug-1760 (the pure broken-connection predicate this reuses),
E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller
failover items)

90
.wolf/bugs/bug-1976.md Normal file
View file

@ -0,0 +1,90 @@
---
id: "bug-1976"
status: "fixed"
status_updated: "2026-10-28"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-28T05:37:09Z"
last_seen: "2026-10-28T05:26:00Z"
occurrences: 1
files: ["src/controller/job.c", "src/controller/testsuite/test_dispatch_commit.c"]
tags: ["external-license-resources", "limits", "shared-resources", "dispatch", "e2-phase3"]
related: ["bug-1860"]
fix_commits: ["76ddd75f"]
summary: "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced"
---
# bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced
## Description
**Observed:**
E2 Phase 3 t1 E2E: 4 projA jobs with `-R "rusage[extlic=1]"` against a
per-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran
instead of the 4th PENDing with reason limit_global. Live gdb on the
scheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with
3 projA jobs RUN: the vres charge never landed.
**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job
re-admits it; projB unaffected (own per-project instance).
**Impact:** every scheduler-side global limit (the limit vres mechanism)
was advisory only — no quota was ever enforced, for ANY shared resource
carried via allocs_shared, since dispatch-time adoption overwrote the
scheduler's own charged accounts with a shared-stripped copy.
**Found by:** spec task (external-license-resources E2 Phase 3 E2E).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.39
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: debug (build.dbg)
- Code: feature/license-resources @ 76ddd75f (bug present since dispatch
commit was introduced, f4b1ba4c-era)
- Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries
in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh
## Reproduction
1. Push a config with a per-project limit (limits.yaml `extlic_share`,
cap 3.0) and an EXTERNAL resource extlic with ≥4 free.
2. Submit 4 jobs `-P projA -R "rusage[extlic=1]"`; observe all 4 RUN.
3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs.
- Frequency: deterministic.
- Evidence: this record + t1 E2E transcript in session (2026-10-28).
## Root cause
The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)
rebuilt `j->job.allocs` from the scheduler's dispatch payload but
migrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,
allocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`
was neither moved nor freed (a leak on top). That rebuilt record is
what `sched_job_update_fun()` re-sends to the scheduler as ground
truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,
core_events.c) rebuilds `job->accounts` from exactly that wire copy —
so `accounts_shared == NULL`, `sched_account_jobs` charges nothing to
the vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's
own commit (`sched_commit_decision`, scheduler.c:1806-1812) built the
accounts correctly — the controller's "ground truth" clobbered it.
## Fix
job.c dispatch-time commit: migrate the shared records alongside the
host records — `MOVE_PTR(j->job.allocs->allocs_shared,
alloc->allocs_shared); j->job.allocs->nres_shared =
alloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the
caller frees the payload; also closes the leak).
## Verification
- Test: `test_dc_success_migrates_shared_allocs`
(src/controller/testsuite/test_dispatch_commit.c). RED before the fix
("Expected 2 Was 0" on nres_shared), GREEN after.
- Suites: full meson test 136/136 OK (build.dbg), including controller
unit suite 1361/0.
- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill
one RUN job → pending projA re-admitted, external free pool
(fake_lic) enforced independently; all daemons healthy throughout.
## References
- Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the
stripped copy authoritative), external-license-resources design.md
§3.5/§3.6 (per-app→per-project terminology).

85
.wolf/bugs/bug-1977.md Normal file
View file

@ -0,0 +1,85 @@
---
id: "bug-1977"
status: "fixed"
status_updated: "2026-10-28"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-10-28T11:56:58Z"
last_seen: "2026-10-28T11:56:58Z"
occurrences: 1
files: ["src/controller/testsuite/test_config_resource_reconcile.c"]
tags: ["testsuite", "asan"]
related: []
fix_commits: ["8c1995c2"]
---
# bug-1977: crr_build_config callers' stack arrays smaller than builder -- ASAN stack overflow
## Description
**Observed:**
```
ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc...
WRITE of size 8 at 0x7ffc... thread T0
#0 crr_build_config testsuite/test_config_resource_reconcile.c:126 (dists[3] = &d4)
#1 test_reconcile_resources_registers_and_seeds_shared_pools testsuite/test_config_resource_reconcile.c:161
```
Every test in the file that calls `crr_build_config` crashes the same way;
the full controller suite reports 137/137 FAIL (meson reports each fixture
crash as its own failure).
**Expected:** all four callers pass arrays large enough for the builder's
fixtures; the suite stays green.
**Impact:** the whole controller testsuite was unrunnable under ASAN since the
external-license-resources test fixtures landed (commit 2fcacac2) -- any
regression in the suite was invisible behind the deterministic crash.
**Found by:** ASAN -- full regression build for the license-resource feature
(`src/tools/xce-build.sh controller`, profile=asan).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: host ldd reports glibc 2.39
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: feature/license-resources @ 30c51ea1-dirty (introduced by 2fcacac2)
- Deployment: n/a (unit testsuite only)
## Reproduction
1. `src/tools/xce-build.sh controller` (asan profile) -- meson test crashes:
`controller - xce:controller test` FAIL with the stack-buffer-overflow above.
- Frequency: deterministic
- Evidence: src/testlogs.archive/xce.build/this-20261028-114408-rc1/meson-logs/testlog.txt (bug-1971 failure-preservation mechanism)
## Root cause
`crr_build_config` (test_config_resource_reconcile.c:70) registers FIVE
resources (lic1/hostres/dyn1/extlic/slots -> writes `resources[4]`) and FOUR
distributions (writes `dists[3]`) and FIVE vals (writes `vals[4]`). The four
caller test functions declared only
`resources[4] / dists[3] / vals[4]`, so the builder's writes to index 4 / 3 /
4 overflowed the callers' stack arrays. When the extlic E2 tests were added to
the builder's fixture set (2fcacac2), none of the caller declarations were
resized. (Earlier builds of this branch compiled only a worker-focused subset
-- `test_config_resource_reconcile` was not linked -- so the overflow went
unnoticed until the full controller suite was built.)
## Fix
Resize all four caller arrays to match the builder: `resources[5]`,
`dists[4]`, `vals[5]`, each with a bug-1977 comment pointing at the sizing
contract. Files: src/controller/testsuite/test_config_resource_reconcile.c
(test_reconcile_resources_registers_and_seeds_shared_pools,
test_reconcile_resources_registers_new_defs_past_eexist,
test_reconcile_resources_hot_applies_changed_definition,
test_reconcile_resources_is_idempotent_and_hot_applies).
## Verification
- Test: whole-file suite `controller - xce:controller test` (meson): RED
137/137 before the fix, GREEN after.
- Suites: `src/tools/xce-build.sh controller` profile=asan: 6/6 OK
(sched push, sched worker change, apiserver push, host runusage, no-fork
rule, controller 91s).
- Live: n/a (testsuite-only defect; no production code touched).
## References
- Related: none

110
.wolf/bugs/bug-1978.md Normal file
View file

@ -0,0 +1,110 @@
---
id: "bug-1978"
status: "fixed"
status_updated: "2026-10-28"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-28T12:47:59Z"
last_seen: "2026-10-28T12:47:59Z"
occurrences: 3
files: ["src/esd/src/esd.c"]
tags: ["esd", "reconnect", "failover"]
related: ["bug-1760"]
fix_commits: ["72301fd5"]
---
# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever
## Description
**Observed:**
```
t2 /tmp/xceesd-d.log:
[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)
[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s)
[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)
... (every 10s poll, forever)
```
After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on
every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,
the poll loop runs, the collector script fires every 10s, the status file
refreshes with `stale: false` -- only pushes_fail climbs.
**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and
pushes resume (pushes_ok advances, `xce resources distribution list` shows
extlic).
**Impact:** the entire external-license value chain dies at the first
controller restart: extlic never reaches c->sharedinst, the scheduler never
hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever
(admission sees free=0). Found during the license-resource regression: the
regression config push (gen 19) succeeded but `resources distribution list`
showed NO extlic row -- daemon健康、值链断。
**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline
evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller
restarts in /opt/xce/logs/xce_controller.log, zero extres joins after
10:58, and esd strace showing zero network sends (debounce suppressed -- no
rotate ever triggered again).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.39 (t2 host)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build) -- failing binary was the de-nix deploy
/opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build)
- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de
- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1
## Reproduction
1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`);
observe pushes_ok advance (the boot join works).
2. Restart the t1 controller (or kill it long enough for the connection to
drop and transparently reconnect).
3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`.
Controller log logs no new extres join.
- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts
+ the Oct-26 original "unknown role 5" cousin)
- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log,
t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing)
## Root cause
esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect
sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a
re-JOIN: the fresh controller endpoint is created by handle_connection()
(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets
priv->role = HostRole_extres (join.c handle_join_). The transparent
reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket
but never re-runs the role join, and the client library's rejoin hook
(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the
un-joined endpoint's resval update falls through BOTH branches of
do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and
returns rc = -EPERM == -1 forever. (The worker solved this exact shape with
the needs_rejoin contract, controller_connector.c; the scheduler likewise.
The esd -- written later -- re-used the flag machinery but dropped the
re-join itself.)
## Fix
src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and
g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing
`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread
(outside the library's failover lock -- the xcecd_on_reconnect no-I/O
constraint), and only then does the full re-push. Join failure keeps the
flag set (next tick retries -- the worker's self-healing cadence).
## Verification
- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push
+ test_reconnect_flags_recover_from_any_previous_state
(src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED
with the old esd.c (compile error: g_needs_rejoin did not exist -- the
re-join state was not represented at all), GREEN with the fix: 13/13.
- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build
rc=0.
- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2
(restart controller, observe "esd: re-joined after reconnect" + pushes_ok
advancing + extlic row in `resources distribution list`).
## References
- Related: bug-1760 (the connection-broken predicate that makes the
debounce-stall recovery work), worker needs_rejoin precedent
(controller_connector.c), bug-1522 (register-after-start ordering).

3
scripts/check-no-fork.py
View file

@ -21,7 +21,8 @@ import os
import re
import sys
COVERED = ["src/controller", "src/scheduler", "src/lib", "src/worker"]
COVERED = ["src/controller", "src/scheduler", "src/lib", "src/worker",
"src/esd"]
EXCLUDED_DIRS = {"testsuite", "test", "tests"}
# Fork-family calls. uv_spawn() is libuv's fork()+exec on Linux.
BANNED = re.compile(

32
src/configs/limits.yaml
View file

@ -93,3 +93,35 @@ limits:
resname: mem
value: 0.2
percent: true
# external-license-resources (E2 Phase 3): strategy-layer quotas on the
# externally reported extlic pool. These ride the EXISTING limit vres
# mechanism -- no new daemon, no new admission rule -- on top of Phase
# 1's underlying-resource check (total >= uncomit + wanted). Two
# distribution shapes:
#
# 1. per-project shares: projA and projB each hold their own
# "limit=extlic_share/p=<proj>" usage instance under a 3.0 cap
# (independent accounting per project; the scope lists who may
# match at all). literal members here; named project groups are
# project-config territory (out of this spec's scope).
- name: extlic_share
projects:
members:
- projA
- projB
resname: extlic
value: 3.0
per_project: true
# 2. per-queue distribution: queues each get their own
# "limit=extlic_qdist/q=<queue>" instance under a 4.0 cap -- how a
# site partitions one license pool across queues.
- name: extlic_qdist
queues:
members:
- normal
- night
resname: extlic
value: 4.0
per_queue: true

37
src/configs/resources.yaml
View file

@ -5,6 +5,7 @@ resources:
type: numeric
order: decrease
consumable: true
builtin: true
description: "job slot count (built-in)"
# built-in: dynamic memory load index reported by workers
@ -12,6 +13,7 @@ resources:
type: numeric
interval: 60
consumable: true
builtin: true
order: decrease
description: "available memory in bytes (built-in dynamic load index; CLI display converts to a human-readable unit)"
@ -21,6 +23,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "1-minute load average (built-in dynamic load index)"
@ -28,6 +31,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "15-minute load average (built-in dynamic load index)"
@ -35,6 +39,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "15-second load average (built-in dynamic load index)"
@ -42,6 +47,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "cpu utilization"
@ -49,6 +55,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "page ratio"
@ -56,6 +63,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "login count"
@ -63,13 +71,15 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "idle time"
- name: swp
type: numeric
interval: 60
consumable: true
builtin: true
order: decrease
description: "swap size"
@ -77,13 +87,15 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: increase
description: "io rate"
- name: tmp
type: numeric
interval: 60
consumable: true
builtin: true
order: decrease
description: "available tmp disk space in bytes (built-in dynamic load index; CLI display converts to a human-readable unit)"
@ -91,34 +103,42 @@ resources:
# controller/worker.c add_hostattrs_resvals()
- name: type
type: string
builtin: true
description: "host OS/arch type (built-in host attribute)"
- name: model
type: string
builtin: true
description: "host hardware model (built-in host attribute)"
- name: cpuf
type: numeric
builtin: true
description: "CPU speed factor (built-in host attribute)"
- name: maxcpus
type: numeric
builtin: true
description: "max CPU count (built-in host attribute)"
- name: maxmem
type: numeric
builtin: true
description: "max memory in bytes (built-in host attribute; CLI display converts to a human-readable unit)"
- name: maxswap
type: numeric
builtin: true
description: "max swap in bytes (built-in host attribute; CLI display converts to a human-readable unit)"
- name: maxtmp
type: numeric
builtin: true
description: "max tmp disk space in bytes (built-in host attribute; CLI display converts to a human-readable unit)"
- name: ndisks
type: numeric
builtin: true
description: "disk count (built-in host attribute)"
# dynamic hostbased resource, external load index
@ -269,3 +289,16 @@ resources:
- name: gstring
description: string resource
type: string
# external-license-resources: cluster-wide license pool reported by
# the extres daemon (xceesd) -- config declares only the name/shape,
# the free counts come from the daemon's pushes
- name: extlic
description: externally reported license pool
type: numeric
interval: 0
consumable: true
order: decrease
resourcemap:
- location:
- external

1
src/configs/schema/resources.schema.json
View file

@ -12,6 +12,7 @@
"order": { "type": "string", "enum": ["increase", "decrease", "na"] },
"consumable": { "type": "boolean" },
"release": { "type": "boolean" },
"builtin": { "type": "boolean" },
"description": { "type": "string" },
"resourcemap": {
"type": "array",

53
src/controller/cluster.c
View file

@ -105,6 +105,9 @@ static void handle_endpoint_error(endpoint_t* ep)
case HostRole_apiserver:
handle_apiserver_leave(c, ep);
break;
case HostRole_extres:
handle_extres_leave(c, ep);
break;
default:
ERROR("unknown role %d for endpoint %p", priv->role, ep);
break;
@ -1535,21 +1538,50 @@ int cluster_build_config(cluster_t* c)
int cluster_build_resources(cluster_t* c)
{
int i;
int rc = 0;
if (c->myconf == NULL) {
if ((c->myconf == NULL) || (c->myconf->cluster_config == NULL)) {
return -EINVAL;
}
for (i = 0; i < c->myconf->cluster_config->nres; i++) {
xce_resource_t* res;
int arc;
res = makeResource(c->resmgr, c->myconf->cluster_config->resources[i]);
if (res == NULL) {
return -errno;
arc = resourceApplyDef(c->resmgr,
c->myconf->cluster_config->resources[i]);
if (arc == RES_APPLY_CHANGED) {
/* The registered object was created from a DIFFERENT generation
* of the definition (bug-1854's change-detection contract).
* Keeping the stale shape would freeze the boot-time behavior
* for the daemon's lifetime -- the live sighting: extlic first
* registered with flags=0 (the building tool dropped its
* resourcemap), and every later activation kept refusing the
* extres daemon's pushes with -EPERM because
* update_val_res()'s sharedinst route gates on XCE_RESF_SHARED.
* Update the definition fields IN PLACE: none of them touches
* the registration slot (idx) or accumulated accounting, and
* this mirrors reconcile_config_queues' "updated by
* configuration activation" in-place refresh precedent. */
xce_resource_t* cur = NULL;
xce_resource_t* def = c->myconf->cluster_config->resources[i];
lock(c->resmgr, RTTI_LOCK_WRITE);
if (key_find(c->resmgr, (void*)def->name, (void**)&cur) > 0
&& cur != NULL) {
cur->valtype = def->valtype;
cur->ordtype = def->ordtype;
cur->flags = def->flags;
cur->interval = def->interval;
INFO("resource %s definition updated by configuration "
"activation (flags=0x%x)", def->name, def->flags);
}
unlock(c->resmgr);
} else if (arc < 0) {
rc = arc;
}
}
return 0;
return rc;
}
static void resval_obj_killer(void* arg);
@ -1598,6 +1630,15 @@ static void seed_sharedinst_from_config(cluster_t* c)
|| (r->flags & XCE_RESF_GLOBAL) == 0) {
continue;
}
/* external-license-resources: an EXTERNAL pool's values come only
* from the extres daemon's free-count pushes -- seeding a static
* config value here would mask "daemon not yet up" as a phantom
* capacity (admitting jobs against licenses that do not exist).
* Until the daemon joins, get_resvals reads 0 and jobs PEND with a
* NO_RESOURCE detail naming the resource -- the honest signal. */
if (r->flags & XCE_RESF_EXTERNAL) {
continue;
}
if (r->ndistributions < 1 || r->distributions == NULL
|| r->distributions[0] == NULL || r->distributions[0]->value == NULL) {
continue;

78
src/controller/extres.c Normal file
View file

@ -0,0 +1,78 @@
/*
* extres.c — connected extres daemons (HostRole_extres joins)
*
* external-license-resources Phase 1: the external-resources daemon
* (xceesd) joins the controller with HostRole_extres and pushes
* cluster-level (SHARED) resource values through the ordinary resval
* update pipeline (opupd.c do_update_resvals). Mirrors the apiserver
* join/leave registration in apiserver.c: endpoint + name only, no
* workermgr slot — the daemon is not a schedulable host. The controller
* never pushes TO the daemon; the daemon re-pushes its full value set on
* every (re)connect, so nothing here needs replay/WAL treatment.
*/
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include "cluster.h"
#include "xstring.h"
#include "xce/cluster_def.h"
int add_extres(cluster_t* c, char* name, endpoint_t* ep)
{
int i;
for (i = 0; i < N_EXTRES_MAX; i++) {
if ((c->extresds[i].name != NULL)
&& (strcmp(c->extresds[i].name, name) == 0)) {
/* re-join (reconnect or failover re-join): rebind in place */
c->extresds[i].ep = ep;
return i;
}
}
for (i = 0; i < N_EXTRES_MAX; i++) {
if (c->extresds[i].name == NULL) {
c->extresds[i].name = makeString(name);
if (c->extresds[i].name == NULL) {
return -ENOMEM;
}
c->extresds[i].ep = ep;
return i;
}
}
ERROR("too many extres daemons (max %d); join from %s refused",
N_EXTRES_MAX, name);
return -ENOSPC;
}
int handle_extres_leave(cluster_t* c, endpoint_t* ep)
{
ep_private_t* priv;
int i;
if (ep == NULL) {
return -EINVAL;
}
priv = (ep_private_t*)ep->data;
if (priv == NULL) {
return -EINVAL;
}
i = priv->u.idx;
if ((i < 0) || (i >= N_EXTRES_MAX)) {
SET_ERROR(EINVAL);
return -errno;
}
/* Only clear if it is still our binding — a re-join may have rebound
* the slot to a new endpoint before this old one's close event ran. */
if (c->extresds[i].ep == ep) {
c->extresds[i].ep = NULL;
}
return 0;
}

4
src/controller/include/cluster.h
View file

@ -439,5 +439,9 @@ extern int schedadmin_signal_disconnect(void);
extern int handle_apiserver_leave(cluster_t*, endpoint_t*);
extern int add_apiserver(cluster_t*, char*, endpoint_t*);
extern int apiserver_init(cluster_t*);
/* external-license-resources Phase 1: HostRole_extres daemon registration
* (extres.c) — endpoint + name only, no workermgr slot. */
extern int handle_extres_leave(cluster_t*, endpoint_t*);
extern int add_extres(cluster_t*, char*, endpoint_t*);
#endif

13
src/controller/job.c
View file

@ -1936,6 +1936,19 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
MOVE_PTR(j->job.allocs->allocs, allocs);
alloc->allocs = NULL;
/* external-license-resources Phase 3: the scheduler's dispatch
* payload also carries the shared-pool records (allocs_shared -- the
* per-project/per-queue limit vres charge among them). This rebuild
* is the record sched_job_update_fun() re-sends to the scheduler as
* ground truth (the JOB_UPDATE adoption there rebuilds job->accounts
* from it), so dropping the shared records here starves the
* scheduler's sharedInUse accounting and every global limit admits
* forever. Migrate them (MOVE, not copy -- the caller frees the
* payload) exactly like the per-host records above. */
MOVE_PTR(j->job.allocs->allocs_shared, alloc->allocs_shared);
j->job.allocs->nres_shared = alloc->nres_shared;
alloc->nres_shared = 0;
j->job.state = JOB_STAT_RUN;
/* bug-1697: start_time is now stamped HERE, synchronously alongside

40
src/controller/join.c
View file

@ -248,6 +248,43 @@ static int handle_apiserver_join(cluster_t* c, endpoint_t* ep,
return 0;
}
/*
* handle_extres_join: register the external-resources daemon (xceesd,
* external-license-resources Phase 1). Mirrors handle_apiserver_join's
* lightweight shape — endpoint + priv->u.idx only, no workermgr slot — and
* unlike the apiserver there is no on-join state push: the daemon is the
* pusher (it re-sends its full shared-value set on every reconnect).
*/
static int handle_extres_join(cluster_t* c, endpoint_t* ep,
xce_request_t* request)
{
xce_reply_t reply;
xce_peerinfo_t* from = request->from;
int rc = 0;
TRACE("%s extres daemon join, ep %p", __func__, ep);
init_reply_by_req(&reply, request);
rc = add_extres(c, from->name, ep);
if (rc >= 0) {
ep_private_t* priv = (ep_private_t*)ep->data;
priv->u.idx = rc;
rc = 0;
}
reply.status = rc;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
if (rc < 0) {
ERROR("extres daemon %s join failed, rc %d", from->name, rc);
return rc;
}
return 0;
}
static int handle_join_(cluster_t* c, endpoint_t* ep, xce_request_t* request)
{
ep_private_t* priv;
@ -319,6 +356,9 @@ static int handle_join_(cluster_t* c, endpoint_t* ep, xce_request_t* request)
case HostRole_apiserver:
rc = handle_apiserver_join(c, ep, request);
break;
case HostRole_extres:
rc = handle_extres_join(c, ep, request);
break;
default:
rc = -E_XCE_BAD_ROLE;
ERROR("unknown role %d from host %s", role, from->name);

3
src/controller/meson.build
View file

@ -5,6 +5,7 @@ controller_src = [
'ctrlstatus.c',
'dataex.c',
'egress.c',
'extres.c',
'ha.c',
'ha_convergence.c',
'job.c',
@ -90,6 +91,7 @@ controller_unit_test_src = [
'testsuite/test_ctrl_metrics.c',
'testsuite/test_config_repl.c',
'testsuite/test_worker_join.c',
'testsuite/test_extres_resvals.c',
'testsuite/test_ha_startup.c',
'testsuite/test_ha_election.c',
'testsuite/test_ha_elect_ctx_lifecycle.c',
@ -205,6 +207,7 @@ controller_unit_test_src = [
'config_repl.c',
'slowlane.c',
'cluster.c',
'extres.c',
'resync.c',
]

33
src/controller/opupd.c
View file

@ -23,7 +23,38 @@ static int do_update_resvals(cluster_t* c, endpoint_t* ep,
if (priv != NULL) {
int idx = priv->u.idx;
if ((priv->role == HostRole_worker) && (idx >= 0)) {
if (priv->role == HostRole_extres) {
/* external-license-resources Phase 1: the extres daemon
* (xceesd) pushes cluster-level values with NO workermgr slot
* -- w==NULL routes every entry through update_val_res()'s
* "SHARED 即收" guard (shared values land in c->sharedinst;
* host-based values are refused -EPERM per entry). The reply
* status is 0 for any syntactically valid list: per-entry
* refusals are logged by log_update_resval_failure() and must
* not fail the whole update (mirrors the worker path's
* per-item convention). */
xce_update_data_t* updatemsg = request->body.updatemsg;
xce_resval_list_t* resvals = updatemsg->data.resvals;
xce_resval_list_t applied = { .count = 0, .vals = NULL };
if ((resvals != NULL) && (resvals->count > 0)) {
applied.vals = (xce_resval_t**)calloc(
(size_t)resvals->count, sizeof(xce_resval_t*));
}
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_RESOURCE,
HOOK_BEFORE_EVENT, HOOK_DETAIL_RESOURCE_UPDATE, c,
resvals);
update_resval_list(c, NULL, resvals,
(applied.vals != NULL) ? &applied : NULL);
if (applied.count > 0) {
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_RESOURCE,
HOOK_AFTER_EVENT, HOOK_DETAIL_RESOURCE_UPDATE, c,
&applied);
}
free(applied.vals);
rc = 0;
} else if ((priv->role == HostRole_worker) && (idx >= 0)) {
worker_t* w = NULL;
rc = vget(c->workermgr, idx, (void**)&w);

2
src/controller/testsuite/controller_test.c
View file

@ -40,6 +40,7 @@ extern void test_p3_manual_failover(void);
extern void test_ctrl_metrics(void);
extern void test_config_repl(void);
extern void test_worker_join(void);
extern void test_extres_resvals(void);
extern void test_ha_startup(void);
extern void test_ha_election(void);
extern void test_ha_elect_ctx_lifecycle(void);
@ -256,6 +257,7 @@ int main(int argc, char* argv[])
TEST(test_ctrl_metrics);
TEST(test_config_repl);
TEST(test_worker_join);
TEST(test_extres_resvals);
TEST(test_ha_startup);
TEST(test_ha_election);
TEST(test_ha_elect_ctx_lifecycle);

200
src/controller/testsuite/test_config_resource_reconcile.c
View file

@ -61,13 +61,18 @@ static xce_val_t* crr_shared_val(cluster_t* c, int idx)
* res0 "slots" -- built-in hostbased (no resourcemap): registered, never seeded
* res1 "lic1" -- SHARED|GLOBAL, distribution value 100: seeded
* res2 "hostres"-- SHARED but NOT GLOBAL (per-host shape): NOT seeded
* res3 "dyn1" -- SHARED|GLOBAL, distribution WITHOUT a value: NOT seeded */
* res3 "dyn1" -- SHARED|GLOBAL, distribution WITHOUT a value: NOT seeded
* res4 "extlic" -- SHARED|GLOBAL|EXTERNAL, distribution value 7: NOT
* seeded (external-license-resources: an EXTERNAL pool's
* values come only from the extres daemon's free-count
* pushes; seeding a static config value would mask "daemon
* not yet up" as a phantom capacity) */
static void crr_build_config(cluster_config_t* cfg, xce_resource_t** resources,
res_distribution_t** dists, xce_val_t** vals)
{
static xce_resource_t r0, r1, r2, r3;
static res_distribution_t d1, d2, d3;
static xce_val_t v1, v3;
static xce_resource_t r0, r1, r2, r3, r4;
static res_distribution_t d1, d2, d3, d4;
static xce_val_t v1, v3, v4;
memset(&r0, 0, sizeof(r0));
r0.name = (char*)"slots";
@ -108,11 +113,25 @@ static void crr_build_config(cluster_config_t* cfg, xce_resource_t** resources,
dists[2] = &d3;
vals[3] = NULL;
memset(&r4, 0, sizeof(r4));
r4.name = (char*)"extlic";
r4.flags = XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL;
memset(&d4, 0, sizeof(d4));
memset(&v4, 0, sizeof(v4));
v4.which = Value_u_fvalue;
v4.u.fvalue = 7.0;
d4.value = &v4;
r4.ndistributions = 1;
r4.distributions = (res_distribution_t**)&dists[3];
dists[3] = &d4;
vals[4] = &v4;
resources[0] = &r0;
resources[1] = &r1;
resources[2] = &r2;
resources[3] = &r3;
cfg->nres = 4;
resources[4] = &r4;
cfg->nres = 5;
cfg->resources = resources;
}
@ -123,9 +142,13 @@ static void test_reconcile_resources_registers_and_seeds_shared_pools(void)
cluster_t c;
myconf_t conf;
cluster_config_t cfg;
xce_resource_t* resources[4];
res_distribution_t* dists[3];
xce_val_t* vals[4];
/* bug-1977: the builder registers FIVE resources (indices 0..4) and
* FOUR distributions (indices 0..3) -- the caller arrays must match,
* or the builder's resources[4]/dists[3]/vals[4] writes overflow these
* stacks (ASAN stack-buffer-overflow, deterministic). */
xce_resource_t* resources[5];
res_distribution_t* dists[4];
xce_val_t* vals[5];
xce_resource_t* r;
if (makeString((char*)"crr-probe") == NULL)
@ -181,6 +204,157 @@ static void test_reconcile_resources_registers_and_seeds_shared_pools(void)
xce_resource_t* rb = crr_find(&c, "slots");
TEST_ASSERT_NULL(crr_shared_val(&c, rb->idx));
/* EXTERNAL: daemon-reported free values only -- never seeded from the
* config, even with a valued distribution (external-license-resources). */
xce_resource_t* re = crr_find(&c, "extlic");
TEST_ASSERT_NOT_NULL(re);
TEST_ASSERT_NULL_MESSAGE(crr_shared_val(&c, re->idx),
"an EXTERNAL shared pool must not seed from its config value -- "
"only the extres daemon reports its free counts");
xce_hook_fini(&c.hooksys);
dereference(c.resmgr);
zarray_fini(&c.sharedinst);
}
/* Live-activation partial registration: the resmgr already holds SOME of the
* config's resources (registered at boot from the previous generation) but
* not all (the activation adds a NEW definition -- the external-license-
* resources scenario: gen-12 config without extlic, gen-13 activation adds
* it while the 33 boot resources EEXIST). makeResource() refuses an existing
* name with NULL+EEXIST, so cluster_build_resources() must NOT abort the
* loop at the first EEXIST -- the new definition must still register and
* its shared pool (lic1 here) must still seed. Regression for the live
* sighting: every activation after boot bailed at "slots" (i=0) so NO new
* resource definition ever reached a running controller. */
static void test_reconcile_resources_registers_new_defs_past_eexist(void)
{
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 16, .delta = 8 };
cluster_t c;
myconf_t conf;
cluster_config_t cfg;
/* bug-1977: the builder writes resources[0..4]/dists[0..3]/vals[0..4] --
* caller arrays must match its sizing. */
xce_resource_t* resources[5];
res_distribution_t* dists[4];
xce_val_t* vals[5];
xce_resource_t* r;
xce_val_t* v;
if (makeString((char*)"crr-probe") == NULL)
xstring_table_init(64);
memset(&c, 0, sizeof(c));
memset(&conf, 0, sizeof(conf));
TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX));
c.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(c.resmgr);
zarray_init(&c.sharedinst, 16,
(zarray_item_killer_t)crr_val_killer);
crr_build_config(&cfg, resources, dists, vals);
conf.cluster_config = &cfg;
c.myconf = &conf;
/* Boot path: register ONLY the first definition (slots) -- a smaller
* previous generation. */
{
xce_resource_t* boot = makeResource(c.resmgr, resources[0]);
TEST_ASSERT_NOT_NULL(boot);
}
/* Activation: the full new config (slots EEXISTS, lic1 is new). */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, reconcile_config_resources(&c),
"an activation must tolerate EEXIST on pre-registered resources "
"and keep registering the rest");
r = crr_find(&c, "lic1");
TEST_ASSERT_NOT_NULL_MESSAGE(r,
"the NEW definition must register even though an earlier resource "
"in the config already existed");
v = crr_shared_val(&c, r->idx);
TEST_ASSERT_NOT_NULL_MESSAGE(v,
"the new definition's static shared pool must seed on activation");
TEST_ASSERT_EQUAL_FLOAT(100.0, v->u.fvalue);
/* the pre-existing runtime object must be kept (idempotent pass) */
{
xce_resource_t* slots = crr_find(&c, "slots");
TEST_ASSERT_NOT_NULL(slots);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, slots->idx,
"the boot-registered object keeps its resmgr slot");
}
xce_hook_fini(&c.hooksys);
dereference(c.resmgr);
zarray_fini(&c.sharedinst);
}
/* Hot-apply a CHANGED definition: the resmgr already holds a resource with
* one definition (registered at boot from the previous generation), and the
* activation brings a CHANGED one (flags changed -- the external-license-
* resources live sighting: extlic was first registered from a config built
* WITHOUT the `location: [external]` resourcemap, so its flags were 0; the
* corrected config carries SHARED|GLOBAL|EXTERNAL). Keeping the stale object
* would leave the daemon's pushes refused -EPERM forever (update_val_res
* gates the sharedinst route on XCE_RESF_SHARED) until a manual controller
* restart. The activation must update the definition fields in place --
* idx (runtime state) preserved -- mirroring reconcile_config_queues'
* in-place refresh contract. */
static void test_reconcile_resources_hot_applies_changed_definition(void)
{
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 16, .delta = 8 };
cluster_t c;
myconf_t conf;
cluster_config_t cfg;
/* bug-1977: the builder writes resources[0..4]/dists[0..3]/vals[0..4] --
* caller arrays must match its sizing. */
xce_resource_t* resources[5];
res_distribution_t* dists[4];
xce_val_t* vals[5];
xce_resource_t* r;
static xce_resource_t old_extlic;
if (makeString((char*)"crr-probe") == NULL)
xstring_table_init(64);
memset(&c, 0, sizeof(c));
memset(&conf, 0, sizeof(conf));
TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX));
c.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(c.resmgr);
zarray_init(&c.sharedinst, 16,
(zarray_item_killer_t)crr_val_killer);
/* Boot path: register "extlic" WITHOUT the shared/external flags -- the
* stale gen-13 shape (resourcemap dropped by the building tool). */
memset(&old_extlic, 0, sizeof(old_extlic));
old_extlic.name = (char*)"extlic";
old_extlic.flags = 0;
TEST_ASSERT_NOT_NULL(makeResource(c.resmgr, &old_extlic));
/* Activation: the corrected config where extlic is
* SHARED|GLOBAL|EXTERNAL (r4 in the shared builder). */
crr_build_config(&cfg, resources, dists, vals);
conf.cluster_config = &cfg;
c.myconf = &conf;
TEST_ASSERT_EQUAL_INT_MESSAGE(0, reconcile_config_resources(&c),
"an activation over a stale definition must succeed");
r = crr_find(&c, "extlic");
TEST_ASSERT_NOT_NULL(r);
TEST_ASSERT_EQUAL_HEX32_MESSAGE(
XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL, r->flags,
"the activation must hot-apply the changed flags in place -- a "
"stale flags=0 object would refuse the daemon's pushes forever");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, r->idx,
"the runtime idx (registration slot) must be preserved by the "
"in-place update");
xce_hook_fini(&c.hooksys);
dereference(c.resmgr);
zarray_fini(&c.sharedinst);
@ -197,9 +371,11 @@ static void test_reconcile_resources_is_idempotent_and_hot_applies(void)
cluster_t c;
myconf_t conf;
cluster_config_t cfg;
xce_resource_t* resources[4];
res_distribution_t* dists[3];
xce_val_t* vals[4];
/* bug-1977: the builder writes resources[0..4]/dists[0..3]/vals[0..4] --
* caller arrays must match its sizing. */
xce_resource_t* resources[5];
res_distribution_t* dists[4];
xce_val_t* vals[5];
xce_resource_t* r;
xce_val_t* v;
@ -246,5 +422,7 @@ void test_config_resource_reconcile(void)
{
UnitySetTestFile(__FILE__);
RUN_TEST(test_reconcile_resources_registers_and_seeds_shared_pools);
RUN_TEST(test_reconcile_resources_registers_new_defs_past_eexist);
RUN_TEST(test_reconcile_resources_hot_applies_changed_definition);
RUN_TEST(test_reconcile_resources_is_idempotent_and_hot_applies);
}

73
src/controller/testsuite/test_dispatch_commit.c
View file

@ -393,6 +393,78 @@ static void test_dc_success_notifies_job_update_hook_exactly_once(void)
printf("test_dc_success_notifies_job_update_hook_exactly_once: PASS\n");
}
#define DC_JOBID_SHARED 91006ULL
/* external-license-resources Phase 3 (vres charge) RED/GREEN proof: the
* scheduler's dispatch payload (sched_finalize_job_alloc_, alloc_runner.c)
* carries BOTH the per-host records AND the shared-pool records
* (allocs_shared -- the vres instance "limit=extlic_share/p=projA" charge
* among them). The controller's dispatch-time commit (job.c) rebuilds
* j->job.allocs from that payload, and that rebuilt record is what
* sched_job_update_fun() re-sends to the scheduler as ground truth
* (bug-1860 adoption in the scheduler's JOB_UPDATE handler rebuilds
* job->accounts from it) -- so if the rebuild drops allocs_shared, the
* scheduler's accounts lose every shared record, sharedInUse never
* accumulates the vres charge, and unsatisfy_ admits past the limit
* forever.
*
* RED against the pre-fix code: j->job.allocs->nres_shared stays 0 and
* allocs_shared stays NULL (the rebuild MOVE_PTRs only allocs->allocs).
* GREEN after the fix: the shared records are migrated intact (count,
* names, values), and the payload's own shared pointer is moved out (no
* double free). */
static void test_dc_success_migrates_shared_allocs(void)
{
job_t* j;
xce_job_alloc_t* alloc;
xce_alloc_t** shared;
int rc;
dc_setup();
j = dc_seed_ready(DC_JOBID_SHARED, 0, "dq", "wkr-dc-shared");
alloc = make_job_alloc(DC_JOBID_SHARED, 1,
make_host_allocs(1,
make_host_alloc((char*)"wkr-dc-shared", 1,
make_allocs(1,
make_alloc((char*)"slots", NULL, 0, 1.0f, 0)))));
TEST_ASSERT_NOT_NULL(alloc);
shared = make_allocs(2,
make_alloc((char*)"limit=extlic_share/p=projA", NULL, 0, 1.0f, 0),
make_alloc((char*)"extlic", NULL, 0, 1.0f, 0));
TEST_ASSERT_NOT_NULL(shared);
alloc->allocs_shared = shared;
alloc->nres_shared = 2;
rc = dispatch_a_job_test(&g_dc, NULL, alloc);
TEST_ASSERT_TRUE_MESSAGE(rc >= 0,
"a dispatch to a live, no-iomgr worker endpoint must succeed");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state);
TEST_ASSERT_NOT_NULL(j->job.allocs);
TEST_ASSERT_EQUAL_INT_MESSAGE(2, j->job.allocs->nres_shared,
"the dispatch-time rebuild must migrate the payload's shared-pool "
"records -- dropping them starves the scheduler's vres usage "
"accounting (sharedInUse) and defeats every global limit");
TEST_ASSERT_NOT_NULL(j->job.allocs->allocs_shared);
TEST_ASSERT_EQUAL_STRING("limit=extlic_share/p=projA",
j->job.allocs->allocs_shared[0]->resname);
TEST_ASSERT_EQUAL_FLOAT(1.0f, j->job.allocs->allocs_shared[0]->value);
TEST_ASSERT_EQUAL_STRING("extlic",
j->job.allocs->allocs_shared[1]->resname);
TEST_ASSERT_EQUAL_FLOAT(1.0f, j->job.allocs->allocs_shared[1]->value);
TEST_ASSERT_NULL_MESSAGE(alloc->allocs_shared,
"the shared records must be MOVE_PTR'd out of the payload, not "
"copied -- a copy would double-free when the caller frees it");
free_JobAlloc_ptr(&alloc);
dc_drain_ep(&g_dc_worker_ep);
dc_teardown();
printf("test_dc_success_migrates_shared_allocs: PASS\n");
}
#define DC_JOBID_NOTFOUND 91002ULL
/* Regression safety net (untouched failure branch): a dispatch whose
@ -801,6 +873,7 @@ void test_dispatch_commit(void)
RUN_TEST(test_dc_success_commits_alloc_state_and_resets_counter);
RUN_TEST(test_dc_success_stamps_start_time_synchronously);
RUN_TEST(test_dc_success_notifies_job_update_hook_exactly_once);
RUN_TEST(test_dc_success_migrates_shared_allocs);
RUN_TEST(test_dc_worker_not_found_leaves_job_pend_and_alloc_uncommitted);
RUN_TEST(test_dc_success_appends_wal_job_dispatch_durably);
}

297
src/controller/testsuite/test_extres_resvals.c Normal file
View file

@ -0,0 +1,297 @@
/*
* test_extres_resvals.c — external-license-resources Phase 1 (E2), steps 1-2
*
* The extres daemon (xceesd) pushes cluster-level (SHARED) resource values
* through the same resval update pipeline workers use, but it has NO
* workermgr slot: it joins as HostRole_extres and updates land on
* c->sharedinst directly. Two rules pin the daemon path:
*
* 1. update_resval_list() accepts w==NULL and applies SHARED-resource
* values into c->sharedinst (the only array the daemon can write).
* 2. The "SHARED 即收" guard: a NULL worker may only write SHARED
* resources — a non-SHARED (host-based) resource with w==NULL is
* refused (-EPERM) and NOT counted as applied (bug-1583's applied-list
* contract: downstream hooks must not hear about rejected values).
*
* This pins the opupd-side admission shape BEFORE the do_update_resvals
* role gate is extended to HostRole_extres (opupd itself needs a live
* endpoint, so the seam under test is worker.c's list applier, the same
* XCE_TESTING-exposure convention as apply_join_state in
* test_worker_join.c).
*
* external-license-resources Phase 2 adds rule 3: an EXTERNAL shared
* acceptance also stamps last_report on the myconf cluster-config tree's
* definition -- the apiserver's resource-definition store is fed ONLY by
* the join-time config push (myconf tree, not the live resmgr object), so
* without this the CLI's LAST_UPDATED column would stay "never" forever.
*/
/* rtti.h must come first — it defines 'var' and 'typeinfo_t' used by kvmgr.h */
#include <rtti.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "cluster.h"
#include "controller.capnp.h"
#include "kvmgr.h"
#include "myconf.h"
#include "resobj.h"
#include "worker.h"
#include "xce/types.h"
#include "zarray.h"
/* ── Forward declarations for functions under test ──────────────────── */
/*
* XCE_TESTING exposure in worker.c (same convention as apply_join_state):
* the daemon path calls update_resval_list() with w==NULL.
*/
int update_resval_list(cluster_t* c, worker_t* w, xce_resval_list_t* l,
xce_resval_list_t* applied);
/* ── fixture ────────────────────────────────────────────────────────── */
typedef struct {
cluster_t c;
var resmgr;
/* myconf fixture: one cluster_config_t owning a static resources[] tree
* (the config-push wire source, distinct from the resmgr objects). */
myconf_t myconf;
cluster_config_t ccfg;
xce_resource_t* cfg_res[2];
xce_resource_t cfg_defs[2];
} test_cluster_t;
static test_cluster_t g_te;
static void setup_resmgr(void)
{
kvmgr_params_t str_params = {
.key_type = KVMGR_KEY_STRING,
.nslots = 64,
.delta = 16,
.datakiller = NULL,
};
memset(&g_te, 0, sizeof(g_te));
g_te.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(g_te.resmgr);
/* one SHARED resource (daemon-writable) and one host-based resource */
{
static const char* names[2] = { "lic", "slots" };
static const int shared[2] = { 1, 0 };
xce_resource_t rdef;
int i;
for (i = 0; i < 2; i++) {
memset(&rdef, 0, sizeof(rdef));
rdef.name = (char*)names[i];
rdef.valtype = XCE_VAL_NUMERIC;
rdef.flags = shared[i] ? XCE_RESF_SHARED : 0;
rdef.idx = -1;
TEST_ASSERT_NOT_NULL_MESSAGE(makeResource(g_te.resmgr, &rdef),
names[i]);
}
}
g_te.c.resmgr = g_te.resmgr;
zarray_init(&g_te.c.sharedinst, 64, NULL);
/* the myconf tree mirrors the resmgr registrations by name (the live
* shape: cluster_build_resources applies the config tree INTO resmgr,
* so names match; objects are distinct allocations) */
{
static const char* names[2] = { "lic", "slots" };
for (int i = 0; i < 2; i++) {
memset(&g_te.cfg_defs[i], 0, sizeof(g_te.cfg_defs[i]));
g_te.cfg_defs[i].name = (char*)names[i];
g_te.cfg_res[i] = &g_te.cfg_defs[i];
}
g_te.ccfg.nres = 2;
g_te.ccfg.resources = g_te.cfg_res;
g_te.myconf.cluster_config = &g_te.ccfg;
g_te.c.myconf = &g_te.myconf;
}
}
static void teardown_resmgr(void)
{
zarray_fini(&g_te.c.sharedinst);
if (g_te.resmgr) dereference(g_te.resmgr);
memset(&g_te, 0, sizeof(g_te));
}
/* Build a one-entry resval list for `resname` with fvalue `value`. The
* xce_val_t and the list are statics so no teardown is needed (makeResource
* interns names; the resval name pointer must outlive the call). */
static xce_resval_list_t g_te_list;
static xce_resval_t* g_te_vals[4];
static xce_resval_t g_te_rv[4];
static xce_val_t g_te_val[4];
static xce_resval_list_t* make_resval_list(const char** names,
const float* values, int n)
{
int i;
for (i = 0; i < n; i++) {
memset(&g_te_val[i], 0, sizeof(g_te_val[i]));
g_te_val[i].which = Value_u_fvalue;
g_te_val[i].u.fvalue = values[i];
memset(&g_te_rv[i], 0, sizeof(g_te_rv[i]));
g_te_rv[i].name = (char*)names[i];
g_te_rv[i].host = NULL;
g_te_rv[i].val = &g_te_val[i];
g_te_vals[i] = &g_te_rv[i];
}
g_te_list.count = n;
g_te_list.vals = g_te_vals;
return &g_te_list;
}
/* ── tests ──────────────────────────────────────────────────────────── */
/* Rule 1: daemon (w==NULL) SHARED write lands in c->sharedinst at the
* resource's idx. */
static void test_extres_shared_resval_lands_in_sharedinst(void)
{
static const char* names[1] = { "lic" };
static const float values[1] = { 4.0f };
xce_resval_list_t* l;
xce_val_t* stored;
int rc;
setup_resmgr();
l = make_resval_list(names, values, 1);
rc = update_resval_list(&g_te.c, NULL, l, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
stored = (xce_val_t*)zarray_get(&g_te.c.sharedinst, 0);
TEST_ASSERT_NOT_NULL(stored);
TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which);
TEST_ASSERT_EQUAL_FLOAT(4.0, stored->u.fvalue);
teardown_resmgr();
}
/* Rule 2 ("SHARED 即收"): w==NULL + non-SHARED resource is refused
* (-EPERM per-item, logged by the caller's failure hook), not applied, and
* not reported as applied -- the list-level call still succeeds, matching
* update_resvals()'s per-item convention. */
static void test_extres_nonshared_resval_refused_eperm(void)
{
static const char* names[2] = { "slots", "lic" };
static const float values[2] = { 8.0f, 4.0f };
xce_resval_list_t* l;
xce_resval_list_t applied = { .count = 0, .vals = NULL };
xce_resval_t** applied_vals;
xce_val_t* stored;
int rc;
setup_resmgr();
applied_vals = (xce_resval_t**)calloc((size_t)2, sizeof(xce_resval_t*));
TEST_ASSERT_NOT_NULL(applied_vals);
applied.vals = applied_vals;
l = make_resval_list(names, values, 2);
rc = update_resval_list(&g_te.c, NULL, l, &applied);
TEST_ASSERT_EQUAL_INT(0, rc); /* list-level call still succeeds */
/* the SHARED entry was applied and reported... */
stored = (xce_val_t*)zarray_get(&g_te.c.sharedinst, 0);
TEST_ASSERT_NOT_NULL(stored);
TEST_ASSERT_EQUAL_FLOAT(4.0, stored->u.fvalue);
TEST_ASSERT_EQUAL_INT(1, applied.count);
TEST_ASSERT_EQUAL_STRING("lic", applied.vals[0]->name);
/* ...the non-SHARED entry ("slots", first in the list) was refused and
* NOT reported as applied */
TEST_ASSERT_EQUAL_STRING("slots", l->vals[0]->name); /* sanity */
TEST_ASSERT_EQUAL_INT(1, applied.count);
teardown_resmgr();
free(applied_vals);
}
/* Rule 3 (Phase 2): an EXTERNAL shared acceptance stamps last_report on the
* myconf cluster-config tree's definition with the current time — the
* config push the apiserver receives on (re)join serializes THAT tree, not
* the live resmgr object, so an unstamped tree would show "never" in the
* CLI's LAST_UPDATED column even while the daemon pushes normally. */
static void test_extres_external_accept_stamps_config_tree(void)
{
static const char* names[1] = { "lic" };
static const float values[1] = { 4.0f };
xce_resval_list_t* l;
time_t before;
time_t after;
setup_resmgr();
/* mark the definition EXTERNAL and confirm both copies start unstamped
* (config decode leaves last_report 0 for YAML-sourced definitions) */
{
xce_resource_t* live = NULL;
lock(g_te.resmgr, RTTI_LOCK_READ);
(void)key_find(g_te.resmgr, (void*)"lic", (void**)&live);
unlock(g_te.resmgr);
TEST_ASSERT_NOT_NULL(live);
live->flags |= XCE_RESF_EXTERNAL;
g_te.cfg_defs[0].flags |= XCE_RESF_EXTERNAL;
TEST_ASSERT_EQUAL_INT(0, live->last_report);
TEST_ASSERT_EQUAL_INT(0, g_te.cfg_defs[0].last_report);
}
time(&before);
l = make_resval_list(names, values, 1);
TEST_ASSERT_EQUAL_INT(0, update_resval_list(&g_te.c, NULL, l, NULL));
time(&after);
TEST_ASSERT_EQUAL_INT((int)before, (int)g_te.cfg_defs[0].last_report);
TEST_ASSERT_TRUE(g_te.cfg_defs[0].last_report >= before);
TEST_ASSERT_TRUE(g_te.cfg_defs[0].last_report <= after);
teardown_resmgr();
}
/* The stamp is EXTERNAL-gated: a plain SHARED (non-EXTERNAL) acceptance
* leaves the config tree untouched — last_report means "external value
* last accepted", and worker-sourced values must never set it. */
static void test_extres_plain_shared_accept_leaves_config_tree(void)
{
static const char* names[1] = { "lic" };
static const float values[1] = { 4.0f };
xce_resval_list_t* l;
setup_resmgr();
/* SHARED but NOT EXTERNAL */
l = make_resval_list(names, values, 1);
TEST_ASSERT_EQUAL_INT(0, update_resval_list(&g_te.c, NULL, l, NULL));
TEST_ASSERT_EQUAL_INT(0, g_te.cfg_defs[0].last_report);
teardown_resmgr();
}
/* ── suite entry point ─────────────────────────────────────────────── */
void test_extres_resvals(void)
{
UnitySetTestFile(__FILE__);
RUN_TEST(test_extres_shared_resval_lands_in_sharedinst);
RUN_TEST(test_extres_nonshared_resval_refused_eperm);
RUN_TEST(test_extres_external_accept_stamps_config_tree);
RUN_TEST(test_extres_plain_shared_accept_leaves_config_tree);
}

16
src/controller/testsuite/test_job_live_gc.c
View file

@ -160,13 +160,21 @@ static void test_sweep_purges_expired_done_job(void)
TEST_ASSERT_EQUAL_INT(3, jlg_job_count());
/* Refcount before the sweep: seed gives the table 1, and nothing else
* holds the job. Read NOW — a correct sweep dereferences down to
* job_destroy and FREES the whole RTTI object, so touching `expired`
* after the sweep is a use-after-free (the assertion that proved
* "refcount reached 0" by reading freed memory was itself the bug;
* destruction is verified by absence below instead). */
TEST_ASSERT_EQUAL_UINT32(1, job_get_refcount(expired));
job_live_gc_sweep(&g_jlg_c);
/* The expired job is gone from the table... */
/* The expired job is gone from the table — and, with the fixture
* holding no other reference, a correct sweep also destroyed it
* (refcount 1 → 0 through its two dereferences). `expired` must not
* be dereferenced past this point. */
TEST_ASSERT_NULL(jlg_find_job(81001));
/* ...and its refcount ran down to destruction (seed refcount 1, sweep
* added 1, dereference after unlink leaves 0 → job_destroy ran). */
TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired));
/* The others are untouched. */
TEST_ASSERT_NOT_NULL(jlg_find_job(81002));

36
src/controller/worker.c
View file

@ -74,6 +74,42 @@ static int update_val_res(cluster_t* c, worker_t* w, xce_resource_t* r,
if (r->flags & XCE_RESF_SHARED) {
/* update shared resource */
array = &(c->sharedinst);
/* external-license-resources Phase 2: the descriptor's reserved
* last_report becomes the "external value freshness" stamp -- the
* extres daemon's accepted pushes are the only writers of an
* EXTERNAL shared value, so every acceptance refreshes it and
* `xce resources` / a stale read can tell "reported recently"
* from "kept old value" (design.md 3.5 运维面). The apiserver's
* definition store is fed by the join-time CONFIG push, which
* serializes c->myconf->cluster_config -- a DIFFERENT copy from
* this live resmgr object -- so that tree's same-named definition
* gets the same stamp or the CLI's LAST_UPDATED column would stay
* "never" between apiserver (re)joins. */
if (r->flags & XCE_RESF_EXTERNAL) {
time_t now = time(NULL);
r->last_report = now;
if ((c->myconf != NULL) && (c->myconf->cluster_config != NULL)) {
cluster_config_t* cc = c->myconf->cluster_config;
for (int i = 0; i < cc->nres; i++) {
xce_resource_t* cr = cc->resources[i];
if ((cr != NULL) && (cr->name != NULL)
&& (strcmp(cr->name, r->name) == 0)) {
cr->last_report = now;
break;
}
}
}
}
} else if (w == NULL) {
/* external-license-resources Phase 1: a NULL worker is the extres
* daemon (HostRole_extres join, no workermgr slot). It may only
* write SHARED (cluster-level) resources -- a host-based resource
* has no per-host storage without a worker, and the daemon must
* never fabricate host values. Refuse loudly. */
return -EPERM;
} else {
/* update host based resource */
array = &(w->host.resvals);

47
src/docs/xce.md
View file

@ -122,8 +122,9 @@ supports a subset of the LSF array syntax.
- **Element number in the job's environment.** A running element finds its
own number in `XCE_JOBINDEX` (1..N). An ordinary job gets
`XCE_JOBINDEX=0`. A user-supplied `XCE_JOBINDEX` is overridden, with a
warning in the worker log. `XCE_JOBID` is the element's own job number, as
before.
warning in the worker log. `XCE_JOBID` is the element's own job number in
its written form (`cluster-apiserver-timestamp-sequence`, the same form
listings and control commands accept).
- **Test harness.** `xce_test jobsub -a N` submits elements 1 through N
(2 <= N <= 32767, regardless of `max_array_size`; `-a 1` submits an
ordinary job). `$jobid` is then the array identity and `$jobid[k]` names
@ -156,6 +157,48 @@ These LSF features are not available either:
- **`LSB_JOBINDEX` / `LSB_JOBID` variables** are not set. Scripts read
`XCE_JOBINDEX` (element number) and `XCE_JOBID` (job number) instead.
## Resource requirement (resreq)
A job's resource request is given at submission with
`xcecli jobs submit -R "<request>"` (REST: `resreq`). The grammar accepts
`select[...]`, `rusage[...]`, `span[...]`, `same[...]` and `affinity[...]`
clauses in any order, compound requests (`N*{...} + M*{...}`) and
alternatives (`{...} || {...}`). A job with no request (or an empty one)
is scheduled as `type == any` (any host) — the API layer does not apply a
default; the "default: type == local" note in the API model is historical.
What the scheduler actually enforces today:
- **`select[<expression>]`** — enforced. The expression is evaluated
against each candidate host's registered resources (`resources.yaml`:
`slots`, `mem`, load indexes, host attributes `type`/`model`/... and any
static resources). Workers report `type` as `LINUX64` and `model` as
`Intel64`.
- An unknown resource name in the expression is **tolerated, not
rejected**: the select is dropped and the job is scheduled without it,
with a warning in the scheduler log. Everything after the dropped
select in the same request (e.g. a trailing `rusage[...]`) is dropped
with it. A malformed select (syntax error, type mismatch) rejects the
whole request at submission or scheduling.
- **`rusage[<res>=<value>[:duration=<d>[:decay=<0|1>]] ...]`** — the
value is enforced as a per-task request (multiplied by the job's slot
count; `mem` also feeds memory limits and accounting). The optional
`per` suffix (`value/task`, `value/job`, `value/host`) is parsed and
kept, and `value/job` / `value/host` are honored when computing
job-wide totals, but the allocation request itself is always built
**per task**. `duration` and `decay` are parsed and stored but **not
consumed** — no historical-usage decay is implemented (see gaps.md).
- **`span[...]`, `same[...]`, `affinity[...]`** — parsed for syntax and
**dropped**: a multi-host job may be split across hosts by even
splitting only, with no placement control (see gaps.md #15).
- **Queue-level `res_req` / `gpu_req`** (`queues.yaml`) — the queue's
`res_req` string is carried to the scheduler's queue object but is
**never merged into job allocation**; jobs get only their own request.
`gpu_req` has no wire field at all.
- **Ordering** — requests are evaluated first-fit in submission order
within a queue; there is no `order[...]` clause and no load-based
ordering.
## shell
An interactive bash shell used for troubleshooting and inspection. Not

130
src/esd/include/esd.h Normal file
View file

@ -0,0 +1,130 @@
#if !defined(_ESD_H_)
#define _ESD_H_ 1
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include "config.h"
#include "client.h"
#include "xce/types.h"
/*
* xceesd -- XCE External-resources Daemon (external-license-resources
* Phase 1, design.md 3.1). Joins the controller as HostRole_extres and
* pushes cluster-level FREE values for EXTERNAL shared resources into the
* existing resval pipeline (OPCode_update / Update_data_resvals, host==""
* -- the sharedInst landing shape).
*
* Collectors are plugins over one internal interface; Phase 1 ships the
* script collector (any external resource: license-free wrappers, cloud
* quotas). The flexlm collector is the same interface over `lmstat -a`
* output parsing.
*/
/* One configured collector: a name (must match a resources.yaml
* SHARED|GLOBAL|EXTERNAL resource) and how to get its value. */
typedef struct esd_source_s {
char* name; /* resource name, same as resources.yaml */
char* command; /* script collector: the command to run */
int interval; /* poll interval, seconds (0: use the daemon default) */
long stale_limit; /* seconds a kept value stays trustworthy (0: use
* the daemon default); design.md 3.4 */
double last_value; /* last successfully-parsed value */
time_t last_ok_time; /* when last_value was collected (0: never) */
bool have_value; /* last_value is valid (any push has succeeded) */
bool last_failed; /* last poll failed (stale-value policy input) */
} esd_source_t;
typedef struct esd_config_s {
char* controller_host; /* controller address (legacy single) */
int controller_port;
char* peers; /* --peers style candidate list, or NULL */
int poll_interval; /* daemon default poll interval, seconds */
long stale_limit; /* daemon default stale limit, seconds */
char* status_file; /* status JSON path (NULL: no file) */
int nsrc;
esd_source_t* src; /* array of nsrc sources */
} esd_config_t;
/* Daemon-wide counters and identity for the status file (Phase 2,
* design.md 3.5 "xceadmin extres status"): the daemon fills one and hands
* it to esd_status_write each poll round. */
typedef struct esd_stats_s {
time_t started_at; /* daemon start, epoch seconds */
time_t now; /* snapshot time (filled by esd_status_write when 0) */
long pushes_ok; /* successful resval pushes since start */
long pushes_fail; /* failed pushes since start */
} esd_stats_t;
/*
* Parse esd.yaml:
*
* controller: 172.31.1.30:9990 # or peers: "h1:9990,h2:9990"
* poll_interval: 30
* resources:
* - name: Verilog_XL
* collector: script
* command: /etc/xce/lic-report.sh
* interval: 15 # optional per-source override
*
* Returns 0 on success and fills *cfg_out (caller frees with
* esd_config_fini). Returns -errno-shaped negatives on failure: -EINVAL
* for a structurally-invalid file, -ENOENT for a missing one. Unknown
* keys are ignored (forward-compatible, same convention as the
* controller's own YAML loader).
*/
extern int esd_config_load(const char* path, esd_config_t* cfg_out);
extern void esd_config_fini(esd_config_t* cfg);
/*
* Run one source's collector command and parse its output. The script
* collector contract (design.md 3.4): the command prints either
* "<name> <value>" (elim-style) or "<value>" (a bare number), one line;
* leading/trailing whitespace is ignored. stdout only -- stderr is
* forwarded to the log untouched.
*
* Returns the parsed value, or NAN when the command failed or its output
* did not parse (caller decides the stale-value policy -- the daemon's
* default is keep-last-value + WARN, fail-open, design.md 3.4).
*
* posix_spawn(), not fork(): the daemon is multi-threaded (the client
* library runs its own transport thread), and a forked child that logged
* or allocated before exec could deadlock on a lock another thread held
* (tech.md "Process Creation"). The child gets an empty signal mask,
* default signal dispositions, and waits with a watchdog kill at
* esd_script_timeout_ms.
*/
extern double esd_script_collect(const char* command);
/* Script-collector watchdog: how long a collection command may run before
* it is killed (default 10s; main.c overrides from the CLI). */
extern int esd_script_timeout_ms;
/* Default stale limit for sources (and the config default) -- design.md
* 3.4's example: 150s. A source whose collector keeps failing past this
* many seconds since its last good value reports stale=true in the status
* file (the daemon itself keeps the last value -- fail-open). */
#define ESD_STALE_LIMIT_DEFAULT 150
/* Stale predicate (Phase 2): true when the source's last good value is
* older than its stale limit -- `now` vs src->last_ok_time, the source's
* own stale_limit when set, else the daemon default. A source with no
* value yet is never stale (there is nothing being kept alive; admission
* already treats the pool as empty through the normal NO_RESOURCE path).
* A last_ok_time in the future (clock skew) is not stale. */
extern bool esd_source_is_stale(const esd_source_t* src, time_t now,
long default_limit);
/* Write the daemon status JSON (Phase 2, "xceadmin extres status"): one
* object with the daemon identity, push counters, and per-source rows
* (name, last_value, last_ok_time, stale). `stats->now == 0` is filled
* with time(NULL). Written atomically (tmp + rename) so a reader never
* sees a partial file. Returns 0, or -errno-shaped negative (the caller
* logs; the status file is best-effort observability and must never take
* the poll loop down). */
extern int esd_status_write(const char* path, const esd_config_t* cfg,
esd_stats_t* stats);
#endif

42
src/esd/meson.build Normal file
View file

@ -0,0 +1,42 @@
# xceesd -- XCE External-resources Daemon (external-license-resources
# Phase 1, design.md 3.1). Joins the controller as HostRole_extres and
# pushes cluster-level FREE values into the existing resval pipeline.
esd_srcs = [
'src/esd_config.c',
'src/esd_script.c',
'src/esd_status.c',
'src/esd.c',
]
esd_deps = [
libxce_dep,
yaml_deps,
tinylog_dep,
]
esd_incdir = include_directories('include', '../include', '../lib')
xceesd = executable('xceesd',
esd_srcs + ['src/main.c'],
include_directories: [esd_incdir],
dependencies: esd_deps,
install: true)
# Unit tests: white-box #include of the core sources (config parse,
# script-output parse, status writer, and -- since bug-1978 -- esd.c's
# reconnect-flag contract; esd_run's live-join sequence is still covered by
# the t1 E2E) -- same convention as the worker's resource_sync_test.
esd_test_incdir = include_directories('include', '../include', '../lib', '.')
esd_test_deps = [libxce_dep, yaml_deps, tinylog_dep, unity_dep]
esd_test = executable('esd_test',
['testsuite/test_esd_core.c'],
include_directories: [esd_test_incdir],
dependencies: esd_test_deps,
install: false)
test('esd core test', esd_test,
suite: 'esd',
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])

363
src/esd/src/esd.c Normal file
View file

@ -0,0 +1,363 @@
/* esd.c -- the xceesd daemon body (external-license-resources Phase 1,
* design.md 3.1): join the controller as HostRole_extres, poll the
* configured collectors, and push every refreshed shared value as a
* host=="" resval delta -- the sharedInst landing shape the controller's
* opupd extres branch and the scheduler's DELTA sharedInst route expect.
*
* Reconnect semantics (design.md 3.5 step 6, "重连即全量重推"): every
* (re)join is followed by a FULL push of every source's last-known value
* -- the same contract the worker's job-state resync re-report has
* (9d7628e): the controller keeps no durable state for this daemon, so a
* fresh controller (boot or failover promotion) must be told everything
* before the next poll tick.
*/
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "esd.h"
#include "tlog.h"
#include "xce/cluster.h"
/* The daemon's client-library context and identity (from main.c's CLI). */
typedef struct esd_daemon_s {
xcecli_ctx_t apictx;
esd_config_t cfg;
const char* from; /* join identity ("xceesd") */
int running; /* 0 once the main loop should exit */
xce_cluster_config_t cluster_cfg; /* validated peers list (HA mode) */
esd_stats_t stats; /* status-file counters (Phase 2) */
} esd_daemon_t;
static esd_daemon_t g_esd;
int esd_push_value(const char* name, double value)
{
xce_resval_t resval;
xce_resval_t* valptr = &resval; /* the wire takes an array of POINTERS
* (xce_resval_t**, typemap.h) -- passing
* &resval itself made the encoder read
* the struct's first word as the next
* element pointer and strlen() garbage
* (live segfault, 2026-10-26) */
xce_val_t val;
int rc;
memset(&resval, 0, sizeof(resval));
memset(&val, 0, sizeof(val));
val.which = Value_u_fvalue;
val.u.fvalue = value;
/* host == "" is THE landing-shape marker: the controller's opupd
* extres branch drops the worker lookup entirely and routes the entry
* to update_val_res's SHARED branch (c->sharedinst), and the
* scheduler's DELTA path treats host==""/NULL as sharedInst (mirrors
* snapshot_load.c's bug-1923 precedent). */
resval.name = (char*)name;
resval.host = (char*)"";
resval.val = &val;
rc = xcecli_update_resvals(g_esd.apictx, 1, &valptr);
if (rc < 0) {
ERROR("esd push %s=%f failed (rc=%d)", name, value, rc);
g_esd.stats.pushes_fail++;
} else {
TRACE("esd pushed %s=%f", name, value);
g_esd.stats.pushes_ok++;
}
return rc;
}
/* One poll pass: collect every source, remember the last good value, and
* push only what CHANGED (the debounce -- design.md step 6 "轮询去抖").
* A failed collection keeps the previous value and does not push (fail-
* open: the controller keeps scheduling against the last known good
* count; design.md 3.4 stale policy default).
*
* Phase 2: every source records last_ok_time on a good collect (the
* stale predicate's input, exposed via the status file); a failure past
* the source's stale limit escalates from the one-shot WARN to a per-
* round stale WARN -- the operator sees a kept-old-value daemon without
* having to guess from the absence of "pushed" lines. */
int esd_poll_round(void)
{
int i;
int pushed = 0;
time_t now = time(NULL);
for (i = 0; i < g_esd.cfg.nsrc; i++) {
esd_source_t* src = &(g_esd.cfg.src[i]);
double v;
v = esd_script_collect(src->command);
if (isnan(v)) {
if (!src->last_failed) {
WARN("source %s: collection failed -- keeping last known "
"value %f (fail-open)",
src->name, src->have_value ? src->last_value : 0.0);
} else if (esd_source_is_stale(src, now,
g_esd.cfg.stale_limit)) {
WARN("source %s: STALE for %lds (limit %lds) -- last good "
"value %f kept (fail-open); check the collector",
src->name,
(long)(now - src->last_ok_time),
(src->stale_limit > 0) ? src->stale_limit
: g_esd.cfg.stale_limit,
src->last_value);
}
src->last_failed = true;
continue;
}
src->last_failed = false;
src->last_ok_time = now;
if (src->have_value && (src->last_value == v)) {
continue; /* unchanged -- the debounce */
}
if (esd_push_value(src->name, v) == 0) {
src->have_value = true;
src->last_value = v;
pushed++;
}
}
return pushed;
}
/* Full re-push of every source's last-known value (reconnect semantics:
* the controller we just joined knows nothing about us). Sources with no
* value yet are skipped -- nothing to push, the poll round will fill
* them. */
int esd_push_all(void)
{
int i;
int pushed = 0;
for (i = 0; i < g_esd.cfg.nsrc; i++) {
esd_source_t* src = &(g_esd.cfg.src[i]);
if (!src->have_value) {
continue;
}
if (esd_push_value(src->name, src->last_value) == 0) {
pushed++;
}
}
INFO("esd full re-push: %d/%d source(s)", pushed, g_esd.cfg.nsrc);
return pushed;
}
/* --rejoin hook (client library contract): the library calls this while
* holding its failover lock -- no I/O, no xcecli_* calls (would self-
* deadlock, the worker's xcecd_on_reconnect lesson, controller_connector.c).
* We only set flags; the main loop consumes them after the lock is free. */
static volatile int g_needs_full_push = 0;
static volatile int g_needs_rejoin = 0;
static int esd_on_reconnect(xcecli_ctx_t ctx, void* arg)
{
(void)ctx;
(void)arg;
/* bug-1978: a re-PUSH alone is not enough -- the fresh controller
* endpoint was created by handle_connection() with priv->role =
* HostRole_client, and only OPCode_join sets it to HostRole_extres
* (join.c). Without the re-join every resval push lands in
* do_update_resvals's neither-branch and is refused -EPERM forever
* (the live sighting: pushes_ok frozen, rc=-1 on every attempt after
* a controller restart). Mirror the worker's needs_rejoin contract:
* the main loop re-issues the role join, then does the full re-push. */
g_needs_rejoin = 1;
g_needs_full_push = 1;
return 0;
}
int esd_run(void)
{
int next_full_push = 1; /* the FIRST join needs the full push too */
/* Join sequence mirrors the worker's xcecd_join_cluster (controller_
* connector.c): build ONE validated candidate list -- peers from
* esd.yaml when given, else the single controller host:port -- and
* hand it to the client library, which owns failover rotation and
* standby redirects inside the single xcecli_join call. */
{
xce_client_config_t cconfig;
memset(&cconfig, 0, sizeof(cconfig));
if (g_esd.cfg.peers != NULL) {
int rc = xce_parse_peers(g_esd.cfg.peers, &g_esd.cluster_cfg);
if (rc < 0) {
ERROR("esd: cannot parse peers \"%s\"", g_esd.cfg.peers);
return -EINVAL;
}
strncpy(g_esd.cluster_cfg.identity, g_esd.from,
sizeof(g_esd.cluster_cfg.identity) - 1);
rc = xce_cluster_config_validate(&g_esd.cluster_cfg);
if (rc < 0) {
ERROR("esd: invalid peers config (rc=%d)", rc);
return rc;
}
rc = xce_cluster_config_to_client_config(&g_esd.cluster_cfg,
&cconfig);
if (rc < 0) {
ERROR("esd: candidate list conversion failed (rc=%d)", rc);
return rc;
}
} else {
cconfig.nmasters = 1;
cconfig.masters[0] = g_esd.cfg.controller_host;
cconfig.master_ports[0] = g_esd.cfg.controller_port;
}
cconfig.from = (char*)g_esd.from;
cconfig.key = (char*)"keepsecret$";
{
int rc = xcecli_ctx_init(&g_esd.apictx, &cconfig);
if (rc < 0) {
ERROR("esd: client ctx init failed (rc=%d)", rc);
return rc;
}
}
if (xcecli_start(g_esd.apictx) < 0) {
ERROR("esd: cannot reach controller %s:%d",
g_esd.cfg.controller_host, g_esd.cfg.controller_port);
xcecli_ctx_fini(g_esd.apictx);
return -ECONNREFUSED;
}
/* AFTER xcecli_start(): start() resets rejoin_cb (the worker's
* task-2.3 lesson, controller_connector.c). */
xcecli_set_reconnect_handler(g_esd.apictx, esd_on_reconnect, NULL);
{
int rc = xcecli_join(g_esd.apictx, HostRole_extres, NULL);
if (rc < 0) {
ERROR("esd: join failed (rc=%d)", rc);
xcecli_stop(g_esd.apictx);
xcecli_ctx_fini(g_esd.apictx);
return rc;
}
}
}
INFO("esd: joined cluster as %s (%d source(s))", g_esd.from,
g_esd.cfg.nsrc);
while (g_esd.running) {
int i;
if (g_needs_rejoin) {
/* bug-1978: a transparent reconnect just landed -- the fresh
* controller endpoint defaults to HostRole_client, so the role
* join must be re-issued BEFORE any push (an un-joined push is
* refused -EPERM). This runs on the main loop, outside the
* library's failover lock (the xcecd_on_reconnect constraint).
* Failure keeps the flag set: the next tick retries, mirroring
* the worker's needs_rejoin self-healing cadence. rc >= 0 is
* success for the same reason as the worker's rejoin (the
* controller's reply.status for a plain extres join is 0, but
* the failover wrapper surfaces its own codes on success). */
int rc = xcecli_join(g_esd.apictx, HostRole_extres, NULL);
if (rc >= 0) {
g_needs_rejoin = 0;
INFO("esd: re-joined after reconnect (rc=%d)", rc);
} else {
WARN("esd: re-join after reconnect failed (rc=%d) -- "
"retrying next tick",
rc);
}
}
if (g_needs_full_push) {
/* A transparent reconnect just landed (new primary after
* failover, or the old controller restarted): it has no
* record of this daemon -- push everything we know. */
g_needs_full_push = 0;
next_full_push = 0;
(void)esd_push_all();
}
if (next_full_push) {
next_full_push = 0;
(void)esd_push_all();
}
/* bug (live sighting 2026-10-26): with every source unchanged, the
* debounce suppresses ALL pushes -- and a reconnect only ever fires
* INSIDE a push's failover call. A controller restart while our
* value sits still left the daemon idle on a dead connection
* forever (no socket() for 50+ minutes, pushes never resumed).
* The broken-connection flag is set asynchronously by the
* transport's RDHUP handler (bug-1760's pure predicate): when it
* latches, force one full re-push -- the write attempt triggers the
* rotate/reconnect, and esd_on_reconnect's flags keep the
* re-join + full re-push contract for the failover path too
* (bug-1978: the re-push alone was refused -EPERM on the un-joined
* fresh endpoint). */
if (g_esd.cfg.nsrc > 0 && xcecli_is_connection_broken(g_esd.apictx)) {
INFO("esd: connection broken -- forcing a full re-push to "
"reconnect");
(void)esd_push_all();
}
(void)esd_poll_round();
/* Phase 2: refresh the status file every round (best-effort; the
* poll loop must survive a bad path). */
if (g_esd.cfg.status_file != NULL) {
/* one WARN per failure would spam a broken path every round;
* log on the transition only */
static bool status_warned = false;
int src = esd_status_write(g_esd.cfg.status_file, &g_esd.cfg,
&g_esd.stats);
if (src < 0) {
if (!status_warned) {
WARN("esd: cannot write status file %s (rc=%d) -- "
"continuing without it",
g_esd.cfg.status_file, src);
status_warned = true;
}
} else {
status_warned = false;
}
}
/* sleep in 1s slices so a signal stop lands promptly */
for (i = 0; (i < g_esd.cfg.poll_interval) && g_esd.running; i++) {
sleep(1);
}
}
xcecli_stop(g_esd.apictx);
xcecli_ctx_fini(g_esd.apictx);
return 0;
}
void esd_stop(void)
{
g_esd.running = 0;
}
void esd_init(esd_config_t* cfg, const char* from)
{
memset(&g_esd, 0, sizeof(g_esd));
g_esd.cfg = *cfg;
g_esd.from = from;
g_esd.running = 1;
g_esd.stats.started_at = time(NULL);
}

311
src/esd/src/esd_config.c Normal file
View file

@ -0,0 +1,311 @@
/* esd_config.c -- esd.yaml parsing for xceesd (external-license-resources
* Phase 1, design.md 3.4). libcyaml, same loader pattern as the
* autoscaler's prov_conf.c: schema-driven, unknown keys ignored
* (CYAML_CFG_IGNORE_UNKNOWN_KEYS), explicit field validation after load
* (a source without name or command can never report -- fail loudly at
* load, not silently at runtime). */
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <cyaml/cyaml.h>
#include "esd.h"
typedef struct yaml_esd_source_s {
char* name;
char* collector;
char* command;
long interval; /* 0 = daemon default */
long stale_limit; /* 0 = daemon default (Phase 2, design.md 3.4) */
} yaml_esd_source_t;
typedef struct yaml_esd_config_s {
char* controller;
char* peers;
long poll_interval;
long stale_limit; /* 0 = ESD_STALE_LIMIT_DEFAULT */
char* status_file; /* NULL = no status file */
yaml_esd_source_t* resources;
unsigned resources_count;
} yaml_esd_config_t;
static const cyaml_schema_field_t esd_source_fields_schema[] = {
CYAML_FIELD_STRING_PTR("name", CYAML_FLAG_POINTER, yaml_esd_source_t,
name, 1, CYAML_UNLIMITED),
CYAML_FIELD_STRING_PTR("collector", CYAML_FLAG_POINTER, yaml_esd_source_t,
collector, 1, CYAML_UNLIMITED),
CYAML_FIELD_STRING_PTR("command", CYAML_FLAG_POINTER, yaml_esd_source_t,
command, 1, CYAML_UNLIMITED),
CYAML_FIELD_INT("interval", CYAML_FLAG_OPTIONAL, yaml_esd_source_t,
interval),
CYAML_FIELD_INT("stale_limit", CYAML_FLAG_OPTIONAL, yaml_esd_source_t,
stale_limit),
CYAML_FIELD_END
};
static const cyaml_schema_value_t esd_source_schema = {
CYAML_VALUE_MAPPING(CYAML_FLAG_DEFAULT, yaml_esd_source_t,
esd_source_fields_schema)
};
static const cyaml_schema_field_t esd_config_fields_schema[] = {
CYAML_FIELD_STRING_PTR("controller", CYAML_FLAG_OPTIONAL,
yaml_esd_config_t, controller, 1, CYAML_UNLIMITED),
CYAML_FIELD_STRING_PTR("peers", CYAML_FLAG_OPTIONAL, yaml_esd_config_t,
peers, 1, CYAML_UNLIMITED),
CYAML_FIELD_INT("poll_interval", CYAML_FLAG_OPTIONAL, yaml_esd_config_t,
poll_interval),
CYAML_FIELD_INT("stale_limit", CYAML_FLAG_OPTIONAL, yaml_esd_config_t,
stale_limit),
CYAML_FIELD_STRING_PTR("status_file", CYAML_FLAG_OPTIONAL,
yaml_esd_config_t, status_file, 1, CYAML_UNLIMITED),
CYAML_FIELD_SEQUENCE("resources", CYAML_FLAG_POINTER, yaml_esd_config_t,
resources, &esd_source_schema, 0, CYAML_UNLIMITED),
CYAML_FIELD_END
};
static const cyaml_schema_value_t esd_config_schema = {
CYAML_VALUE_MAPPING(CYAML_FLAG_POINTER, yaml_esd_config_t,
esd_config_fields_schema)
};
/* "host:port" -> (host, port); returns 0 on success, -EINVAL when port
* missing/non-numeric/out of range. Mutates `spec` (splits at the colon). */
static int esd_parse_endpoint(char* spec, char** host_out, int* port_out)
{
char* colon;
char* endp = NULL;
long port;
if ((spec == NULL) || (*spec == '\0')) {
return -EINVAL;
}
colon = strrchr(spec, ':');
if ((colon == NULL) || (colon[1] == '\0')) {
return -EINVAL;
}
port = strtol(colon + 1, &endp, 10);
if ((endp == NULL) || (*endp != '\0') || (port <= 0) || (port > 65535)) {
return -EINVAL;
}
*colon = '\0';
*host_out = spec;
*port_out = (int)port;
return 0;
}
/* File-content variant, split out from the file loader for testability:
* the tests run esd_config_load_path() against real files, so both layers
* stay production code. */
int esd_config_load_path(const char* dir, const char* name,
esd_config_t* cfg_out)
{
static cyaml_config_t cyaml_cfg = {
.mem_fn = cyaml_mem,
.log_level = CYAML_LOG_WARNING,
.flags = CYAML_CFG_IGNORE_UNKNOWN_KEYS,
};
yaml_esd_config_t* ycfg = NULL;
cyaml_err_t err;
char* path = NULL;
esd_config_t cfg;
int i;
int rc;
if ((dir == NULL) || (name == NULL) || (cfg_out == NULL)) {
return -EINVAL;
}
memset(cfg_out, 0, sizeof(*cfg_out));
memset(&cfg, 0, sizeof(cfg));
if (asprintf(&path, "%s/%s", dir, name) < 0) {
return -ENOMEM;
}
err = cyaml_load_file(path, &cyaml_cfg, &esd_config_schema,
(void**)&ycfg, NULL);
free(path);
if (err != CYAML_OK) {
/* libcyaml collapses file-not-found into generic errors; map the
* obvious one so the daemon's startup log can say "missing file"
* vs "bad file" honestly. */
char probe[512];
FILE* f;
snprintf(probe, sizeof(probe), "%s/%s", dir, name);
f = fopen(probe, "r");
if (f == NULL) {
return -ENOENT;
}
fclose(f);
return -EINVAL;
}
if (ycfg == NULL) {
return -EINVAL;
}
/* Validation first: no controller target and no peers list means the
* daemon could never connect to anything. */
if ((ycfg->controller == NULL) && (ycfg->peers == NULL)) {
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
return -EINVAL;
}
if (ycfg->controller != NULL) {
char* host = NULL;
int port = 0;
if (esd_parse_endpoint(ycfg->controller, &host, &port) < 0) {
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
return -EINVAL;
}
/* esd_parse_endpoint splits ycfg->controller in place -- the host
* pointer points INTO the cyaml-owned string. Copy out before the
* cyaml_free below releases the whole decoded tree. */
cfg.controller_host = strdup(host);
cfg.controller_port = port;
if (cfg.controller_host == NULL) {
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
return -ENOMEM;
}
}
cfg.peers = (ycfg->peers != NULL) ? strdup(ycfg->peers) : NULL;
cfg.poll_interval = (int)ycfg->poll_interval;
cfg.stale_limit = ycfg->stale_limit;
cfg.status_file = (ycfg->status_file != NULL) ? strdup(ycfg->status_file)
: NULL;
cfg.nsrc = (int)ycfg->resources_count;
cfg.src = NULL;
if ((cfg.peers == NULL) && (ycfg->peers != NULL)) {
esd_config_fini(&cfg);
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
return -ENOMEM;
}
if ((cfg.status_file == NULL) && (ycfg->status_file != NULL)) {
esd_config_fini(&cfg);
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
return -ENOMEM;
}
if (cfg.nsrc > 0) {
cfg.src = (esd_source_t*)calloc((size_t)cfg.nsrc,
sizeof(esd_source_t));
if (cfg.src == NULL) {
esd_config_fini(&cfg);
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
return -ENOMEM;
}
}
rc = 0;
for (i = 0; i < cfg.nsrc; i++) {
yaml_esd_source_t* ys = &(ycfg->resources[i]);
if ((ys->name == NULL) || (ys->command == NULL)) {
rc = -EINVAL;
break;
}
cfg.src[i].name = strdup(ys->name);
cfg.src[i].command = strdup(ys->command);
cfg.src[i].interval = (int)ys->interval;
cfg.src[i].stale_limit = ys->stale_limit;
if ((cfg.src[i].name == NULL) || (cfg.src[i].command == NULL)) {
rc = -ENOMEM;
break;
}
}
cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0);
if (rc < 0) {
esd_config_fini(&cfg);
return rc;
}
*cfg_out = cfg;
return 0;
}
int esd_config_load(const char* path, esd_config_t* cfg_out)
{
char* dir;
char* name;
char* slash;
int rc;
if ((path == NULL) || (cfg_out == NULL)) {
return -EINVAL;
}
dir = strdup(path);
if (dir == NULL) {
return -ENOMEM;
}
slash = strrchr(dir, '/');
if (slash == NULL) {
name = dir;
dir = strdup(".");
if (dir == NULL) {
free(name);
return -ENOMEM;
}
} else {
*slash = '\0';
name = slash + 1;
if (*name == '\0') {
free(dir);
return -EINVAL;
}
if (dir == name - 1) {
/* path was "/name" -- dir is "" */
free(dir);
dir = strdup("/");
if (dir == NULL) {
return -ENOMEM;
}
}
}
rc = esd_config_load_path(dir, name, cfg_out);
free(dir);
return rc;
}
void esd_config_fini(esd_config_t* cfg)
{
int i;
if (cfg == NULL) {
return;
}
free(cfg->controller_host);
free(cfg->peers);
free(cfg->status_file);
for (i = 0; i < cfg->nsrc; i++) {
free(cfg->src[i].name);
free(cfg->src[i].command);
}
free(cfg->src);
memset(cfg, 0, sizeof(*cfg));
}

230
src/esd/src/esd_script.c Normal file
View file

@ -0,0 +1,230 @@
/* esd_script.c -- the script collector for xceesd (external-license-
* resources Phase 1, design.md 3.1): runs a user command with posix_spawn,
* reads its stdout, and parses "<value>" or "<name> <value>" from it.
*
* posix_spawn(), not fork(): the daemon is multi-threaded (the client
* library runs its own transport thread), and a forked child that logged
* or allocated before exec could deadlock on a lock another thread held
* (tech.md "Process Creation" -- same rationale as the autoscaler's
* plug_mgr.c run_script). The child starts with an empty signal mask and
* default signal dispositions.
*
* The watchdog: the pipe read is bounded by esd_script_timeout_ms --
* a hung collection command must not stall the poll loop (the value it
* was going to report belongs to a dead world anyway). We enforce it
* with poll(2) on the read end + a kill() at deadline; waitpid reaps.
*/
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <math.h>
#include <poll.h>
#include <signal.h>
#include <spawn.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>
#include "esd.h"
#include "tlog.h"
/* Script-collector watchdog, seconds of wall clock a collection command
* may run before it is killed (design.md 3.4: the poll loop must survive
* a hung script). Overridable from main.c. */
int esd_script_timeout_ms = 10 * 1000;
/*
* esd_parse_script_output -- the collector contract (design.md 3.4): the
* first non-empty line carries either a bare number or "<name> <value>".
* Leading/trailing whitespace ignored; anything else is garbage -> NAN
* (the caller owns the stale-value policy).
*/
double esd_parse_script_output(const char* out)
{
const char* p;
char* endp = NULL;
double v;
if (out == NULL) {
return NAN;
}
p = out;
while (*p != '\0') {
/* skip horizontal whitespace and newlines */
while ((*p == ' ') || (*p == '\t') || (*p == '\r') || (*p == '\n')) {
p++;
}
if (*p == '\0') {
break;
}
v = strtod(p, &endp);
if ((endp != NULL) && (endp != p)) {
/* accept trailing junk after the number (a name may follow in
* "value name" order too -- the number is what we need) */
return v;
}
/* skip this non-numeric token and try the next one */
while ((*p != '\0') && (*p != ' ') && (*p != '\t') && (*p != '\r')
&& (*p != '\n')) {
p++;
}
}
return NAN;
}
/* The argv for the child: run the command via sh -c, so a plain
* "/path/script.sh arg1 arg2" line works without the daemon ever
* splitting shell words itself (the operator owns quoting). */
static int esd_script_spawn(const char* command, int out_fd, pid_t* pid_out)
{
posix_spawn_file_actions_t fa;
posix_spawnattr_t attr;
sigset_t none, all;
char* argv[] = { (char*)"sh", (char*)"-c", (char*)command, NULL };
extern char** environ;
pid_t pid;
int rc;
sigemptyset(&none);
sigfillset(&all);
rc = posix_spawn_file_actions_init(&fa);
if (rc == 0) {
rc = posix_spawn_file_actions_adddup2(&fa, out_fd, STDOUT_FILENO);
}
if (rc == 0) {
rc = posix_spawn_file_actions_addopen(&fa, STDERR_FILENO, "/dev/null",
O_WRONLY, 0);
}
if (rc == 0) {
rc = posix_spawnattr_init(&attr);
}
if (rc == 0) {
rc = posix_spawnattr_setflags(
&attr, POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF);
}
if (rc == 0) {
rc = posix_spawnattr_setsigmask(&attr, &none);
}
if (rc == 0) {
rc = posix_spawnattr_setsigdefault(&attr, &all);
}
if (rc == 0) {
rc = posix_spawn(&pid, "/bin/sh", &fa, &attr, argv, environ);
}
posix_spawnattr_destroy(&attr);
posix_spawn_file_actions_destroy(&fa);
if (rc != 0) {
return -rc;
}
*pid_out = pid;
return 0;
}
double esd_script_collect(const char* command)
{
int fds[2];
pid_t pid = -1;
double value = NAN;
char buf[4096];
size_t total = 0;
int elapsed_ms = 0;
int rc;
int wstatus = 0;
struct pollfd pfd;
if ((command == NULL) || (*command == '\0')) {
return NAN;
}
if (pipe(fds) != 0) {
ERROR("script collect: pipe failed (%s)", strerror(errno));
return NAN;
}
rc = esd_script_spawn(command, fds[1], &pid);
close(fds[1]); /* parent's copy -- child holds the write end now */
if (rc < 0) {
close(fds[0]);
ERROR("script collect: spawn failed for \"%s\" (%s)", command,
strerror(-rc));
return NAN;
}
/* Bound the read at the watchdog deadline: a script that hangs (a
* blocked lmstat, a wedged NFS mount) must not stall the poll loop. */
pfd.fd = fds[0];
pfd.events = POLLIN;
while (total < sizeof(buf) - 1) {
int pret = poll(&pfd, 1, 100);
if (pret < 0) {
if (errno == EINTR) {
continue;
}
ERROR("script collect: poll failed (%s)", strerror(errno));
break;
}
if (pret == 0) {
elapsed_ms += 100;
if (elapsed_ms >= esd_script_timeout_ms) {
WARN("script collect: command \"%s\" exceeded %d ms -- "
"killing it, value not refreshed this round",
command, esd_script_timeout_ms);
kill(pid, SIGKILL);
waitpid(pid, &wstatus, 0);
close(fds[0]);
return NAN;
}
continue;
}
if (pfd.revents & (POLLERR | POLLHUP)) {
ssize_t n = read(fds[0], buf + total, sizeof(buf) - 1 - total);
if (n > 0) {
total += (size_t)n;
}
break;
}
if (pfd.revents & POLLIN) {
ssize_t n = read(fds[0], buf + total, sizeof(buf) - 1 - total);
if (n <= 0) {
break; /* EOF or error: done reading */
}
total += (size_t)n;
}
}
buf[total] = '\0';
close(fds[0]);
/* reap regardless of which path brought us here */
while (waitpid(pid, &wstatus, 0) < 0) {
if (errno != EINTR) {
break;
}
}
if (total == 0) {
WARN("script collect: command \"%s\" produced no output", command);
return NAN;
}
value = esd_parse_script_output(buf);
if (isnan(value)) {
WARN("script collect: command \"%s\" output did not parse as a "
"value (got %.60s)", command, buf);
}
return value;
}

123
src/esd/src/esd_status.c Normal file
View file

@ -0,0 +1,123 @@
/* esd_status.c -- xceesd status file (external-license-resources Phase 2,
* design.md 3.5 "xceadmin extres status"). The daemon writes one JSON
* snapshot per poll round to a configured path; xceadmin reads it on the
* daemon host. No new wire opcode: the PR-2 proxy formalizes a status
* channel later, and a local file keeps this observability surface
* strictly read-only and crash-safe.
*
* Atomicity: write to <path>.tmp then rename(2) -- a concurrent reader
* (xceadmin) either sees the previous complete file or the new one, never
* a partial write (the fake-license E2E's lic_state lesson: a reader
* observing a truncated file mistakes it for data loss).
*/
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include "esd.h"
bool esd_source_is_stale(const esd_source_t* src, time_t now,
long default_limit)
{
long limit;
long age;
if ((src == NULL) || (!src->have_value) || (src->last_ok_time == 0)) {
return false;
}
/* Future last_ok_time (caller clock skew): age would go negative and
* read as fresh -- that is the honest reading, keep it by clamping. */
age = (now > src->last_ok_time) ? (long)(now - src->last_ok_time) : 0;
limit = (src->stale_limit > 0) ? src->stale_limit : default_limit;
if (limit <= 0) {
limit = ESD_STALE_LIMIT_DEFAULT;
}
return age > limit;
}
int esd_status_write(const char* path, const esd_config_t* cfg,
esd_stats_t* stats)
{
char tmp[1024];
FILE* f;
int i;
time_t now;
int rc = 0;
if ((path == NULL) || (cfg == NULL) || (stats == NULL)) {
return -EINVAL;
}
now = (stats->now != 0) ? stats->now : time(NULL);
snprintf(tmp, sizeof(tmp), "%s.tmp", path);
f = fopen(tmp, "w");
if (f == NULL) {
return -errno;
}
if (fprintf(f,
"{\n"
" \"daemon\": \"xceesd\",\n"
" \"pid\": %ld,\n"
" \"started_at\": %ld,\n"
" \"generated_at\": %ld,\n"
" \"pushes_ok\": %ld,\n"
" \"pushes_fail\": %ld,\n"
" \"sources\": [\n",
(long)getpid(), (long)stats->started_at, (long)now,
stats->pushes_ok, stats->pushes_fail) < 0) {
rc = -EIO;
goto out;
}
for (i = 0; i < cfg->nsrc; i++) {
const esd_source_t* src = &(cfg->src[i]);
bool stale =
esd_source_is_stale(src, now, cfg->stale_limit);
char valbuf[64];
if (src->have_value) {
snprintf(valbuf, sizeof(valbuf), "%.17g", src->last_value);
} else {
snprintf(valbuf, sizeof(valbuf), "null");
}
if (fprintf(f,
" {\"name\": \"%s\", \"last_value\": %s, "
"\"last_ok_time\": %ld, \"stale\": %s}%s\n",
(src->name != NULL) ? src->name : "",
valbuf,
(long)src->last_ok_time, stale ? "true" : "false",
(i + 1 < cfg->nsrc) ? "," : "") < 0) {
rc = -EIO;
goto out;
}
}
if (fprintf(f, " ]\n}\n") < 0) {
rc = -EIO;
goto out;
}
out:
if (fclose(f) != 0) {
rc = (rc == 0) ? -EIO : rc;
}
if (rc == 0) {
if (rename(tmp, path) != 0) {
rc = -errno;
}
}
if (rc != 0) {
unlink(tmp); /* never leave the .tmp behind for a reader to trip on */
}
return rc;
}

127
src/esd/src/main.c Normal file
View file

@ -0,0 +1,127 @@
/* main.c -- xceesd entry point (external-license-resources Phase 1,
* design.md 3.1). Minimal on purpose: parse the CLI, load esd.yaml, run
* the poll loop until SIGINT/SIGTERM. */
#include <argp.h>
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "esd.h"
#include "tlog.h"
static char doc[] = "xceesd -- XCE External-resources Daemon";
static char args_doc[] = "";
const char* argp_program_bug_address = BUG_EMAIL;
struct arguments {
char* logfile;
char* config;
int verbose;
int timeout_ms;
};
#define OPT_TIMEOUT 256
static struct argp_option options[] = {
{ "logfile", 'l', "FILE", 0, "The full path of log file" },
{ "config", 'c', "FILE", 0,
"Path of the daemon configuration file (default: /etc/xce/esd.yaml)" },
{ "verbose", 'v', 0, 0, "Produce verbose log" },
{ "script-timeout", OPT_TIMEOUT, "MS", 0,
"Watchdog for collection commands in milliseconds (default: 10000)" },
{ 0 }
};
static error_t parse_opt(int key, char* arg, struct argp_state* state)
{
struct arguments* state_args = state->input;
switch (key) {
case 'l':
state_args->logfile = arg;
break;
case 'c':
state_args->config = arg;
break;
case 'v':
state_args->verbose++;
break;
case OPT_TIMEOUT:
state_args->timeout_ms = atoi(arg);
break;
default:
return ARGP_ERR_UNKNOWN;
}
return 0;
}
static struct argp argp = { options, parse_opt, args_doc, doc };
static esd_config_t g_cfg;
static void sig_handler(int signo)
{
(void)signo;
esd_stop();
}
int main(int argc, char* argv[])
{
struct arguments arguments;
char* logfile = "/tmp/xceesd.log";
char* loglevel = "info";
char* config = "/etc/xce/esd.yaml";
int rc;
memset(&arguments, 0, sizeof(arguments));
argp_parse(&argp, argc, argv, 0, 0, &arguments);
if (arguments.logfile != NULL) {
logfile = arguments.logfile;
}
if (arguments.config != NULL) {
config = arguments.config;
}
if (arguments.verbose > 0) {
loglevel = "debug";
}
if (arguments.timeout_ms > 0) {
esd_script_timeout_ms = arguments.timeout_ms;
}
mylog_init(logfile, loglevel, 16 * 1024 * 1024, 4);
rc = esd_config_load(config, &g_cfg);
if (rc < 0) {
fprintf(stderr, "xceesd: cannot load config %s: %s\n", config,
(rc == -ENOENT) ? "file not found"
: (rc == -EINVAL) ? "invalid file"
: strerror(-rc));
return 1;
}
INFO("xceesd starting: config %s, %d source(s), poll %ds", config,
g_cfg.nsrc, g_cfg.poll_interval > 0 ? g_cfg.poll_interval : 30);
if (g_cfg.poll_interval <= 0) {
g_cfg.poll_interval = 30;
}
signal(SIGINT, sig_handler);
signal(SIGTERM, sig_handler);
esd_init(&g_cfg, "xceesd");
rc = esd_run();
esd_config_fini(&g_cfg);
mylog_fini();
INFO("xceesd stopped (rc=%d)", rc);
return (rc == 0) ? 0 : 1;
}

434
src/esd/testsuite/test_esd_core.c Normal file
View file

@ -0,0 +1,434 @@
/* test_esd_core.c -- external-license-resources Phase 1, step 6 TDD: the
* xceesd skeleton's pure core -- esd.yaml parsing and the script
* collector's output parsing -- white-box #include of the sources under
* test (same pattern as the worker's resource_sync_test /
* controller_connector_test). The join/reconnect wiring lives in esd.c
* against the live client library and is exercised by the t1 E2E, not
* here; everything this file drives must be deterministic. */
#include <errno.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <unity/unity.h>
#include "esd.h"
/* White-box: reach the config schema/parse internals. */
#include "../src/esd_config.c"
/* White-box: the script collector's parse half (esd_parse_script_output);
* the spawn half (esd_script_collect) is exercised by the t1 E2E with a
* real fake-license script, not here. */
#include "../src/esd_script.c"
/* White-box: the stale predicate and the status-file writer (Phase 2,
* design.md 3.5 "运维面"). */
#include "../src/esd_status.c"
/* White-box: the reconnect-flag contract (bug-1978). esd.c's main loop
* drives the live client library, but the reconnect handler's flag contract
* -- set BOTH rejoin and full-push, and the loop consuming rejoin first --
* is unit-testable here without a controller: esd_run() itself is not
* linked (its join sequence would need a live primary); we drive
* esd_on_reconnect and read the loop's guard statics directly. */
#include "../src/esd.c"
static char g_tmpdir[256];
void setUp(void)
{
snprintf(g_tmpdir, sizeof(g_tmpdir), "/tmp/xce_esd_testXXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(g_tmpdir));
}
void tearDown(void)
{
if (g_tmpdir[0] != '\0') {
char path[300];
/* tests only write flat files inside g_tmpdir */
snprintf(path, sizeof(path), "rm -rf %s", g_tmpdir);
if (system(path) != 0) {
/* best effort */
}
g_tmpdir[0] = '\0';
}
}
static void write_file(const char* dir, const char* name, const char* content)
{
char path[512];
FILE* f;
snprintf(path, sizeof(path), "%s/%s", dir, name);
f = fopen(path, "w");
TEST_ASSERT_NOT_NULL(f);
fputs(content, f);
fclose(f);
}
static char* slurp(const char* path)
{
FILE* f;
long len;
char* buf;
f = fopen(path, "r");
if (f == NULL) {
return NULL;
}
fseek(f, 0, SEEK_END);
len = ftell(f);
fseek(f, 0, SEEK_SET);
buf = (char*)malloc((size_t)len + 1);
if (buf == NULL) {
fclose(f);
return NULL;
}
if (fread(buf, 1, (size_t)len, f) != (size_t)len) {
fclose(f);
free(buf);
return NULL;
}
buf[len] = '\0';
fclose(f);
return buf;
}
/* ── esd_config_load ─────────────────────────────────────────────────── */
static void test_config_load_full_file(void)
{
esd_config_t cfg;
int rc;
memset(&cfg, 0, sizeof(cfg));
write_file(g_tmpdir, "esd.yaml",
"controller: 172.31.1.30:9990\n"
"poll_interval: 30\n"
"resources:\n"
" - name: Verilog_XL\n"
" collector: script\n"
" command: /etc/xce/lic-report.sh\n"
" interval: 15\n"
" - name: cloud_gpu_quota\n"
" collector: script\n"
" command: /etc/xce/quota-report.sh\n");
rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_STRING("172.31.1.30", cfg.controller_host);
TEST_ASSERT_EQUAL_INT(9990, cfg.controller_port);
TEST_ASSERT_EQUAL_INT(30, cfg.poll_interval);
TEST_ASSERT_EQUAL_INT(2, cfg.nsrc);
TEST_ASSERT_NOT_NULL(cfg.src);
TEST_ASSERT_EQUAL_STRING("Verilog_XL", cfg.src[0].name);
TEST_ASSERT_EQUAL_STRING("/etc/xce/lic-report.sh", cfg.src[0].command);
TEST_ASSERT_EQUAL_INT(15, cfg.src[0].interval);
TEST_ASSERT_EQUAL_STRING("cloud_gpu_quota", cfg.src[1].name);
TEST_ASSERT_EQUAL_STRING("/etc/xce/quota-report.sh", cfg.src[1].command);
TEST_ASSERT_EQUAL_INT(0, cfg.src[1].interval); /* no override */
esd_config_fini(&cfg);
}
static void test_config_load_missing_file_is_enoent(void)
{
esd_config_t cfg;
int rc;
memset(&cfg, 0, sizeof(cfg));
rc = esd_config_load_path(g_tmpdir, "no-such-file.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(-ENOENT, rc);
}
static void test_config_load_bad_shape_is_einval(void)
{
esd_config_t cfg;
int rc;
memset(&cfg, 0, sizeof(cfg));
/* a source without a name is structurally invalid -- the controller
* would refuse the push and the operator would have no idea which
* entry died; fail loudly at load. */
write_file(g_tmpdir, "esd.yaml",
"controller: 172.31.1.30:9990\n"
"resources:\n"
" - collector: script\n"
" command: /bin/true\n");
rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
/* a source without a command can never report -- same rule. */
memset(&cfg, 0, sizeof(cfg));
write_file(g_tmpdir, "esd.yaml",
"controller: 172.31.1.30:9990\n"
"resources:\n"
" - name: lic\n"
" collector: script\n");
rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
}
/* ── script collector output parsing (parse contract, not the spawn) ── */
static void test_parse_output_bare_number(void)
{
double v = esd_parse_script_output("4\n");
TEST_ASSERT_FALSE(isnan(v));
TEST_ASSERT_EQUAL_FLOAT((float)4.0, (float)v);
}
static void test_parse_output_name_value(void)
{
/* elim-style "<name> <value>" -- name may mismatch; the source it is
* configured for is authoritative, the name is decorative. */
double v = esd_parse_script_output("Verilog_XL 7\n");
TEST_ASSERT_FALSE(isnan(v));
TEST_ASSERT_EQUAL_FLOAT((float)7.0, (float)v);
}
static void test_parse_output_whitespace_and_negative(void)
{
double v = esd_parse_script_output(" \t 3.5 \n");
TEST_ASSERT_FALSE(isnan(v));
TEST_ASSERT_EQUAL_FLOAT((float)3.5, (float)v);
v = esd_parse_script_output("lic -1\n");
TEST_ASSERT_FALSE(isnan(v));
TEST_ASSERT_EQUAL_FLOAT((float)-1.0, (float)v);
}
static void test_parse_output_garbage_is_nan(void)
{
double v;
v = esd_parse_script_output("");
TEST_ASSERT_TRUE(isnan(v));
v = esd_parse_script_output("\n");
TEST_ASSERT_TRUE(isnan(v));
v = esd_parse_script_output("no digits here\n");
TEST_ASSERT_TRUE(isnan(v));
v = esd_parse_script_output(NULL);
TEST_ASSERT_TRUE(isnan(v));
}
/* ── Phase 2: stale_limit config parsing ────────────────────────────── */
static void test_config_stale_limit_parse(void)
{
esd_config_t cfg;
int rc;
memset(&cfg, 0, sizeof(cfg));
/* per-source override wins; a source without one takes the daemon
* default; the daemon default itself is optional with a sane built-in
* (design.md 3.4: stale_limit in esd.yaml, fail-open keep-old-value). */
write_file(g_tmpdir, "esd.yaml",
"controller: 172.31.1.30:9990\n"
"poll_interval: 10\n"
"stale_limit: 200\n"
"resources:\n"
" - name: lic_a\n"
" collector: script\n"
" command: /bin/a.sh\n"
" stale_limit: 60\n"
" - name: lic_b\n"
" collector: script\n"
" command: /bin/b.sh\n");
rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(200, cfg.stale_limit);
TEST_ASSERT_EQUAL_INT(60, cfg.src[0].stale_limit);
TEST_ASSERT_EQUAL_INT(0, cfg.src[1].stale_limit); /* 0 = use default */
esd_config_fini(&cfg);
}
/* ── Phase 2: stale predicate ───────────────────────────────────────── */
static void test_stale_predicate(void)
{
esd_source_t src;
memset(&src, 0, sizeof(src));
src.name = (char*)"lic";
src.have_value = true;
src.last_value = 3.0;
/* Never report a value -> nothing to call stale; admission treats the
* pool as 0 through the normal NO_RESOURCE path. */
src.last_ok_time = 0;
TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60));
/* Fresh success (10s ago, limit 60) -> not stale. */
src.last_ok_time = 90;
TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60));
/* Exactly at the limit -> not stale (strictly-greater rule, so an
* interval == stale_limit config never flaps). */
src.last_ok_time = 40;
TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60));
/* Past the limit -> stale. */
src.last_ok_time = 39;
TEST_ASSERT_TRUE(esd_source_is_stale(&src, 100, 60));
/* A source whose own stale_limit override is set uses it. */
src.stale_limit = 10;
TEST_ASSERT_TRUE(esd_source_is_stale(&src, 100, 60));
src.last_ok_time = 95;
TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60));
/* last_ok_time newer than `now` (clock skew between test harness and
* caller) must not read as stale. */
src.last_ok_time = 105;
TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60));
}
/* ── Phase 2: status file ───────────────────────────────────────────── */
static void test_status_write_basic(void)
{
esd_config_t cfg;
esd_stats_t stats;
char path[512];
char* text;
time_t now = time(NULL);
char expect_lvt[128];
memset(&cfg, 0, sizeof(cfg));
cfg.nsrc = 1;
cfg.src = (esd_source_t*)calloc(1, sizeof(esd_source_t));
TEST_ASSERT_NOT_NULL(cfg.src);
cfg.src[0].name = (char*)"extlic";
cfg.src[0].have_value = true;
cfg.src[0].last_value = 3.0;
/* fresh success: last good value 5s ago, limit default 150 -> not
* stale (the marks-stale test below covers the other side) */
cfg.src[0].last_ok_time = now - 5;
cfg.stale_limit = ESD_STALE_LIMIT_DEFAULT;
memset(&stats, 0, sizeof(stats));
stats.started_at = 1000;
stats.pushes_ok = 7;
stats.pushes_fail = 1;
snprintf(path, sizeof(path), "%s/status.json", g_tmpdir);
TEST_ASSERT_EQUAL_INT(0, esd_status_write(path, &cfg, &stats));
text = slurp(path);
TEST_ASSERT_NOT_NULL(text);
/* The fields xceadmin extres status reads, and the stale flag driven
* by the same predicate the tests above pin. */
TEST_ASSERT_TRUE(strstr(text, "\"daemon\": \"xceesd\"") != NULL);
TEST_ASSERT_TRUE(strstr(text, "\"name\": \"extlic\"") != NULL);
snprintf(expect_lvt, sizeof(expect_lvt), "\"last_value\": %.17g", 3.0);
TEST_ASSERT_TRUE(strstr(text, expect_lvt) != NULL);
TEST_ASSERT_TRUE(strstr(text, "\"last_ok_time\": ") != NULL);
TEST_ASSERT_TRUE(strstr(text, "\"stale\": false") != NULL);
TEST_ASSERT_TRUE(strstr(text, "\"pushes_ok\": 7") != NULL);
TEST_ASSERT_TRUE(strstr(text, "\"pushes_fail\": 1") != NULL);
free(text);
free(cfg.src);
}
static void test_status_write_marks_stale(void)
{
esd_config_t cfg;
esd_stats_t stats;
char path[512];
char* text;
time_t now = time(NULL);
memset(&cfg, 0, sizeof(cfg));
cfg.nsrc = 1;
cfg.src = (esd_source_t*)calloc(1, sizeof(esd_source_t));
TEST_ASSERT_NOT_NULL(cfg.src);
cfg.src[0].name = (char*)"extlic";
cfg.src[0].have_value = true;
cfg.src[0].last_value = 2.0;
/* last success 1 hour ago, limit 150s -> stale. */
cfg.src[0].last_ok_time = now - 3600;
cfg.src[0].stale_limit = 150;
memset(&stats, 0, sizeof(stats));
snprintf(path, sizeof(path), "%s/status.json", g_tmpdir);
TEST_ASSERT_EQUAL_INT(0, esd_status_write(path, &cfg, &stats));
text = slurp(path);
TEST_ASSERT_NOT_NULL(text);
TEST_ASSERT_TRUE(strstr(text, "\"stale\": true") != NULL);
free(text);
free(cfg.src);
}
/* ── bug-1978: the reconnect-flag contract ───────────────────────────── */
static void test_reconnect_sets_both_rejoin_and_full_push(void)
{
g_needs_rejoin = 0;
g_needs_full_push = 0;
TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x1, NULL));
/* BOTH flags: the re-join must happen before any re-push (an un-joined
* push is refused -EPERM on the fresh endpoint), and the re-push must
* still happen (the fresh controller knows no values). The bug-1978
* defect cleared only g_needs_full_push -- a daemon that re-pushed onto
* an un-joined HostRole_client endpoint, every push -EPERM forever. */
TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin);
TEST_ASSERT_EQUAL_INT(1, g_needs_full_push);
}
static void test_reconnect_flags_recover_from_any_previous_state(void)
{
/* The handler is level-set, not level-toggle: regardless of what a
* previous (already consumed) reconnect left behind, a new callback
* invocation re-arms both flags. */
g_needs_rejoin = 0;
g_needs_full_push = 1; /* stale leftover from an unconsumed cycle */
TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x2, NULL));
TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin);
TEST_ASSERT_EQUAL_INT(1, g_needs_full_push);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_config_load_full_file);
RUN_TEST(test_config_load_missing_file_is_enoent);
RUN_TEST(test_config_load_bad_shape_is_einval);
RUN_TEST(test_parse_output_bare_number);
RUN_TEST(test_parse_output_name_value);
RUN_TEST(test_parse_output_whitespace_and_negative);
RUN_TEST(test_parse_output_garbage_is_nan);
RUN_TEST(test_config_stale_limit_parse);
RUN_TEST(test_stale_predicate);
RUN_TEST(test_status_write_basic);
RUN_TEST(test_status_write_marks_stale);
RUN_TEST(test_reconnect_sets_both_rejoin_and_full_push);
RUN_TEST(test_reconnect_flags_recover_from_any_previous_state);
return UNITY_END();
}

13
src/frontend/apiserver/backend/convert.go
View file

@ -286,12 +286,13 @@ func cQueueToModel(queue *C.xce_queue_t) *models.QueueInfo {
// cResourceToModel extracts C xce_resource_t fields and maps them to ResourceDefinitionInfo.
func cResourceToModel(res *C.xce_resource_t) *models.ResourceDefinitionInfo {
return rawResourceToModel(rawResourceData{
name: goString(res.name),
desc: goString(res.desc),
valType: int(res.valtype),
ordType: int(res.ordtype),
flags: int(res.flags),
interval: int(res.interval),
name: goString(res.name),
desc: goString(res.desc),
valType: int(res.valtype),
ordType: int(res.ordtype),
flags: int(res.flags),
interval: int(res.interval),
lastReport: int64(res.last_report),
})
}

14
src/frontend/apiserver/backend/convert_pure.go
View file

@ -578,12 +578,13 @@ type rawQueueData struct {
// rawResourceData mirrors the C xce_resource_t fields used by the Go model.
type rawResourceData struct {
name string
desc string
valType int
ordType int
flags int
interval int
name string
desc string
valType int
ordType int
flags int
interval int
lastReport int64
}
// interactiveURLFromExdata extracts the interactive session URL from a job
@ -828,6 +829,7 @@ func rawResourceToModel(r rawResourceData) *models.ResourceDefinitionInfo {
OrderType: r.ordType,
Flag: r.flags,
Interval: r.interval,
LastReport: r.lastReport,
}
}

18
src/frontend/apiserver/backend/handler.go
View file

@ -5,6 +5,7 @@ import (
"apiserver/store"
logger "apiserver/utils"
"sync"
"time"
"go.uber.org/zap"
)
@ -249,6 +250,23 @@ func (h *DefaultHandler) OnResourceLocationUpdated(res *models.ResourceDistribut
key := res.ResourceName + ":" + res.Location
store.ResourceLocations.Put(key, *res)
// external-license-resources Phase 2: the definition store is fed ONLY
// by the join-time config push, whose snapshot carries whatever the
// config tree had at push time -- so an EXTERNAL pool's last_report
// would stay 0 ("never") for the whole apiserver process lifetime even
// while the extres daemon pushes normally. A shared-value push (the
// daemon's landing shape: Location "") for an EXTERNAL definition is
// exactly an accepted external report at the controller, so refresh the
// stamp here. Host-sourced values (Location != "") and non-EXTERNAL
// resources must not touch it: last_report means "external value last
// accepted", and worker reports are a different freshness domain.
if res.Location == "" {
if def, ok := store.Resources.Get(res.ResourceName); ok && (def.Flag&models.EXTERNAL) != 0 {
def.LastReport = time.Now().Unix()
store.Resources.Put(def.ResourceName, def)
}
}
// resource-value-verification task 4.2 (Requirements 5.1, 5.2, 5.4): trace
// the stored value at its point of application, mirroring the scheduler's
// sibling trace log added in task 4.1 (core_events.c). One typed "value"

69
src/frontend/apiserver/backend/integration_test.go
View file

@ -8,6 +8,7 @@ import (
"apiserver/utils"
"sync/atomic"
"testing"
"time"
"go.uber.org/zap"
)
@ -516,3 +517,71 @@ func TestIntegration_CrossFeature_SequentialConfigPushes(t *testing.T) {
t.Errorf("heartbeatIntervalSecs = %d, want 60", hb)
}
}
// --- 5.6 Phase 2 (external-license-resources): a shared-value push refreshes
// the stored EXTERNAL definition's last_report. The definition store is fed
// only by the join-time config push, whose snapshot carries whatever the
// config tree had at push time — without this refresh the CLI's LAST_UPDATED
// column would show "never" for the whole apiserver process lifetime even
// while the extres daemon pushes normally. The daemon landing shape for a
// cluster-level value is Location=="" (host:"" on the wire). Worker-sourced
// host values must NOT touch the stamp. ---
func TestIntegration_ExternalSharedPushRefreshesDefinitionLastReport(t *testing.T) {
resetIntegrationStores()
h := &DefaultHandler{}
// config push seeds the definition store: extlic is EXTERNAL (0x10),
// mem is a plain host-based resource.
dispatchConfigPush(h, nil, []models.ResourceDefinitionInfo{
{ResourceName: "extlic", Flag: 0x80 | 0x10, LastReport: 0},
{ResourceName: "mem", Flag: 0x80, LastReport: 0},
}, nil, &models.ClusterInfo{ClusterName: "c1"})
before := time.Now().Unix()
// daemon-shaped shared push for extlic (Location "")
h.OnResourceLocationUpdated(&models.ResourceDistributionInfo{
ResourceName: "extlic", Location: "", NumericValue: 4.0,
ValueType: int(models.NUMERIC), IsDynamic: true,
})
got, ok := store.Resources.Get("extlic")
if !ok {
t.Fatal("extlic definition should exist after config push")
}
if got.LastReport < before {
t.Errorf("extlic LastReport = %d, want >= %d (refreshed by the shared push)", got.LastReport, before)
}
// host-sourced push for mem must not stamp its definition
h.OnResourceLocationUpdated(&models.ResourceDistributionInfo{
ResourceName: "mem", Location: "node01", NumericValue: 65536.0,
ValueType: int(models.NUMERIC), IsDynamic: true,
})
gotMem, _ := store.Resources.Get("mem")
if gotMem.LastReport != 0 {
t.Errorf("mem LastReport = %d, want 0 (host values never stamp)", gotMem.LastReport)
}
}
// The refresh only applies to EXTERNAL definitions: a shared push for a
// non-EXTERNAL resource (e.g. a dynamic shared pool reported by workers)
// leaves last_report untouched.
func TestIntegration_SharedPushDoesNotStampNonExternalDefinition(t *testing.T) {
resetIntegrationStores()
h := &DefaultHandler{}
dispatchConfigPush(h, nil, []models.ResourceDefinitionInfo{
{ResourceName: "plainshared", Flag: 0x80 | 0x08, LastReport: 0},
}, nil, &models.ClusterInfo{ClusterName: "c1"})
h.OnResourceLocationUpdated(&models.ResourceDistributionInfo{
ResourceName: "plainshared", Location: "", NumericValue: 2.0,
ValueType: int(models.NUMERIC), IsDynamic: true,
})
got, _ := store.Resources.Get("plainshared")
if got.LastReport != 0 {
t.Errorf("plainshared LastReport = %d, want 0 (not EXTERNAL)", got.LastReport)
}
}

15
src/frontend/apiserver/models/resources.go
View file

@ -67,6 +67,10 @@ var RESOURCE_DEFINITION_DEFAULT_COLUMN = []string{
"order_type",
"flag",
"interval",
// external-license-resources Phase 2: SelectFields drops row keys not
// listed here, so the external value's last-report timestamp must be a
// default column or it never reaches API clients / the CLI.
"last_report",
}
var ResourceFieldMappings = map[string]string{
@ -77,8 +81,12 @@ var ResourceFieldMappings = map[string]string{
var ResourcesAvailableTypes = []string{"static", "dynamic"}
const (
BUILTIN = 0x01
DYNAMIC = 0x02
BUILTIN = 0x01
DYNAMIC = 0x02
// external-license-resources: EXTERNAL=0x10 mirrors C's XCE_RESF_EXTERNAL
// (src/include/xce/types.h) — the value comes from an external daemon,
// not from workers.
EXTERNAL = 0x10
RELEASE = 0x20
CONSUMABLE = 0x80
)
@ -132,6 +140,9 @@ type ResourceDefinitionInfo struct {
OrderType int `json:"order_type"`
Flag int `json:"flag"`
Interval int `json:"interval"`
// external-license-resources Phase 2: epoch seconds the external value
// was last accepted by the controller (0 = never / not external).
LastReport int64 `json:"last_report"`
}
var RESOURCE_DISTRIBUTION_SUPPORT_SORT = []string{

47
src/frontend/apiserver/models/resources_test.go Normal file
View file

@ -0,0 +1,47 @@
package models
import (
"encoding/json"
"testing"
)
// external-license-resources Phase 2: last_report must be part of the default
// column list for /resources/definition — SelectFields drops any row key not
// listed there, so without it the API response (and the CLI LAST_UPDATED
// column) never sees the timestamp even though the model carries it.
func TestResourceDefinitionDefaultColumnIncludesLastReport(t *testing.T) {
found := false
for _, col := range RESOURCE_DEFINITION_DEFAULT_COLUMN {
if col == "last_report" {
found = true
break
}
}
if !found {
t.Errorf("RESOURCE_DEFINITION_DEFAULT_COLUMN = %v, missing \"last_report\"", RESOURCE_DEFINITION_DEFAULT_COLUMN)
}
}
// ResourceDefinitionInfo must serialize last_report with the json tag the CLI
// model (xcecli/models/resource.go) unmarshals.
func TestResourceDefinitionInfoJSONCarriesLastReport(t *testing.T) {
info := ResourceDefinitionInfo{
ResourceName: "extlic",
LastReport: 1800000000,
}
data, err := json.Marshal(info)
if err != nil {
t.Fatalf("json.Marshal: %v", err)
}
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
t.Fatalf("json.Unmarshal: %v", err)
}
v, ok := m["last_report"]
if !ok {
t.Fatalf("serialized body %s has no last_report key", data)
}
if v.(float64) != 1800000000 {
t.Errorf("last_report = %v, want 1800000000", v)
}
}

14
src/frontend/xcecli/cmd/jobs/submit.go
View file

@ -896,10 +896,20 @@ func (s *stringValue) Set(val string) error {
func (s *stringValue) Type() string { return "string" }
func (js *JobSubmitter) getEnvironmentVariables() []string {
// The downstream wire path (apiserver ConvertStringToJsonb) splits each
// entry at the FIRST '=' (strings.SplitN 2), so values containing '='
// (A=b=c -> A = "b=c") round-trip fine -- dropping them here used to be
// a silent data loss, not a necessary restriction. A nameless entry
// (=value) cannot become a map key, so it is the one case that must be
// rejected -- say so instead of dropping it silently.
var envVars []string
for _, env := range js.subEnvVars {
// Skip environment variables with multiple equal signs
if strings.Count(env, "=") > 1 {
if strings.HasPrefix(env, "=") {
fmt.Printf("warning: ignoring environment variable '%s': empty name -- name=value entries only\n", env)
continue
}
if !strings.Contains(env, "=") {
fmt.Printf("warning: ignoring environment variable '%s': no '=' -- name=value entries only\n", env)
continue
}
envVars = append(envVars, env)

44
src/frontend/xcecli/cmd/jobs/submit_test.go
View file

@ -1,8 +1,10 @@
package jobs
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
@ -133,6 +135,48 @@ func TestSubmitHelpShowsArrayForms(t *testing.T) {
}
}
// getEnvironmentVariables must keep entries whose value contains '=' (the
// apiserver splits at the FIRST '=', so A=b=c is A -> "b=c") -- dropping
// them used to be silent data loss. A nameless or '='-less entry cannot
// become a map key, so it is warned about and dropped.
func TestGetEnvironmentVariables_KeepsMultiEqualSign(t *testing.T) {
js := NewJobSubmitter(nil)
js.subEnvVars = []string{"A=1", "B=b=c", "C=", "=novalue", "novalue"}
r, w, pipeErr := os.Pipe()
if pipeErr != nil {
t.Fatalf("pipe: %v", pipeErr)
}
old := os.Stdout
os.Stdout = w
done := make(chan string)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, r)
done <- buf.String()
}()
got := js.getEnvironmentVariables()
os.Stdout = old
_ = w.Close()
warn := <-done
want := []string{"A=1", "B=b=c", "C="}
if len(got) != len(want) {
t.Fatalf("env = %q, want %q", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("env[%d] = %q, want %q", i, got[i], want[i])
}
}
if !strings.Contains(warn, "=novalue") || !strings.Contains(warn, "warning:") {
t.Errorf("warning output = %q, want it to name the nameless entry", warn)
}
if !strings.Contains(warn, "novalue") {
t.Errorf("warning output = %q, want it to name the '='-less entry", warn)
}
}
// Attach marks the terminal-attach kind of TTY job so the worker may reap
// ttyd when no client ever connects; --browse keeps the web-viewed
// semantics and must not set it.

7
src/frontend/xcecli/common/output/page_print_resources.go
View file

@ -3,6 +3,7 @@ package output
import (
"fmt"
"strings"
"time"
"xcecli/models"
)
@ -143,6 +144,8 @@ func (p *ResourcesPagePrinter) initColumnWidthsForDef(resourcesDef *models.Resou
width = len(resourceDef.ResourceType)
case models.HEADER_RESOURCE_DEF_DESC:
width = len(resourceDef.Description)
case models.HEADER_RESOURCE_DEF_LAST_REPORT:
width = len(formatJobTime(time.Unix(resourceDef.LastReport, 0)))
}
if !p.wideFormat && width > 15 {
@ -270,6 +273,8 @@ func (p *ResourcesPagePrinter) formatRowsForDef(resourcesDef *models.ResourceDef
row = appendOrDefault(row, FormatStringField(builtinValue, width, truncatFlag, true), "-")
case models.HEADER_RESOURCE_DEF_INTERVAL:
row = appendOrDefault(row, fmt.Sprintf("%"+align+"*d", width, resourceDef.Interval), "0")
case models.HEADER_RESOURCE_DEF_LAST_REPORT:
row = appendOrDefault(row, FormatStringField(fmtUnixTime(resourceDef.LastReport), width, truncatFlag, true), "-")
}
}
rows = append(rows, row)
@ -354,6 +359,8 @@ func printResourcesDefVerboseOutput(resourceDef *models.ResourceDefResponse) {
val = releaseValue
case models.HEADER_RESOURCE_DEF_INTERVAL:
val = fmt.Sprintf("%d", resourceDef.Interval)
case models.HEADER_RESOURCE_DEF_LAST_REPORT:
val = fmtUnixTime(resourceDef.LastReport)
}
widths1[i] = max(len(h), len(val))
row1 = append(row1, FormatStringField(val, widths1[i], false, true))

46
src/frontend/xcecli/models/resource.go
View file

@ -7,17 +7,18 @@ import (
)
const (
HEADER_RESOURCE_NAME = "RESOURCE"
HEADER_RESOURCE_VALUE = "VALUE"
HEADER_RESOURCE_LOC = "LOCATION"
HEADER_RESOURCE_DEF_TYPE = "TYPE"
HEADER_RESOURCE_DEF_ORDER = "ORDER"
HEADER_RESOURCE_DEF_BUILTIN = "BUILTIN"
HEADER_RESOURCE_DEF_CONSUMABLE = "CONSUMABLE"
HEADER_RESOURCE_DEF_DYNAMIC = "DYNAMIC"
HEADER_RESOURCE_DEF_RELEASE = "RELEASE"
HEADER_RESOURCE_DEF_INTERVAL = "INTERVAL"
HEADER_RESOURCE_DEF_DESC = "DESCRIPTION"
HEADER_RESOURCE_NAME = "RESOURCE"
HEADER_RESOURCE_VALUE = "VALUE"
HEADER_RESOURCE_LOC = "LOCATION"
HEADER_RESOURCE_DEF_TYPE = "TYPE"
HEADER_RESOURCE_DEF_ORDER = "ORDER"
HEADER_RESOURCE_DEF_BUILTIN = "BUILTIN"
HEADER_RESOURCE_DEF_CONSUMABLE = "CONSUMABLE"
HEADER_RESOURCE_DEF_DYNAMIC = "DYNAMIC"
HEADER_RESOURCE_DEF_RELEASE = "RELEASE"
HEADER_RESOURCE_DEF_INTERVAL = "INTERVAL"
HEADER_RESOURCE_DEF_LAST_REPORT = "LAST_UPDATED"
HEADER_RESOURCE_DEF_DESC = "DESCRIPTION"
)
var ResourceDistFieldsMap = map[string]string{
@ -28,21 +29,23 @@ var ResourceDistFieldsMap = map[string]string{
// ResourceDefFieldsMap maps resource definition field headers to API field names
var ResourceDefFieldsMap = map[string]string{
HEADER_RESOURCE_NAME: "res_name",
HEADER_RESOURCE_DEF_DESC: "description",
HEADER_RESOURCE_DEF_TYPE: "type",
HEADER_RESOURCE_DEF_ORDER: "order",
HEADER_RESOURCE_DEF_BUILTIN: "builtin",
HEADER_RESOURCE_DEF_CONSUMABLE: "consumable",
HEADER_RESOURCE_DEF_DYNAMIC: "dynamic",
HEADER_RESOURCE_DEF_RELEASE: "release",
HEADER_RESOURCE_DEF_INTERVAL: "interval",
HEADER_RESOURCE_NAME: "res_name",
HEADER_RESOURCE_DEF_DESC: "description",
HEADER_RESOURCE_DEF_TYPE: "type",
HEADER_RESOURCE_DEF_ORDER: "order",
HEADER_RESOURCE_DEF_BUILTIN: "builtin",
HEADER_RESOURCE_DEF_CONSUMABLE: "consumable",
HEADER_RESOURCE_DEF_DYNAMIC: "dynamic",
HEADER_RESOURCE_DEF_RELEASE: "release",
HEADER_RESOURCE_DEF_INTERVAL: "interval",
HEADER_RESOURCE_DEF_LAST_REPORT: "last_report",
}
var ResourceDefDetailHeaders = []string{
HEADER_RESOURCE_DEF_TYPE,
HEADER_RESOURCE_DEF_ORDER,
HEADER_RESOURCE_DEF_INTERVAL,
HEADER_RESOURCE_DEF_LAST_REPORT,
HEADER_RESOURCE_DEF_BUILTIN,
HEADER_RESOURCE_DEF_DYNAMIC,
HEADER_RESOURCE_DEF_RELEASE,
@ -177,6 +180,7 @@ type ResourceDef struct {
OrderType string `json:"-"` // populated from order_type int
Flag map[string]interface{} `json:"flag"`
Interval int `json:"interval"`
LastReport int64 `json:"last_report"` // Unix seconds of the last external value report (0 = never)
}
func (r *ResourceDef) UnmarshalJSON(data []byte) error {
@ -187,6 +191,7 @@ func (r *ResourceDef) UnmarshalJSON(data []byte) error {
OrderType string `json:"order_type"`
Flag map[string]interface{} `json:"flag"`
Interval int `json:"interval"`
LastReport int64 `json:"last_report"`
}
var a Alias
if err := UnmarshalOrderMapRow(data, &a); err != nil {
@ -198,5 +203,6 @@ func (r *ResourceDef) UnmarshalJSON(data []byte) error {
r.OrderType = a.OrderType
r.Flag = a.Flag
r.Interval = a.Interval
r.LastReport = a.LastReport
return nil
}

14
src/include/xce/cluster_def.h
View file

@ -27,6 +27,7 @@
#define N_CONTROLLER_MAX (16)
#define N_APISERVER_MAX (16)
#define N_EXTRES_MAX (8)
enum {
XCE_CONTROLLER_HOOK_JOB = 0,
@ -96,6 +97,18 @@ typedef struct {
endpoint_t* ep;
} cluster_apiserver_t;
/* external-license-resources Phase 1: connected extres daemons (xceesd).
* Registered at HostRole_extres join like the apiserver table -- endpoint +
* name only, no workermgr slot (the daemon is not a schedulable host). The
* controller never pushes TO the daemon (the daemon pulls nothing and
* re-pushes its full value set on every reconnect), so this table exists
* for leave-cleanup and future admin surface (xceadmin extres status). */
typedef struct {
mylock_t lock;
char* name;
endpoint_t* ep;
} cluster_extres_t;
typedef struct cluster_s {
myconf_t* myconf;
xce_peerinfo_t* myhost; /* points at myhost_info once initialised */
@ -108,6 +121,7 @@ typedef struct cluster_s {
zarray_t sharedinst;
controller_t controllers[N_CONTROLLER_MAX];
cluster_apiserver_t apiservers[N_APISERVER_MAX];
cluster_extres_t extresds[N_EXTRES_MAX];
int myidx;
int master;
int ncontrollers;

3
src/lib/request.c
View file

@ -1414,7 +1414,8 @@ int xce_join_req(endpoint_t* ep, int role, hostattrs_t* attrs)
if ((role != HostRole_client) &&
(role != HostRole_worker) &&
(role != HostRole_apiserver) &&
(role != HostRole_scheduler)) {
(role != HostRole_scheduler) &&
(role != HostRole_extres)) {
return -EINVAL;
}
/* clang-format on */

17
src/lib/resreq.c
View file

@ -473,6 +473,23 @@ int parse_resreq_primary(xce_primary_resreq_t* primary, char* resreqString,
if (data.selectString != NULL && data.selectString[0] != '\0') {
rc = parse_select(primary, data.selectString, resmgr, errmsg);
if (rc < 0) {
/* -ENOENT (select names an unregistered resource) is tolerated
* by the resreq grammar -- xce_resreq.peg keeps the primary
* without the clause (jobs rely on it, the scheduler's own
* "type == any" default among them when no "type" resource is
* registered). That tolerance is silent, and the early return
* below also skips every clause that follows in this primary,
* so say so: a job whose select and rusage are both gone is
* scheduled with none of the constraints it asked for.
* -EINVAL/-ENOMEM failures need no WARN here -- the grammar
* turns them into a whole-resreq rejection the caller logs. */
if (rc == -ENOENT) {
WARN("resreq: select '%s' references an unregistered "
"resource -- dropping the select and skipping any "
"rusage[] clause of this primary; the job is scheduled "
"without these constraints",
data.selectString);
}
kill_resreq_data(&data);
return rc;
}

28
src/lib/testsuite/test_select.c
View file

@ -128,6 +128,33 @@ test_select_parse_unknown_resource(void)
sel_optimize_test_reset();
}
static void
test_select_unknown_resource_skips_rusage(void)
{
/* The -ENOENT tolerance (bug-1887) drops the select AND the primary's
* rusage clause with it (parse_resreq_primary returns before parsing
* rusage) -- now with a WARN. Assert the documented drop semantics:
* rc carries -ENOENT and no rusage survived, so callers can never
* mistake the tolerated failure for a fully-parsed primary. */
char * errmsg = NULL;
xce_primary_resreq_t resreq;
var resmgr = make_test_resmgr();
sel_optimize_set(0);
int rc = parse_resreq_primary(&resreq, "unknownres==linux rusage[mem=2]",
resmgr, &errmsg);
TEST_ASSERT_EQUAL_INT(-ENOENT, rc);
TEST_ASSERT_NULL(resreq.select);
TEST_ASSERT_EQUAL_INT(0, resreq.rusages.count);
kill_primary_resreq(&resreq);
FREEUP(errmsg);
free_test_resmgr(resmgr);
sel_optimize_test_reset();
}
static void
test_select_parse_type_mismatch(void)
{
@ -2346,6 +2373,7 @@ test_select(void)
RUN_TEST(test_select_parse_valid_expr);
RUN_TEST(test_select_parse_valid_any);
RUN_TEST(test_select_parse_unknown_resource);
RUN_TEST(test_select_unknown_resource_skips_rusage);
RUN_TEST(test_select_parse_type_mismatch);
RUN_TEST(test_select_input_len_overflow);

3
src/meson.build
View file

@ -76,6 +76,7 @@ if component != 'xcecli'
subdir('controller')
subdir('worker')
subdir('autoscaler')
subdir('esd')
subdir('scheduler')
subdir('frontend/apiserver')
subdir('xceadmin')
@ -88,7 +89,7 @@ if component != 'xcecli'
find_program('python3'),
args: [meson.project_source_root() / '..' / 'scripts' / 'check-no-fork.py',
meson.project_source_root() / '..'],
suite: ['lib', 'controller', 'scheduler', 'worker'])
suite: ['lib', 'controller', 'scheduler', 'worker', 'esd'])
configure_file(
input: 'tools/appimage/entry.sh',

48
src/scheduler/src/core_events.c
View file

@ -1184,10 +1184,16 @@ int xce_core_push_events(xce_sched_core_t* core, xce_core_event_t* events,
if ((key_find(core->resmgr, res->name, (void**)&found)
== 0)
&& (found != NULL)) {
found->valtype = res->valtype;
found->ordtype = res->ordtype;
found->flags = res->flags;
found->interval = res->interval;
found->valtype = res->valtype;
found->ordtype = res->ordtype;
found->flags = res->flags;
found->interval = res->interval;
/* external-license-resources Phase 2: the extres
* freshness stamp (controller stamps it at each
* accepted push); resobj.c's makeResource excludes
* last_report from definition comparison, so this
* is metadata, not an identity change. */
found->last_report = res->last_report;
} else {
WARN("%s RESOURCES_PUSH: makeResource failed for "
"%s and lookup not found",
@ -1234,16 +1240,27 @@ int xce_core_push_events(xce_sched_core_t* core, xce_core_event_t* events,
}
for (r = 0; r < deltas->count; r++) {
xce_resval_t* rv = deltas->vals[r];
xce_sched_worker_t* w = NULL;
xce_resource_t* res = NULL;
xce_resval_t* rv = deltas->vals[r];
xce_sched_worker_t* w = NULL;
xce_resource_t* res = NULL;
int is_shared;
if (rv == NULL) {
continue;
}
w = sched_get_worker(core, rv->host);
if (w == NULL) {
/* external-license-resources Phase 1: a NULL/"" host marks a
* SHARED (cluster-level) value -- the extres daemon pushes
* license free counts this way (worker pushes always name a
* host). Land it in core->sharedInst, mirroring
* snapshot_load()'s host==NULL/"" precedent (bug-1923).
* Before this rule the empty host went through
* sched_get_worker() -> NULL -> WARN-skip, so daemon-pushed
* shared values never reached the scheduler. */
is_shared = (rv->host == NULL) || (rv->host[0] == '\0');
w = is_shared ? NULL : sched_get_worker(core, rv->host);
if (!is_shared && (w == NULL)) {
WARN("%s RESVALS_DELTA: unknown worker %s, skipping",
__func__, rv->host);
continue;
@ -1256,6 +1273,19 @@ int xce_core_push_events(xce_sched_core_t* core, xce_core_event_t* events,
continue;
}
if (is_shared) {
/* bug-1585 ownership rule: dup_val(), not a shallow
* rv->val store -- same hazard as the per-worker store
* below. */
zarray_put(&(core->sharedInst), res->idx, dup_val(rv->val));
if ((rv->val != NULL)
&& (rv->val->which == Value_u_fvalue)) {
TRACE("resval applied host=%s resource=%s value=%f",
"(shared)", res->name, rv->val->u.fvalue);
}
continue;
}
/* bug-1585: dup_val(), not a shallow `rv->val` store -- see
* this file's own top-of-file comment on the `dup_val`
* extern declaration for the full ownership hazard this

37
src/scheduler/src/resource.c
View file

@ -159,11 +159,26 @@ static int unsatisfy_(xce_sched_core_t* core, int nres, int* idxs, float* total,
count = 0;
for (i = 0; i < nres; i++) {
float uncomit = uncomit_at_(uncomitted, idxs[i]);
float uncomit = uncomit_at_(uncomitted, idxs[i]);
xce_resource_t* r = NULL;
int is_external;
if (total[i] < inuse[i] + uncomit + wanted[i]) {
TRACE("%s resource %d total %f inuse %f uncomit %f wanted %f",
__func__, idxs[i], total[i], inuse[i], uncomit, wanted[i]);
/* external-license-resources: an EXTERNAL resource's pushed value
* is already a FREE count (license server's remaining, net of both
* external consumers and our own checked-out jobs), so re-subtracting
* the scheduler's own usage bookkeeping (sharedInUse / w->resInUse)
* would count our usage twice. Skip the inuse term for these:
* EXTERNAL: total >= uncomit + wanted
* non-EXTERNAL: total >= inuse + uncomit + wanted
* (r->flags is fetched here for the trace line's resource name too --
* the flag was already reachable in this loop's vget). */
vget(core->resmgr, idxs[i], (void**)&r);
is_external = (r != NULL) && (r->flags & XCE_RESF_EXTERNAL);
if (total[i] < ((is_external ? 0.0f : inuse[i]) + uncomit + wanted[i])) {
TRACE("%s resource %d total %f inuse %f uncomit %f wanted %f%s",
__func__, idxs[i], total[i], is_external ? 0.0f : inuse[i],
uncomit, wanted[i], is_external ? " (external, inuse skipped)" : "");
count++;
}
@ -175,13 +190,17 @@ static int unsatisfy_(xce_sched_core_t* core, int nres, int* idxs, float* total,
j = 0;
for (i = 0; i < nres; i++) {
float uncomit = uncomit_at_(uncomitted, idxs[i]);
float uncomit = uncomit_at_(uncomitted, idxs[i]);
xce_resource_t* r = NULL;
int is_external;
if (total[i] < inuse[i] + uncomit + wanted[i]) {
xce_resource_t* r = NULL;
float need = inuse[i] + uncomit + wanted[i] - total[i];
vget(core->resmgr, idxs[i], (void**)&r);
is_external = (r != NULL) && (r->flags & XCE_RESF_EXTERNAL);
if (total[i] < ((is_external ? 0.0f : inuse[i]) + uncomit + wanted[i])) {
float need = (is_external ? 0.0f : inuse[i]) + uncomit + wanted[i]
- total[i];
vget(core->resmgr, idxs[i], (void**)&r);
TRACE("%s unmet resource %d (%s) shortfall %f", __func__, idxs[i],
r ? r->name : "?", need);

BIN
src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay
View file

Binary file not shown.

BIN
src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay
View file

Binary file not shown.

BIN
src/scheduler/testsuite/fixtures/repl_trace_baseline/sim_output.replay
View file

Binary file not shown.

19
src/scheduler/testsuite/meson.build
View file

@ -1634,6 +1634,23 @@ sched_limit_plugin_test = executable('sched_limit_plugin_test',
dependencies: [scheduler_test_deps],
install: false)
# external-license-resources Phase 3 (LSF 策略层): white-box regression
# coverage for the "GLOBAL limit on a GLOBAL|SHARED|EXTERNAL resource"
# shape (the per-project/per-queue/per-user license-quota mapping onto the
# existing limit vres mechanism). It #includes ../src/plugins/limit.c +
# limit_config.c directly (same convention as sched_limit_plugin_test
# above) so it drives the real match_job_global_limit_() -- `static`,
# otherwise unreachable. Unlike test_limit_plugin.c, its get_resvals()/
# put_resvals() are FAITHFUL COPIES of resource.c's (the value path is the
# subject), and the fixture carries a real core->resmgr +
# sharedInst/sharedInUse -- see the file's own header comment.
sched_limit_project_scope_test = executable(
'sched_limit_project_scope_test',
['test_limit_project_scope.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
install: false)
# White-box regression test for bug-1686 (live SIGSEGV in
# fcfs_cursor_enter_queue() <- builtin_fcfs_next_job(), a heap-use-after-free
# on priv->cursor/job_cursor -- raw ordq_item_t* values that persist across
@ -2070,6 +2087,8 @@ test('scheduler snapshot load allocator regression test',
env: ['ASAN_OPTIONS=detect_leaks=0'])
test('scheduler limit plugin test', sched_limit_plugin_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])
test('scheduler limit project scope test', sched_limit_project_scope_test,
suite: 'scheduler', env: ['ASAN_OPTIONS=detect_leaks=0'])
test('scheduler builtin fcfs cursor uaf test',
sched_builtin_fcfs_cursor_uaf_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])

92
src/scheduler/testsuite/test.c
View file

@ -730,6 +730,97 @@ test_core_safe_allocation_chain_driver_parity(void)
destroy(core_sim->resmgr);
}
/*
* external-license-resources Phase 1 (E2), design.md 3.3: the EXTERNAL
* accounting rule. An EXTERNAL resource's "total" as pushed by the extres
* daemon is already a FREE value F (license server's free count, net of
* both external consumers and our own checked-out jobs) -- so re-subtracting
* the scheduler's own usage bookkeeping (sharedInUse / w->resInUse) inside
* the admission formula would double-count X. The unsatisfy_ loop must
* therefore skip the inuse term for resources flagged XCE_RESF_EXTERNAL:
*
* EXTERNAL: total >= uncomit + wanted
* non-EXTERNAL: total >= inuse + uncomit + wanted (unchanged)
*
* Shared leg (sched_check_shared_satisfy_): a SHARED|EXTERNAL resource with
* total=F=4, sharedInUse=3, uncomit=0 must ADMIT wanted=4 (F >= 0+4) even
* though the legacy formula (4 < 3+0+4) would deny it -- and the same shape
* WITHOUT the EXTERNAL flag must still deny (legacy semantics pinned).
*/
static void test_unsatisfy_external_skips_inuse_shared_leg(void)
{
xce_sched_core_t core;
xce_resource_t wire_resource;
xce_resource_t* made;
xce_resource_t* r = NULL;
xvec_t uncomitted;
xvec_idval_t wanted, unmeet;
xce_val_t total, inuse;
int rc;
memset(&core, 0, sizeof(core));
/* makeResource() -> resobj_init() -> makeString() needs the intern table
* (same per-test init as test_meet_host_selection_string_match). */
xstring_table_init(139);
init_test_resmgr(&core);
zarray_init(&core.sharedInst, 4, NULL);
zarray_init(&core.sharedInUse, 4, NULL);
/* Resource "lic": SHARED (cluster-wide pool) + EXTERNAL (free-value
* reporting) -- the shape mapping.py's `location: [external]` produces.
* idx 0. */
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = "lic";
wire_resource.idx = 0;
wire_resource.flags = XCE_RESF_SHARED | XCE_RESF_EXTERNAL;
made = (xce_resource_t*)makeResource(core.resmgr,
&wire_resource);
TEST_ASSERT_NOT_NULL(made);
TEST_ASSERT_TRUE(key_find(core.resmgr, (void*)"lic", (void**)&r) >= 0);
TEST_ASSERT_NOT_NULL(r);
memset(&total, 0, sizeof(total));
total.which = Value_u_fvalue;
total.u.fvalue = 4.0f;
memset(&inuse, 0, sizeof(inuse));
inuse.which = Value_u_fvalue;
inuse.u.fvalue = 3.0f;
zarray_put(&core.sharedInst, 0, &total);
zarray_put(&core.sharedInUse, 0, &inuse);
xvec_init(&uncomitted, 1, sizeof(float));
xvec_idval_init(&wanted, 1);
xvec_idval_store(&wanted, 0, 4.0f);
xvec_idval_init(&unmeet, 1);
/* EXTERNAL: F=4 >= uncomit(0) + wanted(4) -> grant, despite inuse=3. */
rc = sched_check_shared_satisfy(&core, &uncomitted, &wanted, &unmeet);
TEST_ASSERT_EQUAL_INT(1, rc);
xvec_idval_fini(&wanted);
xvec_idval_fini(&unmeet);
/* Same shape without EXTERNAL (plain SHARED pool): the legacy formula
* must still apply -- 4 < 3+0+4 -> deny. Overwrite the flag in place. */
r->flags = XCE_RESF_SHARED;
xvec_init(&uncomitted, 1, sizeof(float));
xvec_idval_init(&wanted, 1);
xvec_idval_store(&wanted, 0, 4.0f);
xvec_idval_init(&unmeet, 1);
rc = sched_check_shared_satisfy(&core, &uncomitted, &wanted, &unmeet);
TEST_ASSERT_EQUAL_INT(0, rc);
xvec_idval_fini(&wanted);
xvec_idval_fini(&unmeet);
xvec_fini(&uncomitted);
zarray_fini(&core.sharedInst);
zarray_fini(&core.sharedInUse);
destroy(core.resmgr);
}
#define TEST(x) \
do { \
if (check_test(&arguments, #x)) { \
@ -759,6 +850,7 @@ int main(int argc, char* argv[])
TEST(test_sched_update_existing_queue_null_payload);
TEST(test_dead_pushmsg_symbols_absent);
TEST(test_core_safe_allocation_chain_driver_parity);
TEST(test_unsatisfy_external_skips_inuse_shared_leg);
return UNITY_END();
}

225
src/scheduler/testsuite/test_core_events.c
View file

@ -900,6 +900,99 @@ static void test_job_update_run_adopts_controller_allocation(void)
"ownership of the allocation moves to the scheduler's job");
}
/* bug-1976 companion (external-license-resources Phase 3): the controller's
* RUN adoption is the scheduler's ONLY view of a dispatch (the controller is
* ground truth), so the adopted wire alloc must carry the shared-pool
* records too -- the limit vres charge among them (allocs_shared; the
* controller-side drop was bug-1976 itself, fixed in dispatch_a_job_()).
* Without them make_job_account() builds accounts_shared == NULL and
* sched_account_jobs() never charges the vres: sharedInUse stays 0.0 and
* unsatisfy_'s non-EXTERNAL branch (total >= inuse + uncomit + wanted)
* admits past every global limit forever. This pins the scheduler side of
* the round trip: an adoption whose wire alloc HAS shared records must
* produce accounts whose accounts_shared mirror them. */
static void test_job_update_run_adoption_carries_shared_allocs(void)
{
xce_sched_core_t core;
xce_core_event_t event;
xce_job_t* job1;
xce_job_t updated;
xce_sched_job_t* job;
xce_alloc_t* a_slots;
xce_host_alloc_t* h0;
xce_host_alloc_t* hallocs[1];
xce_job_alloc_t* jalloc;
xce_alloc_t** shared;
int rc;
fixture_init(&core, &g_full_plugin);
job1 = make_job(71, JOB_STAT_PEND, "normal", "alice", "projA");
memset(&event, 0, sizeof(event));
event.type = XCE_CORE_EVENT_JOBS_PUSH;
event.u.jobs_push.njobs = 1;
event.u.jobs_push.jobs = &job1;
rc = xce_core_push_events(&core, &event, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
job = sched_get_job(&core, 71);
TEST_ASSERT_NOT_NULL(job);
/* the exact shape the controller re-sends post-fix: per-host slots +
* two shared records -- the limit vres instance charge and the
* underlying EXTERNAL pool draw. */
a_slots = make_alloc((char*)"slots", NULL, 0, 1.0f, 0);
h0 = make_host_alloc((char*)"hostC", 1, make_allocs(1, a_slots));
hallocs[0] = h0;
jalloc = make_job_alloc(71, 1, make_host_allocs(1, hallocs[0]));
shared = make_allocs(2,
make_alloc((char*)"limit=extlic_share/p=projA", NULL, 0, 1.0f, 0),
make_alloc((char*)"extlic", NULL, 0, 1.0f, 0));
TEST_ASSERT_NOT_NULL(shared);
jalloc->allocs_shared = shared;
jalloc->nres_shared = 2;
memset(&updated, 0, sizeof(updated));
updated.jobid = 71;
updated.state = JOB_STAT_RUN;
updated.allocs = jalloc;
memset(&event, 0, sizeof(event));
event.type = XCE_CORE_EVENT_JOB_UPDATE;
event.u.job_update.jobid = 71;
event.u.job_update.job = &updated;
rc = xce_core_push_events(&core, &event, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, job->state);
TEST_ASSERT_NOT_NULL(job->allocs);
/* the accounts the charge loop walks must mirror the shared records --
* this is the exact property bug-1976's controller drop destroyed */
TEST_ASSERT_NOT_NULL(job->accounts);
TEST_ASSERT_EQUAL_INT_MESSAGE(2, job->accounts->nres_shared,
"adoption must rebuild accounts WITH the shared records -- without "
"them sched_account_jobs() never charges the vres and every global "
"limit admits forever (bug-1976)");
TEST_ASSERT_NOT_NULL(job->accounts->accounts_shared);
TEST_ASSERT_EQUAL_STRING("limit=extlic_share/p=projA",
job->accounts->accounts_shared[0]->resname);
TEST_ASSERT_EQUAL_FLOAT(1.0f,
job->accounts->accounts_shared[0]->reserved);
TEST_ASSERT_EQUAL_STRING("extlic",
job->accounts->accounts_shared[1]->resname);
TEST_ASSERT_EQUAL_FLOAT(1.0f,
job->accounts->accounts_shared[1]->reserved);
/* allocation ownership moves; the job's own records survive intact */
TEST_ASSERT_NULL(updated.allocs);
TEST_ASSERT_EQUAL_INT(2, job->allocs->nres_shared);
TEST_ASSERT_EQUAL_STRING("limit=extlic_share/p=projA",
job->allocs->allocs_shared[0]->resname);
}
/* Companion: a resync claiming a genuinely TERMINAL state (DONE/EXIT) for a
* RUN-with-real-allocation job must still apply -- this is the legitimate
* "job finished while this scheduler was disconnected" case bug-1734's own
@ -2005,9 +2098,140 @@ test_jobs_push_picker_rejection_still_tracks_job_and_succeeds(void)
g_counts.job_push_rc = 0;
}
/* external-license-resources Phase 1 (E2), landing-shape pin: a
* RESVALS_DELTA entry whose host is NULL or "" is a SHARED (cluster-level)
* value pushed by the extres daemon, not a per-worker value -- it must land
* in core->sharedInst at the resource's idx, mirroring snapshot_load()'s
* host==NULL/"" precedent (bug-1923) and scheduler.c's legacy DELTA loop
* needs the same rule. Controller's HOOK_DETAIL_RESOURCE_UPDATE notify is
* delivered to the core as RESVALS_DELTA (handler.c
* sched_handle_sched_update_resvals -> XCE_SCHED_OP_UPDATE_RESVALS), and
* worker pushes carry a NAMED host while the daemon's shared values carry
* host=="" -- so the empty host is the only marker the DELTA path has to
* route by. */
static void test_resvals_delta_empty_host_lands_in_sharedinst(void)
{
xce_sched_core_t core;
xce_core_event_t event;
xce_resource_t wire_resource;
xce_resource_t* resource_ptrs[1];
xce_resval_t empty_host_resval;
xce_resval_t* delta_vals[1];
xce_resval_list_t deltas;
xce_val_t val_empty;
xce_val_t* stored;
xce_resource_t* found = NULL;
int rc;
fixture_init(&core, &g_full_plugin);
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = "lic";
wire_resource.idx = 3;
resource_ptrs[0] = &wire_resource;
memset(&event, 0, sizeof(event));
event.type = XCE_CORE_EVENT_RESOURCES_PUSH;
event.u.resources_push.n = 1;
event.u.resources_push.resources = resource_ptrs;
rc = xce_core_push_events(&core, &event, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&val_empty, 0, sizeof(val_empty));
val_empty.which = Value_u_fvalue;
val_empty.u.fvalue = 5.0;
memset(&empty_host_resval, 0, sizeof(empty_host_resval));
empty_host_resval.name = "lic";
empty_host_resval.host = "";
empty_host_resval.val = &val_empty;
delta_vals[0] = &empty_host_resval;
deltas.count = 1;
deltas.vals = delta_vals;
memset(&event, 0, sizeof(event));
event.type = XCE_CORE_EVENT_RESVALS_DELTA;
event.u.resvals_delta.deltas = &deltas;
rc = xce_core_push_events(&core, &event, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_TRUE(key_find(core.resmgr, (void*)"lic",
(void**)&found));
TEST_ASSERT_NOT_NULL(found);
stored = (xce_val_t*)zarray_get(&(core.sharedInst), found->idx);
TEST_ASSERT_NOT_NULL(stored);
TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which);
TEST_ASSERT_EQUAL_FLOAT(5.0, stored->u.fvalue);
}
/* Same landing rule for the NULL spelling: snapshot_load (bug-1923) treats
* host==NULL as shared; the DELTA path must survive a NULL host (skip the
* per-worker lookup instead of strlen(NULL)) and land the value in
* core->sharedInst identically. */
static void test_resvals_delta_null_host_lands_in_sharedinst(void)
{
xce_sched_core_t core;
xce_core_event_t event;
xce_resource_t wire_resource;
xce_resource_t* resource_ptrs[1];
xce_resval_t null_host_resval;
xce_resval_t* delta_vals[1];
xce_resval_list_t deltas;
xce_val_t val_null;
xce_val_t* stored;
xce_resource_t* found = NULL;
int rc;
fixture_init(&core, &g_full_plugin);
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = "lic";
wire_resource.idx = 3;
resource_ptrs[0] = &wire_resource;
memset(&event, 0, sizeof(event));
event.type = XCE_CORE_EVENT_RESOURCES_PUSH;
event.u.resources_push.n = 1;
event.u.resources_push.resources = resource_ptrs;
rc = xce_core_push_events(&core, &event, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&val_null, 0, sizeof(val_null));
val_null.which = Value_u_fvalue;
val_null.u.fvalue = 4.0;
memset(&null_host_resval, 0, sizeof(null_host_resval));
null_host_resval.name = "lic";
null_host_resval.host = NULL;
null_host_resval.val = &val_null;
delta_vals[0] = &null_host_resval;
deltas.count = 1;
deltas.vals = delta_vals;
memset(&event, 0, sizeof(event));
event.type = XCE_CORE_EVENT_RESVALS_DELTA;
event.u.resvals_delta.deltas = &deltas;
rc = xce_core_push_events(&core, &event, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_TRUE(key_find(core.resmgr, (void*)"lic",
(void**)&found));
TEST_ASSERT_NOT_NULL(found);
stored = (xce_val_t*)zarray_get(&(core.sharedInst), found->idx);
TEST_ASSERT_NOT_NULL(stored);
TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which);
TEST_ASSERT_EQUAL_FLOAT(4.0, stored->u.fvalue);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_resvals_delta_empty_host_lands_in_sharedinst);
RUN_TEST(test_resvals_delta_null_host_lands_in_sharedinst);
RUN_TEST(test_push_events_fires_each_hook_once_per_event_with_picker_idle);
RUN_TEST(test_jobs_push_fires_on_job_push_once_per_job);
RUN_TEST(test_queue_update_replaces_existing_interned_resreq);
@ -2016,6 +2240,7 @@ int main(void)
RUN_TEST(test_jobs_push_duplicate_jobid_run_state_transfers_allocation);
RUN_TEST(test_jobs_push_duplicate_jobid_controller_pend_wins_over_local_run);
RUN_TEST(test_job_update_run_adopts_controller_allocation);
RUN_TEST(test_job_update_run_adoption_carries_shared_allocs);
RUN_TEST(test_jobs_push_duplicate_jobid_terminal_resync_releases_real_allocation);
RUN_TEST(test_jobs_push_duplicate_jobid_run_state_without_allocation_is_refused);
RUN_TEST(test_jobs_push_duplicate_jobid_does_not_regress_terminal_state);

611
src/scheduler/testsuite/test_limit_project_scope.c Normal file
View file

@ -0,0 +1,611 @@
/*
* test_limit_project_scope.c - external-license-resources Phase 3 (LSF
* 策略层, design.md 3.5): per-project/per-queue/per-user token distribution
* maps onto the EXISTING limit vres mechanism. A limit on a GLOBAL
* underlying resource (e.g. the extlic license pool, SHARED|GLOBAL|
* EXTERNAL per mapping.py's location: [external] branch) with a projects:
* scope must:
*
* 1. match jobs by their obj->project string (literal project members --
* make_g_member()'s literal fallback; named project groups are
* project-config's territory, deliberately not exercised here);
* 2. intern a usage instance named "limit=<name>" (no /p= -- per_project
* false means one shared pool for all members of the project scope),
* or "limit=<name>/p=<proj>" when per_project is set (independent
* usage per project under one configured cap);
* 3. write its cap into core->sharedInst at the vres's own idx
* (update_limit_res_value());
* 4. take the matched vres into obj->glimits (match_job_global_limit_
* -> xlist_store), so sched_fill_limit_requests() (resource.c) can
* stage the job's want into alloc_req.shared and the generic satisfy
* check can compare want+uncomit against the cap like any resource.
*
* The mechanical shape match_job_global_limit_ needs from the core:
* - core->resmgr: must already contain the UNDERLYING resource named by
* limit->val.resname (instantiate_limit() returns -ENOENT otherwise)
* AND grows the limit's vres via add_vres() -- exactly what
* sched_init_resources() does live;
* - core->sharedInst: the cap's zarray (update_limit_res_value()'s
* put_resvals target);
* - core->sharedInUse: where the generic check reads RUN jobs' usage
* (init here for parity; not touched by the plugin).
*
* White-box: #include's the real ../src/plugins/limit.c +
* limit_config.c (same convention as test_limit_plugin.c, whose fixture
* this file mirrors -- real group managers, real plugin private state).
* match_job_global_limit_() is `static`, otherwise unreachable outside
* its TU. get_resvals()/put_resvals() are NOT no-op stubs here (unlike
* test_limit_plugin.c, whose tests never exercise the value path): the
* value path IS this file's subject, so both are faithful copies of
* resource.c's (that file is not part of this TU; the copies preserve the
* production zarray protocol -- xce_val_t entries in Value_u_fvalue form,
* created zero-filled on first put, read as 0.0 when absent) so the
* fixture's sharedInst ends in the same state production would leave,
* and the cap assertions below prove the plugin put the RIGHT value at
* the RIGHT idx, not merely that it called something.
*/
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "group.h"
#include "kvmgr.h"
#include "scheduler.h"
#include "xce/types.h"
#include "xcelimit.h"
#include "xstring.h"
/* Faithful copies of resource.c's get_resvals()/put_resvals() (see file
* header). The plugin always calls them with nres == 1, so the copies
* need no dedup-by-duplicate-idx handling (resource.c's bug-1569 block is
* dead weight at nres 1). */
int get_resvals(zarray_t* array, int nres, int* idxs, float* vals)
{
xce_val_t* xvals[1];
int rc;
if ((array == NULL) || (nres < 0) || (idxs == NULL) || (vals == NULL)) {
return -EINVAL;
}
if (nres == 0) {
return 0;
}
memset(vals, 0x0, sizeof(*vals) * nres);
rc = zarray_get_ex(array, nres, idxs, (void**)xvals);
if (rc < 0) {
return rc;
}
if ((xvals[0] != NULL) && (xvals[0]->which == Value_u_fvalue)) {
vals[0] = xvals[0]->u.fvalue;
}
return nres;
}
int put_resvals(zarray_t* array, int nres, int* idxs, float* vals)
{
xce_val_t* xvals[1];
int rc;
if ((array == NULL) || (nres < 0) || (idxs == NULL) || (vals == NULL)) {
return -EINVAL;
}
if (nres == 0) {
return 0;
}
rc = zarray_get_ex(array, nres, idxs, (void**)xvals);
if (rc < 0) {
return rc;
}
if (idxs[0] < 0) {
return nres;
}
if (xvals[0] == NULL) {
xce_val_t* val = (xce_val_t*)calloc(1, sizeof(*val));
if (val == NULL) {
return -ENOMEM;
}
val->which = Value_u_fvalue;
val->u.fvalue = 0.0;
zarray_put(array, idxs[0], val);
xvals[0] = val;
}
xvals[0]->u.fvalue = vals[0];
return nres;
}
/* limit_job_election() calls reset_sched_obj() on the REAL schedobj;
* this TU drives match_job_global_limit_() directly, so a stub purely
* for the linker -- same as test_limit_plugin.c. */
void reset_sched_obj(xce_sched_obj_t* obj)
{
(void)obj;
}
int plugin_register(void* sched, void* plugin)
{
(void)sched;
(void)plugin;
return 0;
}
#include "../src/plugins/limit.c"
#include "../src/plugins/limit_config.c"
void setUp(void) { return; }
void tearDown(void) { return; }
/* Same fixture as test_limit_plugin.c's fixture_init(): real group
* managers for all four consumer kinds (build_limit()'s build_consumer()
* resolves each consumer against its own manager -- the "all" group for
* any scope the config leaves NULL), plus a real limit plugin private
* state. plugin->core is the bug-1964 honest reference. Extended here
* with the two pieces production's xce_sched_core_init() builds and the
* plugin's value path touches: core->resmgr (a kvmgr with no datakiller
* -- suite convention, detect_leaks=0, no teardown) and the
* sharedInst/sharedInUse zarays. */
static void fixture_init(xce_scheduler_t* sched, xce_sched_plugin_t* plugin)
{
kvmgr_params_t resmgr_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 8,
.delta = 4,
.datakiller = NULL };
int rc;
memset(sched, 0, sizeof(*sched));
sched->core.resmgr = makeInstance(&xkvmgr_type, &resmgr_params);
TEST_ASSERT_NOT_NULL(sched->core.resmgr);
groupmgr_init(&(sched->core.ugmgr), 4, 4);
groupmgr_init(&(sched->core.qgmgr), 4, 4);
groupmgr_init(&(sched->core.pgmgr), 4, 4);
groupmgr_init(&(sched->core.hgmgr), 4, 4);
TEST_ASSERT_NOT_NULL(sched->core.ugmgr);
TEST_ASSERT_NOT_NULL(sched->core.qgmgr);
TEST_ASSERT_NOT_NULL(sched->core.pgmgr);
TEST_ASSERT_NOT_NULL(sched->core.hgmgr);
zarray_init(&(sched->core.sharedInst), 8, NULL);
zarray_init(&(sched->core.sharedInUse), 8, NULL);
memset(plugin, 0, sizeof(*plugin));
plugin->sched = sched;
plugin->core = &(sched->core);
rc = limit_init(plugin);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(plugin->data);
}
/* sched_init_resources()'s role, reduced to what the limit plugin needs:
* the UNDERLYING resource must exist in core->resmgr before the first
* match instantiates the vres (instantiate_limit()'s key_find of
* limit->val.resname returns -ENOENT otherwise -- the "unknown resource"
* failure mode test 3 pins). `flags` picks the underlying resource's own
* nature: SHARED|GLOBAL|EXTERNAL is the exact shape mapping.py's
* location: [external] branch produces for extlic. NOTE: makeResource_
* assigns the resource's idx itself (the kvmgr slot), so the caller's
* wire.idx is ignored -- tests must read idx back, never assume it. */
static xce_resource_t* register_resource(xce_sched_core_t* core,
const char* name, int flags)
{
xce_resource_t wire;
xce_resource_t* made = NULL;
memset(&wire, 0, sizeof(wire));
wire.name = (char*)name;
wire.flags = flags;
made = (xce_resource_t*)makeResource(core->resmgr, &wire);
TEST_ASSERT_NOT_NULL(made);
return made;
}
/* push limit configs through the plugin's config hook (both drivers do
* this -- plugin_config_update() live, plugin_config_update_core() sim) */
static int push_limits(xce_sched_plugin_t* plugin, xce_limit_config_t** limits,
int n)
{
xce_conf4sched_t config;
memset(&config, 0, sizeof(config));
config.nlimits = n;
config.limits = limits;
return limit_update_config(plugin, &config);
}
/* a heap limit config: scope = literal project members (heap group config
* -- the plugin's built limit references the group config's members for
* the consumer group's lifetime), absolute value cap */
static xce_limit_config_t* make_project_limit(const char* name,
const char* resname, float cap,
const char* const* projects,
int nprojects)
{
xce_limit_config_t* cfg = (xce_limit_config_t*)calloc(1, sizeof(*cfg));
xce_limit_val_t* val = (xce_limit_val_t*)calloc(1, sizeof(*val));
xce_group_config_t* g = (xce_group_config_t*)calloc(1, sizeof(*g));
char** members = (char**)calloc((size_t)nprojects,
sizeof(char*));
int i;
TEST_ASSERT_NOT_NULL(cfg);
TEST_ASSERT_NOT_NULL(val);
TEST_ASSERT_NOT_NULL(g);
TEST_ASSERT_NOT_NULL(members);
val->resname = (char*)resname;
val->percent = 0;
val->val = cap;
for (i = 0; i < nprojects; i++) {
members[i] = (char*)projects[i];
}
g->members.count = nprojects;
g->members.data = members;
cfg->name = (char*)name;
cfg->val = val;
cfg->projects = g;
return cfg;
}
static void free_limit_config(xce_limit_config_t* cfg)
{
FREEUP(cfg->val);
/* per_queue configs carry the scope in queues and NULL projects (the
* per_project shape is the mirror image) -- either group is optional */
if (cfg->projects != NULL) {
FREEUP(cfg->projects->members.data);
FREEUP(cfg->projects);
}
if (cfg->queues != NULL) {
FREEUP(cfg->queues->members.data);
FREEUP(cfg->queues);
}
FREEUP(cfg);
}
static xce_limit_t* find_limit(xce_sched_plugin_t* plugin, const char* name)
{
limit_plugin_data_t* privdata = (limit_plugin_data_t*)plugin->data;
xce_limit_t* limit = NULL;
key_find(privdata->limits, (void*)name, (void**)&limit);
return limit;
}
static xce_vres_t* find_instance(xce_sched_plugin_t* plugin, const char* name)
{
limit_plugin_data_t* privdata = (limit_plugin_data_t*)plugin->data;
xce_vres_t* vres = NULL;
key_find(privdata->instances, (void*)name, (void**)&vres);
return vres;
}
/* a minimal xce_sched_obj_t: project identity + the glimits xlist the
* plugin stores matched vres into. make_sched_obj() (schedobj.c) builds
* the real one; its project normalization ("default" for empty) is
* core_events.c's job, and the plugin consumes the result as-is -- so the
* fixture hands obj->project the string the scheduler would have. */
static void make_obj(xce_sched_obj_t* obj, const char* project)
{
memset(obj, 0, sizeof(*obj));
obj->user = makeString("user1");
obj->queue = makeString("normal");
obj->project = makeString((char*)project);
xlist_init(&(obj->glimits), 8);
}
/* drive match_job_global_limit_() the way limit_job_election() does:
* throwaway ctx, only ctx.schedobj is read. */
static int match_global(xce_sched_plugin_t* plugin, xce_sched_obj_t* obj,
xce_limit_t* limit)
{
xce_sched_ctx_t ctx = { 0 };
ctx.schedobj = obj;
return match_job_global_limit_(plugin, &ctx, limit);
}
/* ---------------------------------------------------------------------- */
/* 1. A project-scoped limit on a GLOBAL|SHARED|EXTERNAL resource matches
* by obj->project (literal member), is global (lands in glimits --
* is_host_based_limit() keys on the HOST consumer, never the underlying
* resource's own GLOBAL flag), and instantiates the usage instance named
* "limit=<name>" (per_project false -> one shared pool for the project
* scope; no /p= segment). Members outside the scope match nothing and
* store nothing. */
static void test_project_scope_match_and_instantiate(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t plugin;
xce_sched_obj_t obj;
xce_limit_config_t* limits[1];
xce_limit_t* limit;
xce_vres_t* vres;
const char* const scope_a[] = { "projA" };
char vresname[64];
int rc;
fixture_init(&sched, &plugin);
/* the extlic shape: SHARED (cluster pool) + GLOBAL (admitted in the
* shared leg) + EXTERNAL (free-value reporting; Phase 1's unsatisfy_
* rule -- asserted independently in testsuite/test.c). */
register_resource(&(sched.core), "extlic",
XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL);
limits[0] = make_project_limit("extlic_projA", "extlic", 2.0f, scope_a, 1);
TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 1));
limit = find_limit(&plugin, "extlic_projA");
TEST_ASSERT_NOT_NULL(limit);
/* in-scope project matches. match_job_global_limit_()'s own rc
* contract is "0 on success" (its tail rc = xlist_store() returns the
* stored idx -- 0 for the first entry; only < 0 is failure, the same
* rc<0 break limit_job_election() applies), so the success signal is
* rc >= 0 AND the vres landing in obj.glimits -- asserted below. */
make_obj(&obj, "projA");
rc = match_global(&plugin, &obj, limit);
TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc);
/* the instance is keyed by the bare vres name (per_project false) */
snprintf(vresname, sizeof(vresname), "limit=extlic_projA");
vres = find_instance(&plugin, vresname);
TEST_ASSERT_NOT_NULL(vres);
TEST_ASSERT_EQUAL_PTR(limit, vres->limit);
TEST_ASSERT_NOT_NULL(vres->vres);
TEST_ASSERT_EQUAL_STRING(vresname, vres->vres->name);
/* the vres is a REAL resource now: SHARED|GLOBAL (its own nature, from
* add_vres()'s `shared` flag derived from is_host_based_limit()) +
* LIMIT (the satisfy check's PREASON_LIMIT_GLOBAL discriminator). */
TEST_ASSERT_TRUE((vres->vres->flags & XCE_RESF_LIMIT) != 0);
TEST_ASSERT_TRUE((vres->vres->flags & XCE_RESF_SHARED) != 0);
TEST_ASSERT_TRUE((vres->vres->flags & XCE_RESF_GLOBAL) != 0);
/* the underlying resource is linked */
TEST_ASSERT_NOT_NULL(vres->res);
TEST_ASSERT_EQUAL_STRING("extlic", vres->res->name);
/* the matched vres was stored into obj->glimits for
* sched_fill_limit_requests() to stage */
TEST_ASSERT_EQUAL_INT(1, obj.glimits.count);
TEST_ASSERT_EQUAL_PTR(vres, xlist_get(&(obj.glimits), 0));
/* out-of-scope project: no match, nothing stored */
make_obj(&obj, "projB");
rc = match_global(&plugin, &obj, limit);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, obj.glimits.count);
free_limit_config(limits[0]);
}
/* ---------------------------------------------------------------------- */
/* 2. The cap lands in core->sharedInst at the vres's own idx, and the
* per_project variant slices by project ("limit=<name>/p=<proj>"): two
* projects get independent usage entries under one configured cap. */
static void test_cap_value_and_per_project_instance_names(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t plugin;
xce_sched_obj_t obj;
xce_limit_config_t* limits[2];
xce_limit_t* shared_pool;
xce_limit_t* per_project;
xce_vres_t* vres;
xce_val_t* v;
const char* const scope_a[] = { "projA" };
const char* const scope_ab[] = { "projA", "projB" };
char name[64];
int rc;
fixture_init(&sched, &plugin);
register_resource(&(sched.core), "extlic",
XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL);
limits[0] = make_project_limit("extlic_pool", "extlic", 2.0f, scope_a, 1);
/* per-project slicing needs BOTH projects in the limit's scope: the
* scope decides who matches at all; per_project then splits each
* match into its own "limit=<name>/p=<proj>" usage instance. */
limits[1] = make_project_limit("extlic_share", "extlic", 3.0f, scope_ab,
2);
limits[1]->per_project = 1;
TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 2));
/* shared-pool limit: cap written at the instance's own vres idx */
shared_pool = find_limit(&plugin, "extlic_pool");
TEST_ASSERT_NOT_NULL(shared_pool);
make_obj(&obj, "projA");
TEST_ASSERT_GREATER_OR_EQUAL_INT(0, match_global(&plugin, &obj,
shared_pool));
vres = (xce_vres_t*)xlist_get(&(obj.glimits), 0);
TEST_ASSERT_NOT_NULL(vres);
v = (xce_val_t*)zarray_get(&(sched.core.sharedInst), vres->vres->idx);
TEST_ASSERT_NOT_NULL(v);
TEST_ASSERT_EQUAL_FLOAT(2.0f, v->u.fvalue);
/* per-project limit: /p= slices the name, so projA and projB get
* independent instances (independent usage) under one cap */
per_project = find_limit(&plugin, "extlic_share");
TEST_ASSERT_NOT_NULL(per_project);
make_obj(&obj, "projA");
rc = match_global(&plugin, &obj, per_project);
TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc);
make_obj(&obj, "projB");
rc = match_global(&plugin, &obj, per_project);
TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc);
snprintf(name, sizeof(name), "limit=extlic_share/p=projA");
TEST_ASSERT_NOT_NULL(find_instance(&plugin, name));
snprintf(name, sizeof(name), "limit=extlic_share/p=projB");
TEST_ASSERT_NOT_NULL(find_instance(&plugin, name));
free_limit_config(limits[0]);
free_limit_config(limits[1]);
}
/* ---------------------------------------------------------------------- */
/* 2b. The per-queue distribution shape (limits.yaml's extlic_qdist): a
* queues:-scoped limit with per_queue slices its usage instance by queue
* ("limit=<name>/q=<queue>") exactly as the projects shape slices by
* project ("/p=") -- how a site partitions one license pool across
* queues. Also pins that the queue scope does NOT match a project-only
* identity: the consumer matched is the obj's queue, so a different
* queue outside the members list matches nothing. */
static void test_queue_scope_per_queue_instance_names(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t plugin;
xce_sched_obj_t obj;
xce_limit_config_t* limits[1];
xce_limit_config_t* cfg;
xce_group_config_t* qg;
xce_vres_t* vres;
const char* const scope_a[] = { "projA" };
const char* const queues[] = { "normal", "night" };
char** qmembers;
char name[64];
int rc;
int i;
fixture_init(&sched, &plugin);
register_resource(&(sched.core), "extlic",
XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL);
limits[0] = make_project_limit("extlic_qdist", "extlic", 4.0f, scope_a,
1);
cfg = limits[0];
FREEUP(cfg->projects->members.data);
FREEUP(cfg->projects);
cfg->projects = NULL;
cfg->per_queue = 1;
/* queue scope: heap group config, same lifetime contract as the
* project one make_project_limit() builds */
qg = (xce_group_config_t*)calloc(1, sizeof(*qg));
qmembers = (char**)calloc(2, sizeof(char*));
TEST_ASSERT_NOT_NULL(qg);
TEST_ASSERT_NOT_NULL(qmembers);
for (i = 0; i < 2; i++) {
qmembers[i] = (char*)queues[i];
}
qg->members.count = 2;
qg->members.data = qmembers;
cfg->queues = qg;
TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 1));
/* each in-scope queue gets its OWN /q= instance */
make_obj(&obj, "projA");
obj.queue = makeString("normal");
rc = match_global(&plugin, &obj, find_limit(&plugin, "extlic_qdist"));
TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc);
snprintf(name, sizeof(name), "limit=extlic_qdist/q=normal");
vres = find_instance(&plugin, name);
TEST_ASSERT_NOT_NULL_MESSAGE(vres,
"per_queue must slice the usage instance by queue (/q= segment), "
"the queue analogue of the /p= project shape");
TEST_ASSERT_EQUAL_INT(1, obj.glimits.count);
TEST_ASSERT_EQUAL_PTR(vres, xlist_get(&(obj.glimits), 0));
/* cap 4.0 written at this instance's own vres idx */
{
xce_val_t* v = (xce_val_t*)zarray_get(&(sched.core.sharedInst),
vres->vres->idx);
TEST_ASSERT_NOT_NULL(v);
TEST_ASSERT_EQUAL_FLOAT(4.0f, v->u.fvalue);
}
make_obj(&obj, "projA");
obj.queue = makeString("night");
rc = match_global(&plugin, &obj, find_limit(&plugin, "extlic_qdist"));
TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc);
snprintf(name, sizeof(name), "limit=extlic_qdist/q=night");
TEST_ASSERT_NOT_NULL(find_instance(&plugin, name));
/* out-of-scope queue matches nothing */
make_obj(&obj, "projA");
obj.queue = makeString("short");
rc = match_global(&plugin, &obj, find_limit(&plugin, "extlic_qdist"));
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, obj.glimits.count);
free_limit_config(limits[0]);
}
/* ---------------------------------------------------------------------- */
/* 3. The unknown-underlying-resource failure is a clean -ENOENT: a limit
* naming a resource the core never registered must not crash and must
* leave nothing in obj->glimits (the failure mode instantiate_limit()'s
* own comment warns of -- the resname must exist before the first match). */
static void test_unknown_underlying_resource_fails_clean(void)
{
xce_scheduler_t sched;
xce_sched_plugin_t plugin;
xce_sched_obj_t obj;
xce_limit_config_t* limits[1];
const char* const scope_a[] = { "projA" };
int rc;
fixture_init(&sched, &plugin);
/* deliberately NO resource registered */
limits[0] = make_project_limit("ghost_limit", "ghost_lic", 2.0f, scope_a,
1);
TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 1));
TEST_ASSERT_NOT_NULL(find_limit(&plugin, "ghost_limit"));
make_obj(&obj, "projA");
rc = match_global(&plugin, &obj, find_limit(&plugin, "ghost_limit"));
TEST_ASSERT_EQUAL_INT(-ENOENT, rc);
TEST_ASSERT_EQUAL_INT(0, obj.glimits.count);
free_limit_config(limits[0]);
}
int main(void)
{
int rc;
/* group names are interned strings (makeString(), group.c) */
xstring_table_init(139);
UNITY_BEGIN();
RUN_TEST(test_project_scope_match_and_instantiate);
RUN_TEST(test_cap_value_and_per_project_instance_names);
RUN_TEST(test_queue_scope_per_queue_instance_names);
RUN_TEST(test_unknown_underlying_resource_fails_clean);
rc = UNITY_END();
xstring_table_fini();
return rc;
}

5
src/schema/controller.capnp
View file

@ -153,6 +153,11 @@ enum HostRole {
scheduler @2;
controller @3;
apiserver @4;
# external-license-resources Phase 1: the external-resources daemon
# (xceesd) joins with this role to push shared-resource free counts
# (license totals etc.) through the resval update pipeline. Appended
# last so existing enum values keep their wire numbers.
extres @5;
}
struct JoinData $C.mapname("xce_joindata_t") {

83
src/worker/include/elim.h Normal file
View file

@ -0,0 +1,83 @@
#ifndef ELIM_H
#define ELIM_H
#include <stdbool.h>
#include <stddef.h>
/*
* elim -- the LSF-elim-compatible external resource collector for xcecd
* (external-license-resources design §3.3b). Each configured source names a
* command run on this host (posix_spawn, watchdog kill at
* elim_script_timeout_ms); the command prints "<name> <value>" (elim
* format) or a bare number on stdout, and the value registers as a custom
* load index on resource_sync's registry -- flowing the live per-host
* report chain (collect_load_indices -> xcecd_send_load_indices ->
* controller update_val_res -> scheduler RESVALS_DELTA) with zero
* downstream changes.
*
* Per-host semantics by design: an elim script answers "how much does THIS
* host have"; cluster-global pools (license servers) belong to xceesd.
*/
typedef struct elim_source_s {
char* name; /* resource name the script reports */
char* command; /* shell command line (run via /bin/sh -c) */
int interval; /* seconds between re-runs; 0 = every collect */
} elim_source_t;
typedef struct elim_config_s {
int nsrc;
elim_source_t* src; /* array of nsrc sources */
} elim_config_t;
/*
* Parse the elim section of the worker config file:
*
* external_resources:
* - name: dynres1
* command: /opt/xce/scripts/report-dynres1.sh
* interval: 30
*
* Returns 0, -ENOENT (file missing), -EINVAL (bad shape: a source without
* name or command can never report -- fail loudly at load, not silently at
* runtime), -ENOMEM.
*/
extern int elim_config_load(const char* path, elim_config_t* cfg_out);
extern int elim_config_load_path(const char* dir, const char* name,
elim_config_t* cfg_out);
extern void elim_config_fini(elim_config_t* cfg);
/*
* Register every configured source as a custom (non-builtin) load index on
* resource_sync's registry. Returns 0, or -EEXIST if a source's name is
* already taken (another source, or one of the 11 builtins -- a user script
* must not shadow "mem"/"r15s"/...), or -EINVAL/-ENOMEM. Registration
* happens at startup, before the sync thread's first collect -- the
* registry self-initializes for that (the pre-phase "capacity reached (0)"
* trap).
*/
extern int elim_register_sources(const elim_config_t* cfg);
extern void elim_unregister_sources(const elim_config_t* cfg);
/*
* Run one collection command and read its stdout. Returns 0 with `out`
* NUL-terminated, -ETIMEDOUT when the watchdog killed the child,
* -ENODATA on empty output, -EINVAL/-errno on spawn/read failure.
* The child gets an empty signal mask and default dispositions; stderr is
* /dev/null. Never blocks past elim_script_timeout_ms.
*/
extern int elim_script_collect(const char* command, char* out, size_t outsz);
/*
* Parse an elim-format collection output: the first non-empty line carries
* "<name> <value>" or a bare number; leading/trailing whitespace ignored.
* Returns the value, or NAN on garbage/empty (caller owns the fail-open
* policy).
*/
extern double elim_parse_script_output(const char* out);
/* Watchdog, milliseconds a collection command may run before it is killed
* (overridable from tests / future CLI). */
extern int elim_script_timeout_ms;
#endif // ELIM_H

6
src/worker/include/resource_sync.h
View file

@ -19,4 +19,10 @@ bool resource_sync_register_load_index(const char* name, int valtype,
bool is_builtin);
bool resource_sync_unregister_load_index(const char* name);
/* Ensure the registry array exists and the 11 builtin load indices are in
* it -- for external registrars (elim) that run before the sync thread's
* first collect would lazily populate them, so their name-collision checks
* see the builtins. */
void resource_sync_ensure_builtin_load_indices(void);
#endif // RESOURCE_SYNC_H

9
src/worker/include/xcecd.h
View file

@ -2,6 +2,7 @@
#define XCECD_H
#include "controller_connector.h"
#include "elim.h"
#include "hardware_monitor.h"
#include "job_information.h"
#include "node_information.h"
@ -63,6 +64,11 @@ typedef struct {
* xcecd_start() already ran. candidates_count == 0 means no --peers was
* given (single-host default). */
xce_cluster_config_t cluster_cfg;
/* elim sources from the -c config file's external_resources section
* (design §3.3b). Loaded in main.c (elim_config_load); ownership moves
* to xcecd_ctx by xcecd_init(), which registers the sources with
* resource_sync at startup. */
elim_config_t elim;
} xcecd_config_t;
typedef struct {
@ -88,6 +94,9 @@ typedef struct {
char* log_dir;
char* db_path;
char* virtual_name;
/* elim collectors (design §3.3b): sources loaded from the -c config
* file, registered with resource_sync by xcecd_init(). */
elim_config_t elim;
/* Copied from xcecd_config_t by xcecd_init(); consumed by
* controller_connector_init() (xcecd_start()) to populate
* cc_ctx->cluster_cfg BEFORE controller_connector_start() fires the

2
src/worker/meson.build
View file

@ -16,6 +16,7 @@ xcecd_worker_srcs = [
'src/xcecd.c',
'src/connection_handler.c',
'src/resource_sync.c',
'src/elim.c',
'src/node_information.c',
'src/thread_pool.c',
'src/hardware_monitor.c',
@ -95,6 +96,7 @@ xcecd_srcs = [
'src/connection_handler.c',
'src/controller_connector.c',
'src/resource_sync.c',
'src/elim.c',
'src/xcecd_worker_utils.c',
'src/xcecd_resync.c',
'src/hardware_monitor.c',

656
src/worker/src/elim.c Normal file
View file

@ -0,0 +1,656 @@
/* elim.c -- the LSF-elim-compatible external resource collector for xcecd
* (external-license-resources design §3.3b).
*
* The worker already owns half the elim model as live code: resource_sync.c
* keeps a load-index registry (11 builtins shaped after LSF's load index
* name set) whose values flow the per-host report chain every 15 ticks --
* collect_load_indices() -> xcecd_send_load_indices() -> controller
* update_val_res (non-SHARED lands in w->host.resvals) -> scheduler
* RESVALS_DELTA -> sched_check_host_satisfy_. What was missing is the
* ENTRY: a mechanism for a user to register their own index. This module is
* that entry -- config-driven script collectors registered as custom
* (non-builtin) load indices at startup.
*
* posix_spawn(), not fork(): xcecd is multi-threaded (no-fork rule,
* .kiro/steering/tech.md "Process Creation") and a forked child that
* allocated or logged before exec could deadlock on a lock another thread
* held. The spawn/watchdog shape mirrors esd_script.c (the xceesd
* collector, same phase's global leg): pipe, poll(2) in 100ms slices,
* kill(SIGKILL) at elim_script_timeout_ms, waitpid reaps -- a hung script
* must not stall the sync thread.
*/
#define _GNU_SOURCE
#include "elim.h"
#include <errno.h>
#include <fcntl.h>
#include <math.h>
#include <poll.h>
#include <pthread.h>
#include <signal.h>
#include <spawn.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
#include <cyaml/cyaml.h>
#include "resource_sync.h"
#include "xcelog.h"
/* Watchdog, milliseconds of wall clock a collection command may run before
* it is killed (design §3.3b: 脚本超时看门狗 kill). */
int elim_script_timeout_ms = 10 * 1000;
/* ── config parse (libcyaml, esd_config.c pattern) ────────────────────── */
typedef struct yaml_elim_source_s {
char* name;
char* command;
long interval; /* 0 = every collect tick */
} yaml_elim_source_t;
typedef struct yaml_elim_config_s {
yaml_elim_source_t* external_resources;
unsigned external_resources_count;
} yaml_elim_config_t;
static const cyaml_schema_field_t elim_source_fields_schema[] = {
CYAML_FIELD_STRING_PTR("name", CYAML_FLAG_POINTER, yaml_elim_source_t,
name, 1, CYAML_UNLIMITED),
CYAML_FIELD_STRING_PTR("command", CYAML_FLAG_POINTER, yaml_elim_source_t,
command, 1, CYAML_UNLIMITED),
CYAML_FIELD_INT("interval", CYAML_FLAG_OPTIONAL, yaml_elim_source_t,
interval),
CYAML_FIELD_END
};
static const cyaml_schema_value_t elim_source_schema = {
CYAML_VALUE_MAPPING(CYAML_FLAG_DEFAULT, yaml_elim_source_t,
elim_source_fields_schema)
};
static const cyaml_schema_field_t elim_config_fields_schema[] = {
CYAML_FIELD_SEQUENCE("external_resources", CYAML_FLAG_POINTER,
yaml_elim_config_t, external_resources,
&elim_source_schema, 0, CYAML_UNLIMITED),
CYAML_FIELD_END
};
static const cyaml_schema_value_t elim_config_schema = {
CYAML_VALUE_MAPPING(CYAML_FLAG_POINTER, yaml_elim_config_t,
elim_config_fields_schema)
};
int elim_config_load_path(const char* dir, const char* name,
elim_config_t* cfg_out)
{
static cyaml_config_t cyaml_cfg = {
.mem_fn = cyaml_mem,
.log_level = CYAML_LOG_WARNING,
.flags = CYAML_CFG_IGNORE_UNKNOWN_KEYS,
};
yaml_elim_config_t* ycfg = NULL;
cyaml_err_t err;
char* path = NULL;
elim_config_t cfg;
int i;
int rc;
if ((dir == NULL) || (name == NULL) || (cfg_out == NULL)) {
return -EINVAL;
}
memset(cfg_out, 0, sizeof(*cfg_out));
memset(&cfg, 0, sizeof(cfg));
if (asprintf(&path, "%s/%s", dir, name) < 0) {
return -ENOMEM;
}
err = cyaml_load_file(path, &cyaml_cfg, &elim_config_schema,
(void**)&ycfg, NULL);
free(path);
if (err != CYAML_OK) {
/* Map file-not-found apart from bad-file so the startup log can
* say which (esd_config.c's probe). */
char probe[512];
FILE* f;
snprintf(probe, sizeof(probe), "%s/%s", dir, name);
f = fopen(probe, "r");
if (f == NULL) {
return -ENOENT;
}
fclose(f);
return -EINVAL;
}
if (ycfg == NULL) {
return -EINVAL;
}
cfg.nsrc = (int)ycfg->external_resources_count;
cfg.src = NULL;
if (cfg.nsrc > 0) {
cfg.src = (elim_source_t*)calloc((size_t)cfg.nsrc,
sizeof(elim_source_t));
if (cfg.src == NULL) {
cyaml_free(&cyaml_cfg, &elim_config_schema, ycfg, 0);
return -ENOMEM;
}
}
rc = 0;
for (i = 0; i < cfg.nsrc; i++) {
yaml_elim_source_t* ys = &(ycfg->external_resources[i]);
/* A source without name or command can never report -- fail
* loudly at load, not silently at runtime. */
if ((ys->name == NULL) || (ys->command == NULL)) {
rc = -EINVAL;
break;
}
cfg.src[i].name = strdup(ys->name);
cfg.src[i].command = strdup(ys->command);
cfg.src[i].interval = (int)ys->interval;
if ((cfg.src[i].name == NULL) || (cfg.src[i].command == NULL)) {
rc = -ENOMEM;
break;
}
}
cyaml_free(&cyaml_cfg, &elim_config_schema, ycfg, 0);
if (rc < 0) {
elim_config_fini(&cfg);
return rc;
}
*cfg_out = cfg;
return 0;
}
int elim_config_load(const char* path, elim_config_t* cfg_out)
{
char* dir;
char* name;
char* slash;
int rc;
if ((path == NULL) || (cfg_out == NULL)) {
return -EINVAL;
}
dir = strdup(path);
if (dir == NULL) {
return -ENOMEM;
}
slash = strrchr(dir, '/');
if (slash == NULL) {
name = dir;
dir = strdup(".");
if (dir == NULL) {
free(name);
return -ENOMEM;
}
} else {
*slash = '\0';
name = slash + 1;
if (*name == '\0') {
free(dir);
return -EINVAL;
}
if (dir == name - 1) {
/* path was "/name" -- dir is "" */
free(dir);
dir = strdup("/");
if (dir == NULL) {
return -ENOMEM;
}
}
}
rc = elim_config_load_path(dir, name, cfg_out);
free(dir);
return rc;
}
void elim_config_fini(elim_config_t* cfg)
{
int i;
if (cfg == NULL) {
return;
}
for (i = 0; i < cfg->nsrc; i++) {
free(cfg->src[i].name);
free(cfg->src[i].command);
}
free(cfg->src);
memset(cfg, 0, sizeof(*cfg));
}
/* ── output parse (elim contract) ─────────────────────────────────────── */
/*
* The elim contract: the first non-empty line carries "<name> <value>" or
* a bare number; leading/trailing whitespace ignored; anything else is
* garbage -> NAN (the caller owns the fail-open policy). Same shape as
* esd_parse_script_output -- a number is a number wherever it stands.
*/
double elim_parse_script_output(const char* out)
{
const char* p;
char* endp = NULL;
double v;
if (out == NULL) {
return NAN;
}
p = out;
while (*p != '\0') {
while ((*p == ' ') || (*p == '\t') || (*p == '\r') || (*p == '\n')) {
p++;
}
if (*p == '\0') {
break;
}
v = strtod(p, &endp);
if ((endp != NULL) && (endp != p)) {
return v;
}
/* skip this non-numeric token and try the next one */
while ((*p != '\0') && (*p != ' ') && (*p != '\t') && (*p != '\r')
&& (*p != '\n')) {
p++;
}
}
return NAN;
}
/* ── script runner (posix_spawn + watchdog, esd_script.c shape) ───────── */
static int elim_script_spawn(const char* command, int out_fd, pid_t* pid_out)
{
posix_spawn_file_actions_t fa;
posix_spawnattr_t attr;
sigset_t none, all;
char* argv[] = { (char*)"sh", (char*)"-c", (char*)command, NULL };
extern char** environ;
pid_t pid;
int rc;
sigemptyset(&none);
sigfillset(&all);
rc = posix_spawn_file_actions_init(&fa);
if (rc == 0) {
rc = posix_spawn_file_actions_adddup2(&fa, out_fd, STDOUT_FILENO);
}
if (rc == 0) {
rc = posix_spawn_file_actions_addopen(&fa, STDERR_FILENO, "/dev/null",
O_WRONLY, 0);
}
if (rc == 0) {
rc = posix_spawnattr_init(&attr);
}
if (rc == 0) {
rc = posix_spawnattr_setflags(
&attr, POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF);
}
if (rc == 0) {
rc = posix_spawnattr_setsigmask(&attr, &none);
}
if (rc == 0) {
rc = posix_spawnattr_setsigdefault(&attr, &all);
}
if (rc == 0) {
rc = posix_spawn(&pid, "/bin/sh", &fa, &attr, argv, environ);
}
posix_spawnattr_destroy(&attr);
posix_spawn_file_actions_destroy(&fa);
if (rc != 0) {
return -rc;
}
*pid_out = pid;
return 0;
}
int elim_script_collect(const char* command, char* out, size_t outsz)
{
int fds[2];
pid_t pid = -1;
size_t total = 0;
int elapsed_ms = 0;
int rc;
int wstatus = 0;
struct pollfd pfd;
if ((command == NULL) || (*command == '\0') || (out == NULL)
|| (outsz == 0)) {
return -EINVAL;
}
out[0] = '\0';
if (pipe(fds) != 0) {
rc = -errno;
XCECD_LOG(XCE_LOG_ERROR, "elim: pipe failed (%s)", strerror(-rc));
return rc;
}
rc = elim_script_spawn(command, fds[1], &pid);
close(fds[1]); /* parent's copy -- child holds the write end now */
if (rc < 0) {
close(fds[0]);
XCECD_LOG(XCE_LOG_ERROR, "elim: spawn failed for \"%s\" (%s)", command,
strerror(-rc));
return rc;
}
/* Bound the read at the watchdog deadline: a script that hangs must
* not stall the sync thread -- the value it was going to report
* belongs to a dead world anyway. */
pfd.fd = fds[0];
pfd.events = POLLIN;
while (total < outsz - 1) {
int pret = poll(&pfd, 1, 100);
if (pret < 0) {
if (errno == EINTR) {
continue;
}
rc = -errno;
XCECD_LOG(XCE_LOG_ERROR, "elim: poll failed for \"%s\" (%s)",
command, strerror(-rc));
kill(pid, SIGKILL);
waitpid(pid, &wstatus, 0);
close(fds[0]);
return rc;
}
if (pret == 0) {
elapsed_ms += 100;
if (elapsed_ms >= elim_script_timeout_ms) {
XCECD_LOG(XCE_LOG_WARN,
"elim: command \"%s\" exceeded %d ms -- killing "
"it, value not refreshed this round",
command, elim_script_timeout_ms);
kill(pid, SIGKILL);
waitpid(pid, &wstatus, 0);
close(fds[0]);
return -ETIMEDOUT;
}
continue;
}
if (pfd.revents & (POLLERR | POLLHUP)) {
ssize_t n = read(fds[0], out + total, outsz - 1 - total);
if (n > 0) {
total += (size_t)n;
}
break;
}
if (pfd.revents & POLLIN) {
ssize_t n = read(fds[0], out + total, outsz - 1 - total);
if (n <= 0) {
break; /* EOF or error: done reading */
}
total += (size_t)n;
}
}
out[total] = '\0';
close(fds[0]);
/* reap regardless of which path brought us here */
while (waitpid(pid, &wstatus, 0) < 0) {
if (errno != EINTR) {
break;
}
}
if (total == 0) {
XCECD_LOG(XCE_LOG_WARN,
"elim: command \"%s\" produced no output", command);
return -ENODATA;
}
return 0;
}
/* ── getter plumbing: per-source cache behind the context-free API ────── */
/* resource_sync's getter signature is context-free (char* (*)(void)) -- the
* per-source state (command, cached value, next-refresh deadline) lives in
* this static array, indexed by registration order. MAX_LOAD_INDICES (64)
* bounds the registry; the config loader's own parse bounds us to a sane
* subset of that, and elim_register_sources' capacity failure surfaces at
* startup. */
typedef struct {
char* command;
int interval; /* seconds; 0 = refresh every collect */
double value; /* last good parse */
bool have_value;
time_t next_run; /* epoch after which the script re-runs */
} elim_source_state_t;
static elim_source_state_t* g_elim_states = NULL;
static int g_elim_nstates = 0;
static pthread_mutex_t g_elim_mutex = PTHREAD_MUTEX_INITIALIZER;
static char* elim_getter_run(int slot)
{
elim_source_state_t* st = &g_elim_states[slot];
char buf[4096];
int rc;
double v;
char* out_str;
time_t now;
/* interval cache: 0 = run the script on every collect; otherwise the
* script re-runs only once its interval has elapsed since the last
* good run, and the cached value is served in between (the send
* cadence is the sync thread's 15-tick cycle regardless -- the interval
* governs script-run frequency, not push frequency). */
now = time(NULL);
if (st->have_value && (st->interval > 0) && (now < st->next_run)) {
out_str = (char*)malloc(64);
if (out_str == NULL) {
return NULL;
}
snprintf(out_str, 64, "%f", st->value);
return out_str;
}
rc = elim_script_collect(st->command, buf, sizeof(buf));
if (rc < 0) {
/* Fail-open: keep the last known value (the collector died but the
* host is still there; the sync thread keeps reporting). */
if (!st->have_value) {
return strdup("0");
}
return strdup("");
}
v = elim_parse_script_output(buf);
if (isnan(v)) {
XCECD_LOG(XCE_LOG_WARN,
"elim: command \"%s\" output did not parse as a value "
"(got %.60s)", st->command, buf);
if (!st->have_value) {
return strdup("0");
}
return strdup("");
}
st->value = v;
st->have_value = true;
st->next_run = time(NULL) + st->interval;
/* collect_load_indices converts per valtype (fvalue: atof+free) and
* owns the string from here. */
out_str = (char*)malloc(64);
if (out_str == NULL) {
return NULL;
}
snprintf(out_str, 64, "%f", v);
return out_str;
}
/* One trampoline per possible slot; the getter table indexes the static
* state array. (The registry API carries no user pointer.) */
static char* elim_getter_0(void) { return elim_getter_run(0); }
static char* elim_getter_1(void) { return elim_getter_run(1); }
static char* elim_getter_2(void) { return elim_getter_run(2); }
static char* elim_getter_3(void) { return elim_getter_run(3); }
static char* elim_getter_4(void) { return elim_getter_run(4); }
static char* elim_getter_5(void) { return elim_getter_run(5); }
static char* elim_getter_6(void) { return elim_getter_run(6); }
static char* elim_getter_7(void) { return elim_getter_run(7); }
static char* elim_getter_8(void) { return elim_getter_run(8); }
static char* elim_getter_9(void) { return elim_getter_run(9); }
static char* elim_getter_10(void) { return elim_getter_run(10); }
static char* elim_getter_11(void) { return elim_getter_run(11); }
static char* elim_getter_12(void) { return elim_getter_run(12); }
static char* elim_getter_13(void) { return elim_getter_run(13); }
static char* elim_getter_14(void) { return elim_getter_run(14); }
static char* elim_getter_15(void) { return elim_getter_run(15); }
static char* (*const g_elim_getters[])(void) = {
elim_getter_0, elim_getter_1, elim_getter_2, elim_getter_3,
elim_getter_4, elim_getter_5, elim_getter_6, elim_getter_7,
elim_getter_8, elim_getter_9, elim_getter_10, elim_getter_11,
elim_getter_12, elim_getter_13, elim_getter_14, elim_getter_15,
};
#define ELIM_MAX_SOURCES ((int)(sizeof(g_elim_getters) / sizeof(g_elim_getters[0])))
/* There is exactly one getter call site -- collect_load_indices() on the
* sync thread -- but registration happens on main before that thread
* exists; a mutex keeps the state array vs. getter table relationship
* honest without serializing the runs themselves. */
int elim_register_sources(const elim_config_t* cfg)
{
int i;
int rc = 0;
if ((cfg == NULL) || (cfg->nsrc < 0)
|| ((cfg->nsrc > 0) && (cfg->src == NULL))) {
return -EINVAL;
}
if (cfg->nsrc == 0) {
return 0;
}
if (cfg->nsrc > ELIM_MAX_SOURCES) {
XCECD_LOG(XCE_LOG_ERROR,
"elim: %d external resources configured, at most %d "
"supported", cfg->nsrc, ELIM_MAX_SOURCES);
return -EINVAL;
}
pthread_mutex_lock(&g_elim_mutex);
/* The 11 builtins register lazily at collect time; an elim source named
* "mem" registered before them would win the name race and SHADOW a
* builtin (the builtin's own registration then bounces as a dup).
* Force the builtin init first, so a name collision is a clean -EEXIST
* here, at startup, where the operator sees it. */
resource_sync_ensure_builtin_load_indices();
g_elim_states = (elim_source_state_t*)calloc(
(size_t)cfg->nsrc, sizeof(elim_source_state_t));
if (g_elim_states == NULL) {
pthread_mutex_unlock(&g_elim_mutex);
return -ENOMEM;
}
g_elim_nstates = cfg->nsrc;
for (i = 0; i < cfg->nsrc; i++) {
if (!resource_sync_register_load_index(
cfg->src[i].name, Value_u_fvalue, g_elim_getters[i], false)) {
/* Duplicate name (another source, or one of the 11 builtins --
* a user script must not shadow "mem"/"r15s"/...): surface the
* misconfiguration at startup, not as a silent missing value. */
XCECD_LOG(XCE_LOG_ERROR,
"elim: cannot register external resource '%s' -- "
"name already in use",
cfg->src[i].name);
rc = -EEXIST;
break;
}
g_elim_states[i].command = strdup(cfg->src[i].command);
if (g_elim_states[i].command == NULL) {
resource_sync_unregister_load_index(cfg->src[i].name);
rc = -ENOMEM;
break;
}
g_elim_states[i].interval = cfg->src[i].interval;
g_elim_states[i].have_value = false;
g_elim_states[i].next_run = 0;
XCECD_LOG(XCE_LOG_INFO,
"elim: registered external resource '%s' (command: %s, "
"interval: %ds)",
cfg->src[i].name, g_elim_states[i].command,
g_elim_states[i].interval);
}
pthread_mutex_unlock(&g_elim_mutex);
if (rc < 0) {
elim_unregister_sources(cfg);
}
return rc;
}
void elim_unregister_sources(const elim_config_t* cfg)
{
int i;
if ((cfg == NULL) || (cfg->src == NULL)) {
return;
}
pthread_mutex_lock(&g_elim_mutex);
for (i = 0; i < cfg->nsrc; i++) {
resource_sync_unregister_load_index(cfg->src[i].name);
}
for (i = 0; i < g_elim_nstates; i++) {
free(g_elim_states[i].command);
}
free(g_elim_states);
g_elim_states = NULL;
g_elim_nstates = 0;
pthread_mutex_unlock(&g_elim_mutex);
}

48
src/worker/src/main.c
View file

@ -394,10 +394,12 @@ int main(int argc, char* argv[])
char logfile[1024] = { 0 };
char* logdir = NULL;
const char* config_file = NULL;
static struct option long_opts[] = {
{ "peers", required_argument, NULL, OPT_PEERS },
{ "version", no_argument, NULL, OPT_VERSION },
{ "config", required_argument, NULL, 'c' },
{ "help", no_argument, NULL, 'h' },
{ NULL, 0, NULL, 0 }
};
@ -418,12 +420,15 @@ int main(int argc, char* argv[])
STR(VERSION_BUILD_TIME),
STR(VERSION_DIRTY_FINGERPRINT));
while ((opt = getopt_long(argc, argv, "n:l:vh", long_opts, NULL))
while ((opt = getopt_long(argc, argv, "n:l:c:vh", long_opts, NULL))
!= -1) {
switch (opt) {
case 'n':
config.virutal_name = optarg;
break;
case 'c':
config_file = optarg;
break;
case 'l':
if (optarg) {
if (logdir) {
@ -450,9 +455,13 @@ int main(int argc, char* argv[])
return 0;
case 'h':
default:
printf("Usage: %s [-n name] [-l log_dir] [-v] "
printf("Usage: %s [-n name] [-l log_dir] [-c config] [-v] "
"[--peers ctrl-1=host:port,...] "
"[-h help]\n"
" -c FILE elim collector config (external "
"resources reported by\n"
" scripts on this host; design "
"external-license-resources §3.3b)\n"
" --peers LIST HA controller candidate list: "
"[name=]host:port,...\n"
" When omitted, the XCE_CONTROLLER_PEERS "
@ -638,6 +647,41 @@ int main(int argc, char* argv[])
}
#endif
/* elim collectors (design §3.3b): load the external_resources config
* before xcecd_init() registers its sources. No -c (or a file without
* the section) means zero external resources -- plain worker behavior.
* A config that is present but BROKEN stops startup loudly: a worker
* whose operator asked for elim reporting must not silently run
* without it (same fail-loud-at-load rule as esd's config). */
if (config_file != NULL) {
elim_config_t elim_cfg;
int erc = elim_config_load(config_file, &elim_cfg);
if (erc == -ENOENT) {
XCECD_LOG(XCE_LOG_ERROR,
"elim config file not found: %s", config_file);
fprintf(stderr, "xcecd: elim config file not found: %s\n",
config_file);
FREEUP(logdir);
return 1;
}
if (erc < 0) {
XCECD_LOG(XCE_LOG_ERROR,
"invalid elim config file %s (rc=%d) -- a source "
"needs both name and command", config_file, erc);
fprintf(stderr,
"xcecd: invalid elim config file %s (rc=%d)\n",
config_file, erc);
FREEUP(logdir);
return 1;
}
config.elim.nsrc = elim_cfg.nsrc;
config.elim.src = elim_cfg.src; /* ownership -> xcecd_config_t */
XCECD_LOG(XCE_LOG_INFO,
"elim: %d external resource(s) from %s",
config.elim.nsrc, config_file);
}
if (xcecd_config_init(&config) != 0) {
XCECD_LOG(XCE_LOG_ERROR, "Failed to initialize daemon configuration.");
return 1;

60
src/worker/src/resource_sync.c
View file

@ -77,8 +77,17 @@ static void init_builtin_load_indices(void)
* on resource_sync_thread()'s single dedicated pthread in production,
* or single-threaded test code). */
g_load_indices_capacity = MAX_LOAD_INDICES;
g_load_indices = (load_index_info_t*)calloc(g_load_indices_capacity,
sizeof(load_index_info_t));
if (g_load_indices == NULL) {
g_load_indices = (load_index_info_t*)calloc(g_load_indices_capacity,
sizeof(load_index_info_t));
}
/* Never realloc over a populated array: since resource_sync_ensure_
* builtin_load_indices() this function also runs with the array already
* alive (external entries registered first -- elim), and replacing it
* leaked the old one while leaving g_load_indices_count pointing past
* the fresh calloc (strcmp on garbage slots, segfault). The dup check
* in resource_sync_register_load_index() makes re-running the
* registration list idempotent. */
// register built-in load indices
resource_sync_register_load_index("r15s", Value_u_fvalue, node_info_get_15s,
@ -105,6 +114,28 @@ static void init_builtin_load_indices(void)
true);
}
/**
* @brief Ensures the registry exists AND the 11 builtin load indices are
* registered in it, whichever external module (elim) queries or
* registers first. Idempotent; single-threaded callers only in
* practice (startup registration / the sync thread's collect), but
* safe under the registry mutex discipline its own callers keep.
*/
void resource_sync_ensure_builtin_load_indices(void)
{
if (g_load_indices == NULL) {
init_builtin_load_indices();
return;
}
/* init_builtin_load_indices() ran but the array was later torn down
* (resource_sync_destroy's cleanup) and a register-before-init caller
* re-populated it with external entries only -- the builtins must go
* in before the first report. */
if (g_load_indices_count < BUILTIN_LOAD_INDICES_COUNT) {
init_builtin_load_indices();
}
}
/**
* @brief Registers a new load index.
* @param name The name of the load index.
@ -123,6 +154,27 @@ bool resource_sync_register_load_index(const char* name, int valtype,
pthread_mutex_lock(&g_load_indices_mutex);
/* elim (design §3.3b): external registration happens at daemon startup
* -- BEFORE the sync thread's first collect_load_indices() lazily
* allocated the array. A capacity-0 registry here used to reject every
* such registration with "Maximum load indices capacity reached (0)",
* silently dropping every configured external resource. Allocate the
* registry on first use, same as the lazy builtin init does; init_
* builtin_load_indices() itself stays where it is (collect-time) so the
* 11 builtins keep registering exactly once per populated-registry
* window (bug-1756's contract -- its only caller's NULL check). */
if (g_load_indices == NULL) {
g_load_indices_capacity = MAX_LOAD_INDICES;
g_load_indices = (load_index_info_t*)calloc(
g_load_indices_capacity, sizeof(load_index_info_t));
if (g_load_indices == NULL) {
XCECD_LOG(XCE_LOG_ERROR, "Failed to allocate load indices array");
g_load_indices_capacity = 0;
pthread_mutex_unlock(&g_load_indices_mutex);
return false;
}
}
for (int i = 0; i < g_load_indices_count; i++) {
if (strcmp(g_load_indices[i].name, name) == 0) {
XCECD_LOG(XCE_LOG_WARN, "Load index '%s' already exists", name);
@ -194,9 +246,7 @@ bool resource_sync_unregister_load_index(const char* name)
static xce_resval_t* collect_load_indices(int* count)
{
if (g_load_indices == NULL) {
init_builtin_load_indices();
}
resource_sync_ensure_builtin_load_indices();
pthread_mutex_lock(&g_load_indices_mutex);

20
src/worker/src/xcecd.c
View file

@ -395,6 +395,22 @@ int xcecd_init(const xcecd_config_t* config)
* xcecd_start() already ran. */
xcecd_ctx.cluster_cfg = config->cluster_cfg;
/* elim collectors (external-license-resources design §3.3b): register
* the configured external resources as custom load indices BEFORE the
* resource_sync thread starts (xcecd_start() -> resource_sync_start()),
* so the first 15-tick report already carries them. Ownership of
* config->elim moves here (config is read-only). */
xcecd_ctx.elim.nsrc = config->elim.nsrc;
xcecd_ctx.elim.src = config->elim.src;
if (xcecd_ctx.elim.nsrc > 0) {
if (elim_register_sources(&xcecd_ctx.elim) < 0) {
XCECD_LOG(XCE_LOG_ERROR,
"Failed to register elim external resources -- "
"check external_resources config");
goto cleanup_remote_host;
}
}
/* thread pool initializing */
xcecd_ctx.thread_pool = thread_pool_create(config->thread_pool_size);
if (!xcecd_ctx.thread_pool) {
@ -457,6 +473,7 @@ cleanup_process_manager:
cleanup_thread_pool:
thread_pool_destroy(xcecd_ctx.thread_pool);
cleanup_remote_host:
elim_unregister_sources(&xcecd_ctx.elim);
free(xcecd_ctx.remote_host);
free(xcecd_ctx.work_dir);
free(xcecd_ctx.db_path);
@ -575,6 +592,9 @@ void xcecd_cleanup(void)
resource_sync_stop(xcecd_ctx.resource_sync);
resource_sync_destroy(xcecd_ctx.resource_sync);
}
/* elim teardown AFTER resource_sync_destroy (which tears the registry
* down anyway -- this frees the source state and command strings). */
elim_unregister_sources(&xcecd_ctx.elim);
if (xcecd_ctx.db) {
xcecd_db_close(xcecd_ctx.db);
}

7
src/worker/src/xcecd_cgroups.c
View file

@ -1450,7 +1450,12 @@ xce_job_rusages_t* collect_job_rusages(uint64_t job_id)
return NULL;
}
snprintf(unit_name, sizeof(unit_name), "xcecd-worker-%lu.service", job_id);
/* Build the unit name the same way start_systemd_job() did (four-segment
* fmt_jobid_ex form) -- a bare %lu of the packed jobid never matches the
* real unit, so every property lookup below missed and all four counters
* stayed 0 (merge cb0c7fe9 regression). */
make_job_sd_unit_name(unit_name, sizeof(unit_name) - 1, "xcecd-worker",
job_id);
char* escaped_unit_name = replace_chars(unit_name);
if (!escaped_unit_name) {
XCECD_LOG(XCE_LOG_ERROR, "Failed to escape unit name.");

8
src/worker/src/xcecd_exdata.c
View file

@ -452,8 +452,12 @@ void apply_env_vars(env_var_t** env_vars, uint64_t jobid)
XCECD_WORKER_LOG(XCE_LOG_DEBUG, "Applying environment variables...");
// Set some default XCE environment variables
char jobid_str[32];
snprintf(jobid_str, sizeof(jobid_str), "%lu", jobid);
// XCE_JOBID is the four-segment written identity (cluster-apiserver-
// timestamp-sequence, fmt_jobid) -- a bare decimal of the packed uint64
// is unusable in scripts (output-file names, xcecli job selection all
// speak the written form; same convention as every log line).
char jobid_str[64];
fmt_jobid(jobid_str, sizeof(jobid_str), jobid);
if (setenv("XCE_JOBID", jobid_str, 1) != 0) {
XCECD_WORKER_LOG(XCE_LOG_ERROR,
"Error setting environment variable: %s",

11
src/worker/src/xrun.c
View file

@ -69,7 +69,16 @@ int xrun_ipc_send_remote_start(const char* hostname, const char* command)
char* jobid_str = getenv("XCE_JOBID");
if (jobid_str) {
request_msg.data.remote_start.jobid = strtoull(jobid_str, NULL, 10);
/* XCE_JOBID is the four-segment written identity (fmt_jobid);
* parse_jobid() accepts exactly that form (a bare decimal of the
* packed uint64 no longer round-trips through the written form). */
if (parse_jobid(&request_msg.data.remote_start.jobid, jobid_str)
!= 1) {
XCECD_WORKER_LOG(XCE_LOG_WARN,
"xrun_ipc: cannot parse XCE_JOBID '%s'; "
"remote start will not match the job",
jobid_str);
}
}
if (send(sock_fd, &request_msg, sizeof(request_msg), 0) < 0) {

17
src/worker/testsuite/meson.build
View file

@ -45,6 +45,23 @@ test('worker resource_sync test', resource_sync_test,
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])
# external-license-resources design §3.3b: the elim collector -- a
# config-driven script runner (posix_spawn + watchdog) that registers
# user-defined per-host load indices into resource_sync's registry at
# worker startup. White-box #includes elim.c AND resource_sync.c (the
# registry capacity-0-before-init trap is only observable against the real
# registry); see test_elim.c's header comment.
elim_test = executable('elim_test',
'test_elim.c',
include_directories: [worker_test_incdir],
dependencies: worker_test_deps,
install: false)
test('worker elim test', elim_test,
suite: 'worker',
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])
# bug-1554: real maxswap (sysinfo() totalswap)/maxtmp (statvfs("/tmp"))
# detection -- white-box #include of hardware_monitor.c, same convention as
# resource_sync_test above.

566
src/worker/testsuite/test_elim.c Normal file
View file

@ -0,0 +1,566 @@
/*
* test_elim.c -- elim collector module (external-license-resources design
* §3.3b): xcecd gains a config-driven script collector that registers
* user-defined load indices into resource_sync's registry (the previously
* dead external-register API), so per-host "external resources" reported by
* user scripts on each host flow the live load-indices push chain
* (collect_load_indices -> xcecd_send_load_indices -> controller
* update_val_res -> scheduler RESVALS_DELTA -> sched_check_host_satisfy_).
*
* White-box #include of ../src/elim.c (same convention as
* test_resource_sync.c) plus ../src/esd_script-shaped spawn code lives in
* elim.c itself; the config parse is exercised against real temp files
* (esd_test's mkdtemp pattern). The spawn/watchdog half IS unit tested here
* (unlike esd's test, which deferred it to E2E): a real /bin/sh child with
* a real timeout kill is deterministic and cheap, and the watchdog is the
* one piece of elim.c that can hang the sync thread if it regresses.
*
* resource_sync.c is ALSO #included (after elim.c) to exercise the real
* registry rather than a stub: the capacity-0-before-init trap this phase
* fixes (elim registers at startup, before the sync thread's first collect
* lazily allocated the array) is only observable against the real code.
*/
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <unistd.h>
#include <unity/unity.h>
#include "resource_sync.h"
#include "xcecd.h"
/* Real globals the white-box resource_sync.c references (test_resource_sync.c
* convention). */
xcecd_context_t xcecd_ctx;
/* Stubs for the 11 builtin load-index getters (node_information.c, not
* linked here). */
static char* stub_getter(void)
{
return strdup("1.0");
}
char* node_info_get_15s(void) { return stub_getter(); }
char* node_info_get_1m(void) { return stub_getter(); }
char* node_info_get_15m(void) { return stub_getter(); }
char* node_info_get_ut(void) { return stub_getter(); }
char* node_info_get_pg(void) { return stub_getter(); }
char* node_info_get_ls(void) { return stub_getter(); }
char* node_info_get_it(void) { return stub_getter(); }
char* node_info_get_swap(void) { return stub_getter(); }
char* node_info_get_mem(void) { return stub_getter(); }
char* node_info_get_tmp(void) { return stub_getter(); }
char* node_info_get_io(void) { return stub_getter(); }
/* Spy on the send (controller_connector.c, not linked here). */
static int g_send_load_indices_call_count = 0;
bool xcecd_send_load_indices(xce_resval_t** vals, int count)
{
(void)vals;
(void)count;
g_send_load_indices_call_count++;
return true;
}
/* resource_sync.c's resync gate (xcecd_resync.c, not linked here). */
static bool g_test_resync_pending = false;
bool xcecd_resync_is_pending(void) { return g_test_resync_pending; }
#include "../src/elim.c"
#include "../src/resource_sync.c"
static char g_tmpdir[256];
void setUp(void)
{
snprintf(g_tmpdir, sizeof(g_tmpdir), "/tmp/xce_elim_testXXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(g_tmpdir));
memset(&xcecd_ctx, 0, sizeof(xcecd_ctx));
xcecd_ctx.virtual_name = (char*)"t1";
g_test_resync_pending = false;
g_send_load_indices_call_count = 0;
}
void tearDown(void)
{
/* Drop any registrations this test left so the next test starts from a
* clean registry: resource_sync_destroy on a throwaway object runs
* resource_sync_cleanup_load_indices() (static in the #included
* resource_sync.c, reachable only through this public entry). */
resource_sync_destroy(resource_sync_create());
if (g_tmpdir[0] != '\0') {
char path[300];
snprintf(path, sizeof(path), "rm -rf %s", g_tmpdir);
if (system(path) != 0) {
/* best effort */
}
g_tmpdir[0] = '\0';
}
}
static void write_file(const char* dir, const char* name, const char* content)
{
char path[512];
FILE* f;
snprintf(path, sizeof(path), "%s/%s", dir, name);
f = fopen(path, "w");
TEST_ASSERT_NOT_NULL(f);
fputs(content, f);
fclose(f);
}
/* ── elim config parse ────────────────────────────────────────────────── */
static void test_elim_config_load_full_file(void)
{
elim_config_t cfg;
int rc;
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: dynres1\n"
" command: /opt/xce/scripts/report-dynres1.sh\n"
" interval: 30\n"
" - name: gpu_free\n"
" command: /usr/local/bin/gpu-free.sh\n"
" interval: 60\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(2, cfg.nsrc);
TEST_ASSERT_EQUAL_STRING("dynres1", cfg.src[0].name);
TEST_ASSERT_EQUAL_STRING("/opt/xce/scripts/report-dynres1.sh",
cfg.src[0].command);
TEST_ASSERT_EQUAL_INT(30, cfg.src[0].interval);
TEST_ASSERT_EQUAL_STRING("gpu_free", cfg.src[1].name);
TEST_ASSERT_EQUAL_INT(60, cfg.src[1].interval);
elim_config_fini(&cfg);
}
static void test_elim_config_interval_defaults_when_missing(void)
{
elim_config_t cfg;
int rc;
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: dynres1\n"
" command: /opt/xce/scripts/r.sh\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, cfg.nsrc);
/* 0 = "every send tick" -- the getter cache refreshes each time the
* sync thread collects. */
TEST_ASSERT_EQUAL_INT(0, cfg.src[0].interval);
elim_config_fini(&cfg);
}
static void test_elim_config_missing_file_is_enoent(void)
{
elim_config_t cfg;
int rc;
rc = elim_config_load_path(g_tmpdir, "no-such-file.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(-ENOENT, rc);
}
static void test_elim_config_missing_command_is_einval(void)
{
elim_config_t cfg;
int rc;
/* A source without a command can never report -- fail loudly at load,
* not silently at runtime (esd_config.c's validation rule). */
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: dynres1\n"
" interval: 30\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
}
static void test_elim_config_missing_name_is_einval(void)
{
elim_config_t cfg;
int rc;
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - command: /opt/xce/scripts/r.sh\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
}
static void test_elim_config_empty_or_absent_section_is_ok(void)
{
elim_config_t cfg;
int rc;
/* A worker config with no external_resources key is the normal case for
* every host without elim scripts -- must load to zero sources, not
* error. */
write_file(g_tmpdir, "elim.yaml", "external_resources: []\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, cfg.nsrc);
TEST_ASSERT_NULL(cfg.src);
elim_config_fini(&cfg);
}
/* ── elim output parse ────────────────────────────────────────────────── */
static void test_elim_parse_output_name_value_line(void)
{
double v;
/* The elim contract: first non-empty line carries "<name> <value>". */
v = elim_parse_script_output("dynres1 42\n");
TEST_ASSERT_FLOAT_WITHIN(1e-6, 42.0, v);
v = elim_parse_script_output("\n\n# comment-ish noise 7 8\ngpu_free 5.5\n");
TEST_ASSERT_FLOAT_WITHIN(1e-6, 7.0, v);
}
static void test_elim_parse_output_bare_value_line(void)
{
double v;
v = elim_parse_script_output("12\n");
TEST_ASSERT_FLOAT_WITHIN(1e-6, 12.0, v);
v = elim_parse_script_output(" 3.5 \n");
TEST_ASSERT_FLOAT_WITHIN(1e-6, 3.5, v);
}
static void test_elim_parse_output_garbage_is_nan(void)
{
TEST_ASSERT(isnan(elim_parse_script_output("no numbers here\n")));
TEST_ASSERT(isnan(elim_parse_script_output("")));
TEST_ASSERT(isnan(elim_parse_script_output(NULL)));
}
/* ── registry wiring: register at startup, before any collect ─────────── */
/* THE trap this phase fixes: before it, resource_sync_register_load_index
* called at daemon startup (when g_load_indices is still NULL -- the array
* is only allocated by collect_load_indices' lazy init_builtin_load_indices,
* i.e. on the sync thread's first 15th tick) failed with "capacity reached
* (0)" and every elim source silently vanished. Registration must
* self-initialize. */
static void test_register_load_index_before_lazy_init_succeeds(void)
{
bool ok;
/* No collect_load_indices() call ever happened in this process yet. */
TEST_ASSERT_NULL_MESSAGE(g_load_indices,
"precondition: registry not yet initialized");
ok = resource_sync_register_load_index("dynres1", Value_u_fvalue,
stub_getter, false);
TEST_ASSERT_TRUE_MESSAGE(
ok, "registering before the lazy builtin init must not fail with "
"'capacity reached (0)'");
/* And the value must actually flow: collect_load_indices now includes
* the elim index among the builtins. */
xce_resval_t* vals;
int count = 0;
int found = 0;
vals = collect_load_indices(&count);
TEST_ASSERT_NOT_NULL(vals);
for (int i = 0; i < count; i++) {
if (strcmp(vals[i].name, "dynres1") == 0) {
found++;
TEST_ASSERT_EQUAL_STRING("t1", vals[i].host);
TEST_ASSERT_EQUAL_INT(Value_u_fvalue, vals[i].val->which);
TEST_ASSERT_FLOAT_WITHIN(1e-6, 1.0, vals[i].val->u.fvalue);
}
/* The host tag is the join identity for every report (bug-1569). */
TEST_ASSERT_EQUAL_STRING("t1", vals[i].host);
}
TEST_ASSERT_EQUAL_INT(1, found);
free_resvals(vals, count);
}
static void test_elim_sources_register_and_flow_through_registry(void)
{
elim_config_t cfg;
int rc;
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: dynres1\n"
" command: echo dynres1 7\n"
" interval: 0\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
rc = elim_register_sources(&cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
/* The getter runs the script: first collect refreshes the cache. */
xce_resval_t* vals;
int count = 0;
int found = 0;
vals = collect_load_indices(&count);
TEST_ASSERT_NOT_NULL(vals);
for (int i = 0; i < count; i++) {
if (strcmp(vals[i].name, "dynres1") == 0) {
found++;
TEST_ASSERT_EQUAL_INT(Value_u_fvalue, vals[i].val->which);
/* "echo dynres1 7" -> name-value line -> 7.0 */
TEST_ASSERT_FLOAT_WITHIN(1e-6, 7.0, vals[i].val->u.fvalue);
}
}
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, found, "the elim-registered index must reach collect_load_indices");
free_resvals(vals, count);
}
static void test_elim_register_rejects_duplicate_name(void)
{
elim_config_t cfg;
int rc;
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: dynres1\n"
" command: echo dynres1 1\n"
" - name: dynres1\n"
" command: echo dynres1 2\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
/* The registry rejects the dup; elim_register_sources reports the
* failure so the operator sees the misconfiguration at startup. */
TEST_ASSERT_EQUAL_INT(-EEXIST, elim_register_sources(&cfg));
}
static void test_elim_register_rejects_builtin_name_override(void)
{
elim_config_t cfg;
int rc;
/* "mem" is one of the 11 builtins -- a user script must not shadow it
* (the registry rejects the dup; the error surfaces at startup). */
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: mem\n"
" command: echo mem 999\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(-EEXIST, elim_register_sources(&cfg));
}
static void test_elim_unregister_sources_cleans_up(void)
{
elim_config_t cfg;
int rc;
write_file(g_tmpdir, "elim.yaml",
"external_resources:\n"
" - name: dynres1\n"
" command: echo dynres1 1\n");
rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, elim_register_sources(&cfg));
TEST_ASSERT_TRUE(resource_sync_unregister_load_index("dynres1"));
TEST_ASSERT_FALSE(resource_sync_unregister_load_index("dynres1"));
}
/* ── the script runner: real child, real watchdog ─────────────────────── */
static void test_elim_script_collect_runs_command(void)
{
char out[128];
int rc;
const char* script;
script = "/bin/sh -c 'echo dynres1 42'";
rc = elim_script_collect(script, out, sizeof(out));
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_FLOAT_WITHIN(1e-6, 42.0, elim_parse_script_output(out));
}
static void test_elim_script_collect_timeout_kills_child(void)
{
char out[128];
int rc;
int saved_timeout;
const char* script;
/* A script that hangs must be killed at the watchdog deadline, not
* stall the sync thread forever (the esd_script watchdog contract). */
script = "/bin/sh -c 'sleep 60'";
saved_timeout = elim_script_timeout_ms;
elim_script_timeout_ms = 500;
rc = elim_script_collect(script, out, sizeof(out));
elim_script_timeout_ms = saved_timeout;
TEST_ASSERT_EQUAL_INT_MESSAGE(-ETIMEDOUT, rc,
"a hung script must hit the watchdog");
/* And the child must actually be reaped -- no zombie left behind. */
TEST_ASSERT_EQUAL_INT_MESSAGE(-1, waitpid(-1, NULL, WNOHANG),
"no unreaped child may survive the kill");
}
static void test_elim_script_collect_empty_output_is_enodata(void)
{
char out[128];
int rc;
rc = elim_script_collect("/bin/sh -c 'exit 0'", out, sizeof(out));
TEST_ASSERT_EQUAL_INT(-ENODATA, rc);
}
/* ── getter cache semantics: interval ─────────────────────────────────── */
static void test_elim_getter_refreshes_per_interval(void)
{
elim_config_t cfg;
xce_resval_t* vals;
int count = 0;
static char counter_script[512];
static char yaml[1024];
int v1 = -1, v2 = -1;
resource_sync_t* sync;
/* A script whose output changes on every run: the getter must return
* the FRESH value after the interval elapses (refresh), and the cached
* one before it (no re-run). */
snprintf(counter_script, sizeof(counter_script), "%s/counter.sh", g_tmpdir);
{
FILE* f = fopen(counter_script, "w");
TEST_ASSERT_NOT_NULL(f);
fprintf(f,
"#!/bin/sh\n"
"if [ -f %s.n ]; then n=$(cat %s.n); n=$((n+1)); else n=1; fi\n"
"echo $n > %s.n\n"
"echo \"dynres1 $n\"\n",
counter_script, counter_script, counter_script);
fclose(f);
chmod(counter_script, 0755);
}
snprintf(yaml, sizeof(yaml),
"external_resources:\n"
" - name: dynres1\n"
" command: %s\n"
" interval: 3\n",
counter_script);
write_file(g_tmpdir, "elim.yaml", yaml);
TEST_ASSERT_EQUAL_INT(0, elim_config_load_path(g_tmpdir, "elim.yaml", &cfg));
TEST_ASSERT_EQUAL_INT(0, elim_register_sources(&cfg));
/* First collect: getter runs the script (counter=1). */
vals = collect_load_indices(&count);
TEST_ASSERT_NOT_NULL(vals);
for (int i = 0; i < count; i++) {
if (strcmp(vals[i].name, "dynres1") == 0) {
v1 = (int)vals[i].val->u.fvalue;
}
}
free_resvals(vals, count);
TEST_ASSERT_EQUAL_INT(1, v1);
/* Collect again immediately (interval=3 not yet elapsed): the cached
* value is returned, the counter script does NOT advance. */
vals = collect_load_indices(&count);
TEST_ASSERT_NOT_NULL(vals);
for (int i = 0; i < count; i++) {
if (strcmp(vals[i].name, "dynres1") == 0) {
v2 = (int)vals[i].val->u.fvalue;
}
}
free_resvals(vals, count);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, v2, "getter must serve the cached value within its interval");
/* After the interval elapses the getter re-runs the script. Drive it
* through a real resource_sync tick loop rather than sleeping 3s in
* wall-clock: seed the counter and run two 1s ticks. */
sync = resource_sync_create();
TEST_ASSERT_NOT_NULL(sync);
sync->load_report_counter = LOAD_INDICES_INTERVAL_DEFAULT - 1;
TEST_ASSERT_TRUE(resource_sync_start(sync));
/* Two ticks cover the 1s sleep-per-tick of the sync thread; interval 3
* seconds measured from the first run has now elapsed. */
sleep(4);
vals = collect_load_indices(&count);
TEST_ASSERT_NOT_NULL(vals);
for (int i = 0; i < count; i++) {
if (strcmp(vals[i].name, "dynres1") == 0) {
v2 = (int)vals[i].val->u.fvalue;
}
}
free_resvals(vals, count);
resource_sync_stop(sync);
resource_sync_destroy(sync);
TEST_ASSERT_EQUAL_INT_MESSAGE(
2, v2, "getter must refresh the value once its interval elapses");
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_elim_config_load_full_file);
RUN_TEST(test_elim_config_interval_defaults_when_missing);
RUN_TEST(test_elim_config_missing_file_is_enoent);
RUN_TEST(test_elim_config_missing_command_is_einval);
RUN_TEST(test_elim_config_missing_name_is_einval);
RUN_TEST(test_elim_config_empty_or_absent_section_is_ok);
RUN_TEST(test_elim_parse_output_name_value_line);
RUN_TEST(test_elim_parse_output_bare_value_line);
RUN_TEST(test_elim_parse_output_garbage_is_nan);
RUN_TEST(test_register_load_index_before_lazy_init_succeeds);
RUN_TEST(test_elim_sources_register_and_flow_through_registry);
RUN_TEST(test_elim_register_rejects_duplicate_name);
RUN_TEST(test_elim_register_rejects_builtin_name_override);
RUN_TEST(test_elim_unregister_sources_cleans_up);
RUN_TEST(test_elim_script_collect_runs_command);
RUN_TEST(test_elim_script_collect_timeout_kills_child);
RUN_TEST(test_elim_script_collect_empty_output_is_enodata);
RUN_TEST(test_elim_getter_refreshes_per_interval);
return UNITY_END();
}

17
src/worker/testsuite/test_exdata_jobindex.c
View file

@ -6,7 +6,8 @@
* the submission's own variables. XCE_JOBINDEX must be the element number of
* an array-tagged job ID (a one-element array included) and 0 for an
* ordinary job; a submission-supplied XCE_JOBINDEX is ignored with a WARN
* naming the job and the variable; XCE_JOBID handling is unchanged.
* naming the job and the variable; XCE_JOBID is the four-segment written
* identity (fmt_jobid), unchanged in override behavior.
*
* White-box #include of ../src/xcecd_exdata.c (same convention as the other
* tests in this suite). XCECD_WORKER_LOG is redefined to a capture hook
@ -203,20 +204,20 @@ static void test_similar_names_still_applied(void)
unsetenv("xce_jobindex");
}
/* 6.5: XCE_JOBID is the decimal job ID; a submission's own XCE_JOBID still
* overrides it, as before, with no WARN. */
/* 6.5: XCE_JOBID is the four-segment written job identity (fmt_jobid); a
* submission's own XCE_JOBID still overrides it, as before, with no WARN. */
static void test_jobid_unchanged(void)
{
env_var_t* vars = NULL;
char dec[32];
char idstr[64];
snprintf(dec, sizeof(dec), "%lu", (unsigned long)ARRAY_ELEMENT(4));
fmt_jobid(idstr, sizeof(idstr), ARRAY_ELEMENT(4));
apply_env_vars(&vars, ARRAY_ELEMENT(4));
expect_env("XCE_JOBID", dec);
expect_env("XCE_JOBID", idstr);
snprintf(dec, sizeof(dec), "%lu", (unsigned long)ORDINARY_JOB);
fmt_jobid(idstr, sizeof(idstr), ORDINARY_JOB);
apply_env_vars(&vars, ORDINARY_JOB);
expect_env("XCE_JOBID", dec);
expect_env("XCE_JOBID", idstr);
captured_count = 0;
vars = push_var(vars, "XCE_JOBID", "user-value");

11
src/worker/testsuite/test_xcecd_startup_order.c
View file

@ -161,6 +161,17 @@ bool resource_sync_start(resource_sync_t* sync)
}
void resource_sync_stop(resource_sync_t* sync) { (void)sync; }
/* elim surface (elim.c, not linked here -- this test stubs every subsystem
* constructor the same way): xcecd_init() registers the configured sources
* (zero sources in this test's configs, so the real impl would no-op, but
* the symbol must resolve); xcecd_cleanup() unregisters. */
int elim_register_sources(const elim_config_t* cfg)
{
(void)cfg;
return 0;
}
void elim_unregister_sources(const elim_config_t* cfg) { (void)cfg; }
/* Referenced by xcecd_process_command()/xcecd_cleanup() -- never invoked at
* runtime by this test (only xcecd_init()/xcecd_start() are called), but
* their bodies are compiled unconditionally so the symbols must resolve. */

9
src/xceadmin/src/xceadmin/build.py
View file

@ -208,7 +208,9 @@ def _format_resource(r) -> str:
ordtype = _RESOURCE_ORDTYPE_NAMES.get(r.ordtype, r.ordtype)
consumable = bool(r.flags & mapping.RESOURCE_FLAG_CONSUMABLE)
release = bool(r.flags & mapping.RESOURCE_FLAG_RELEASE)
builtin = bool(r.flags & mapping.RESOURCE_FLAG_BUILTIN)
shared = bool(r.flags & mapping.RESOURCE_FLAG_SHARED)
external = bool(r.flags & mapping.RESOURCE_FLAG_EXTERNAL)
parts = [f"name={r.name}", f"type={valtype}", f"order={ordtype}"]
if r.desc:
parts.append(f"desc={r.desc!r}")
@ -216,6 +218,13 @@ def _format_resource(r) -> str:
parts.append(f"interval={r.interval}")
parts.append(f"consumable={consumable}")
parts.append(f"release={release}")
# builtin/external change the resource's semantics (whose values count,
# where they come from) -- always visible when set, never shown bare
# false: absent means the plain user-resource default.
if builtin:
parts.append("builtin=True")
if external:
parts.append("external=True")
if shared:
parts.append("shared=true")
if len(r.distributions) > 0:

124
src/xceadmin/src/xceadmin/cli.py
View file

@ -27,11 +27,14 @@ traceback.
from __future__ import annotations
import json
import os
import time
from collections.abc import Callable
from datetime import datetime, timezone
from importlib.metadata import PackageNotFoundError
from importlib.metadata import version as _package_version
from pathlib import Path
import click
from click.core import ParameterSource
@ -43,6 +46,7 @@ from xcecommon.errors import (
SchedAdminError,
TooManyRedirectsError,
XceAdminError,
XceExtresStatusError,
)
from xceadmin.build import build_cluster_config, dump_cluster_config, write_output
from xceadmin.push import probe_config, push_config
@ -372,6 +376,126 @@ def scheduler_dump(file: str) -> None:
click.echo(text, nl=False) # dump() already ends its own text with "\n" (5.20).
# ---------------------------------------------------------------------------
# extres status (external-license-resources Phase 2 item 4, design.md 3.5):
# read the status JSON xceesd writes on its own host and render it for the
# operator. A LOCAL-file command like scheduler dump -- no --controller, no
# wire: the daemon's status file is the observability surface until the
# PR-2 proxy formalizes a status channel (design.md 3.6; a temporary
# remote protocol would be redesigned out from under us by PR-2's
# provider-lease model). Run it on the daemon's host.
# ---------------------------------------------------------------------------
# Where the deployed esd.yaml's status_file default points. Matches
# esd_config.c's status_file default and the deploy docs; overridable with
# --status-file (the t1 test cluster uses a /tmp path instead).
DEFAULT_STATUS_FILE = "/var/run/xce/esd-status.json"
def _format_extres_status(payload: dict, now: int | None = None) -> str:
"""Renders one decoded esd status object as operator text.
``now`` pins the wall clock for deterministic tests (None = time.time()).
Rendering rules:
- daemon identity line: name + pid
- uptime and push counters on their own lines
- per-source rows: NAME value=<v|-> age=<n>s [-- STALE] or
"no value yet" when last_value is null.
"""
if now is None:
now = int(time.time())
lines = [
f"daemon: {payload.get('daemon', 'xceesd')} (pid {payload.get('pid', '?')})",
]
started_at = payload.get("started_at", 0)
if isinstance(started_at, int) and started_at > 0:
uptime = max(now - started_at, 0)
started_str = datetime.fromtimestamp(started_at, tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%S")
lines.append(f"started at: {started_str} (up {uptime}s)")
lines.append(
f"pushes: {payload.get('pushes_ok', 0)} ok, "
f"{payload.get('pushes_fail', 0)} failed"
)
sources = payload.get("sources", [])
if not sources:
lines.append("sources: (none configured)")
return "\n".join(lines)
lines.append("sources:")
for src in sources:
name = src.get("name", "?")
if src.get("last_value") is None:
lines.append(f" {name}: no value yet")
continue
last_ok = src.get("last_ok_time", 0)
age = max(now - last_ok, 0) if isinstance(last_ok, int) and last_ok > 0 else 0
stale = " -- STALE" if src.get("stale") else ""
lines.append(f" {name}: value={src.get('last_value')} age={age}s{stale}")
return "\n".join(lines)
def _read_extres_status(path: str) -> dict:
"""Reads and validates one esd status file. Raises
:class:`XceExtresStatusError` for every local failure mode (unreadable
path, unparseable JSON, JSON that is not an esd status object) so the
CLI layer never sees a raw OSError/JSONDecodeError.
"""
try:
raw = Path(path).read_text()
except OSError as exc:
raise XceExtresStatusError(
f"cannot read extres status file {path!r}: {exc}", path=path, reason=str(exc)
) from exc
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
raise XceExtresStatusError(
f"extres status file {path!r} is not valid JSON: {exc}", path=path, reason=str(exc)
) from exc
if not isinstance(payload, dict) or "sources" not in payload or "daemon" not in payload:
raise XceExtresStatusError(
f"extres status file {path!r} is not an xceesd status object "
"(missing 'daemon' or 'sources')",
path=path,
reason="wrong schema",
)
return payload
@cli.group()
def extres() -> None:
"""Inspect the external-resources daemon (xceesd)."""
@extres.command("status")
@click.option(
"--status-file",
default=None,
help=f"Path of the xceesd status JSON (default: {DEFAULT_STATUS_FILE}).",
)
def extres_status(status_file: str | None) -> None:
"""Report xceesd's status from its local status file (run on the daemon's host)."""
# Resolved at call time, not decoration time, so tests (and wrappers)
# can re-point the default -- click freezes a non-None default= into
# the command's own default_value at decoration.
if status_file is None:
status_file = DEFAULT_STATUS_FILE
try:
payload = _read_extres_status(status_file)
click.echo(_format_extres_status(payload))
except XceAdminError as e:
raise click.ClickException(e.message) from e
except KeyboardInterrupt:
raise click.ClickException("cancelled") from None
def main() -> None:
"""Console-script entry point (see pyproject.toml [project.scripts])."""
cli()

20
src/xceadmin/src/xcecommon/errors.py
View file

@ -222,3 +222,23 @@ class SchedAdminError(XceAdminError):
self.file = file
self.event_count = event_count
self.started_at = started_at
class XceExtresStatusError(XceAdminError):
"""The xceesd status file (``xceadmin extres status``) could not be
read or decoded (external-license-resources Phase 2 item 4, design.md
3.5; raised by ``xceadmin.cli._read_extres_status``).
Covers every local-file failure mode: the path doesn't exist / isn't
readable, the bytes aren't valid JSON, or the JSON parses but isn't an
esd status object (missing ``daemon``/``sources`` -- e.g. someone
pointed ``--status-file`` at some other JSON file). Same local-file
concern as :class:`ReplayFileError`/:class:`ClusterBinFileError`: no
network framing involved -- ``extres status`` reads the status JSON
xceesd writes on its own host.
"""
def __init__(self, message: str, path: str, reason: str) -> None:
super().__init__(message)
self.path = path
self.reason = reason

35
src/xceadmin/src/xcecommon/mapping.py
View file

@ -64,12 +64,12 @@ reason -- never a silent drop.
``Resource`` (``resources.yaml``)
``valtype``/``ordtype``/``flags`` ARE mapped from YAML
``type``/``order``/``consumable``/``release`` -- the encoding is
documented, just not in ``controller.capnp``/design.md: it lives in
``type``/``order``/``consumable``/``release``/``builtin`` -- the encoding
is documented, just not in ``controller.capnp``/design.md: it lives in
``include/xce/types.h`` (``XCE_VAL_*``, ``xce_order_t``,
``XCE_RESF_CONSUMABLE``/``XCE_RESF_RELEASE``) and is actively read at
runtime (e.g. ``scheduler.c`` copies these three fields straight out
of the parsed resource). See :data:`RESOURCE_VALTYPE`,
``XCE_RESF_CONSUMABLE``/``XCE_RESF_RELEASE``/``XCE_RESF_BUILTIN``) and
is actively read at runtime (e.g. ``scheduler.c`` copies these three
fields straight out of the parsed resource). See :data:`RESOURCE_VALTYPE`,
:data:`RESOURCE_ORDTYPE`, and the flag constants below for the exact
values (also reused by ``xceadmin.build.dump_cluster_config`` to render
these int-encoded fields back to their YAML names). ``idx`` and
@ -205,10 +205,16 @@ _XCE_SCALAR_FIELDS: tuple[tuple[str, str], ...] = (
RESOURCE_VALTYPE: dict[str, int] = {"string": 0, "numeric": 1, "boolean": 2}
RESOURCE_ORDTYPE: dict[str, int] = {"increase": 0, "decrease": 1, "na": 2}
# resources.yaml ``consumable``/``release`` -> Resource.flags bits, mirroring
# include/xce/types.h's XCE_RESF_CONSUMABLE/XCE_RESF_RELEASE.
# resources.yaml ``consumable``/``release``/``builtin`` -> Resource.flags
# bits, mirroring include/xce/types.h's XCE_RESF_CONSUMABLE/
# XCE_RESF_RELEASE/XCE_RESF_BUILTIN.
RESOURCE_FLAG_CONSUMABLE = 0x80
RESOURCE_FLAG_RELEASE = 0x20
# XCE_RESF_BUILTIN: the resource's values are produced by hardcoded worker/
# controller reporters (resource_sync.c's load indices, worker.c's host
# attributes and slots) -- display/accounting code uses it to tell those
# from user-declared resources of the same name space.
RESOURCE_FLAG_BUILTIN = 0x01
# Set together on a resourcemap item whose location is exactly ["all"]
# (cluster-wide static shared pool): XCE_RESF_SHARED | XCE_RESF_GLOBAL in
@ -221,6 +227,14 @@ RESOURCE_FLAG_SHARED = 0x08
RESOURCE_FLAG_GLOBAL = 0x04
RESOURCE_FLAG_SHARED_GLOBAL = RESOURCE_FLAG_SHARED | RESOURCE_FLAG_GLOBAL
# Set together with SHARED|GLOBAL on a resourcemap item whose location is
# exactly ["external"] (external-license-resources): XCE_RESF_EXTERNAL in
# include/xce/types.h. The pool is cluster-wide but its values come only
# from the extres daemon's free-count pushes (host=="" resval deltas) -- the
# controller must NOT seed it from a config value (seed_sharedinst_from_config
# skips EXTERNAL) and the scheduler's admission formula skips its inuse term.
RESOURCE_FLAG_EXTERNAL = 0x10
def new_cluster_config_builder() -> "capnp._DynamicStructBuilder":
@ -413,6 +427,8 @@ def _build_resource(item: dict, target: "capnp._DynamicStructBuilder") -> None:
flags |= RESOURCE_FLAG_CONSUMABLE
if item.get("release"):
flags |= RESOURCE_FLAG_RELEASE
if item.get("builtin"):
flags |= RESOURCE_FLAG_BUILTIN
resourcemap = item.get("resourcemap") or []
if resourcemap:
distributions = target.init("distributions", len(resourcemap))
@ -429,6 +445,11 @@ def _build_resource(item: dict, target: "capnp._DynamicStructBuilder") -> None:
# scheduler can admit jobs against today (see module
# docstring, Resource).
flags |= RESOURCE_FLAG_SHARED_GLOBAL
elif locations == ["external"]:
# Cluster-wide externally-reported pool
# (external-license-resources): daemon pushes the free
# values; config declares only the name/shape.
flags |= RESOURCE_FLAG_SHARED_GLOBAL | RESOURCE_FLAG_EXTERNAL
target.flags = flags

14
src/xceadmin/tests/fixtures/full/configs/resources.yaml vendored
View file

@ -12,6 +12,7 @@ resources:
type: numeric
interval: 60
consumable: false
builtin: true
order: decrease
description: "available memory in MB (built-in dynamic load index)"
@ -197,3 +198,16 @@ resources:
- name: gstring
description: string resource
type: string
# external-license-resources: cluster-wide license pool reported by
# the extres daemon (xceesd) -- config declares only the name/shape,
# the free counts come from the daemon's pushes
- name: extlic
description: externally reported license pool
type: numeric
interval: 0
consumable: true
order: decrease
resourcemap:
- location:
- external

BIN
src/xceadmin/tests/fixtures/full/expected.bin vendored
View file

Binary file not shown.

275
src/xceadmin/tests/test_cli_extres.py Normal file
View file

@ -0,0 +1,275 @@
"""Tests for the ``extres`` command group (external-license-resources
Phase 2 item 4, design.md 3.5 "``xceadmin extres status``").
``extres status`` is a LOCAL-file command -- the inverse slice of
``scheduler dump``: where ``dump()`` proved xceadmin can be a local-file
decoder with no network at all, ``extres status`` reads the status JSON
``xceesd`` writes on its own host (``esd_status_write``,
``src/esd/src/esd_status.c``) and formats it for the operator. No wire,
no ``--controller``, no TLS -- the daemon's status file is deliberately
the observability surface until the PR-2 proxy formalizes a status
channel (design.md 3.6; a temporary remote protocol would be
redesigned out from under us by PR-2's provider-lease model).
Split into its own file (rather than appended to ``test_cli.py``, 1187
lines at the time of writing) to keep both files under the project's
800-line guideline -- the same split precedent as
``test_cli_scheduler_mock.py``.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from click.testing import CliRunner
from xceadmin import cli as cli_module
from xceadmin.cli import cli
# ---------------------------------------------------------------------------
# Fixtures: realistic status JSON in the exact shape esd_status_write()
# emits (field order aside -- we parse, we never substring-match).
# ---------------------------------------------------------------------------
_NOW = 1_800_000_000 # a fixed "now" for deterministic age rendering
@pytest.fixture
def status_file(tmp_path: Path) -> Path:
"""A status file exactly as esd_status_write() writes it for two
sources: one fresh, one stale (last_ok_time 3600s in the past of
_NOW against the daemon's 150s default limit).
"""
path = tmp_path / "esd-status.json"
payload = {
"daemon": "xceesd",
"pid": 12345,
"started_at": _NOW - 7200,
"generated_at": _NOW,
"pushes_ok": 142,
"pushes_fail": 3,
"sources": [
{
"name": "Verilog_XL",
"last_value": 2.0,
"last_ok_time": _NOW - 5,
"stale": False,
},
{
"name": "VCS",
"last_value": 7.0,
"last_ok_time": _NOW - 3600,
"stale": True,
},
],
}
path.write_text(json.dumps(payload))
return path
def _freeze_now(monkeypatch: pytest.MonkeyPatch) -> None:
"""Pin time.time() to _NOW so age rendering is deterministic."""
monkeypatch.setattr(cli_module.time, "time", lambda: _NOW)
# ---------------------------------------------------------------------------
# Formatting helper: pure functions tested directly (the _format_record_
# result precedent in test_cli.py).
# ---------------------------------------------------------------------------
def test_format_status_renders_daemon_identity_and_counters() -> None:
from xceadmin.cli import _format_extres_status
payload = json.loads(
(Path(__file__).parent / "data" / "esd-status-sample.json").read_text()
) if (Path(__file__).parent / "data" / "esd-status-sample.json").exists() else {
"daemon": "xceesd",
"pid": 12345,
"started_at": _NOW - 7200,
"generated_at": _NOW,
"pushes_ok": 142,
"pushes_fail": 3,
"sources": [],
}
text = _format_extres_status(payload, now=_NOW)
assert "xceesd" in text
assert "12345" in text
assert "142" in text
assert "3" in text
def test_format_status_marks_stale_source_with_age() -> None:
from xceadmin.cli import _format_extres_status
payload = {
"daemon": "xceesd",
"pid": 1,
"started_at": _NOW - 100,
"generated_at": _NOW,
"pushes_ok": 0,
"pushes_fail": 0,
"sources": [
{
"name": "VCS",
"last_value": 7.0,
"last_ok_time": _NOW - 3600,
"stale": True,
}
],
}
text = _format_extres_status(payload, now=_NOW)
assert "VCS" in text
assert "STALE" in text
assert "3600s" in text
def test_format_status_fresh_source_shows_age_only() -> None:
from xceadmin.cli import _format_extres_status
payload = {
"daemon": "xceesd",
"pid": 1,
"started_at": _NOW - 100,
"generated_at": _NOW,
"pushes_ok": 0,
"pushes_fail": 0,
"sources": [
{
"name": "Verilog_XL",
"last_value": 2.0,
"last_ok_time": _NOW - 5,
"stale": False,
}
],
}
text = _format_extres_status(payload, now=_NOW)
assert "Verilog_XL" in text
assert "STALE" not in text
assert "5s" in text
def test_format_status_source_with_no_value_yet() -> None:
from xceadmin.cli import _format_extres_status
payload = {
"daemon": "xceesd",
"pid": 1,
"started_at": _NOW - 100,
"generated_at": _NOW,
"pushes_ok": 0,
"pushes_fail": 0,
"sources": [
{
"name": "Verilog_XL",
"last_value": None,
"last_ok_time": 0,
"stale": False,
}
],
}
text = _format_extres_status(payload, now=_NOW)
assert "Verilog_XL" in text
assert "no value yet" in text
# ---------------------------------------------------------------------------
# CLI wiring: arg parsing, file reading, error translation.
# ---------------------------------------------------------------------------
def test_cli_help_shows_extres_command() -> None:
runner = CliRunner()
result = runner.invoke(cli, ["--help"])
assert result.exit_code == 0
assert "extres" in result.output
def test_extres_status_reads_local_file(status_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
_freeze_now(monkeypatch)
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)])
assert result.exit_code == 0, result.output
assert "xceesd" in result.output
assert "Verilog_XL" in result.output
assert "VCS" in result.output
def test_extres_status_defaults_to_etc_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""With no --status-file, the default path is the deployed default
(/var/run/xce/esd-status.json -- where esd.yaml's production
status_file default points); a missing default file is a clean
error, not a traceback.
"""
monkeypatch.setattr(
cli_module, "DEFAULT_STATUS_FILE", str(tmp_path / "nonexistent.json")
)
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status"])
assert result.exit_code != 0
assert "nonexistent.json" in result.output
def test_extres_status_missing_file_clean_error(tmp_path: Path) -> None:
runner = CliRunner()
result = runner.invoke(
cli, ["extres", "status", "--status-file", str(tmp_path / "gone.json")]
)
assert result.exit_code != 0
assert "gone.json" in result.output
assert "Traceback" not in result.output
def test_extres_status_invalid_json_clean_error(tmp_path: Path) -> None:
bad = tmp_path / "bad.json"
bad.write_text("{not json")
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)])
assert result.exit_code != 0
assert "Traceback" not in result.output
def test_extres_status_wrong_schema_clean_error(tmp_path: Path) -> None:
"""JSON that parses but is not an esd status object (e.g. someone
pointed --status-file at some other JSON) fails loud and clean.
"""
bad = tmp_path / "other.json"
bad.write_text(json.dumps({"hello": "world"}))
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)])
assert result.exit_code != 0
assert "Traceback" not in result.output
def test_extres_status_no_controller_needed(monkeypatch: pytest.MonkeyPatch, status_file: Path) -> None:
"""The command must not touch the controller: XCE_CONTROLLER_PEERS
pointing at a dead endpoint changes nothing (the file is local).
"""
monkeypatch.setenv("XCE_CONTROLLER_PEERS", "127.0.0.1:1")
monkeypatch.setenv("XCEADMIN_CONTROLLER", "127.0.0.1:1")
runner = CliRunner()
result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)])
assert result.exit_code == 0, result.output

26
src/xceadmin/tests/test_config_dump.py
View file

@ -49,6 +49,32 @@ def test_dump_lists_each_resource_with_decoded_type_and_order(full_cluster_bin):
assert "consumable=False" in text
def test_dump_renders_builtin_and_external_flags(full_cluster_bin):
"""The flag bits that change a resource's semantics must be visible in a
dump: builtin (worker/controller hardcode the values -- the repo fixture's
load indices and host attributes) and external (values come only from the
extres daemon's pushes -- the extlic pool shape). Both were silently
invisible before: every builtin resource dumped as a plain user resource,
and an external pool dumped as bare shared=true.
"""
text = build.dump_cluster_config(str(full_cluster_bin))
mem_line = next(l for l in text.splitlines() if "name=mem type=numeric" in l)
assert "builtin=True" in mem_line
# a user-declared resource must not show a bare "builtin=" at all
assert "builtin=False" not in text
import yaml as _yaml
configs = str(FIXTURES_DIR / "full" / "configs")
with open(configs + "/resources.yaml", encoding="utf-8") as fh:
names = [r["name"] for r in _yaml.safe_load(fh)["resources"]
if (r.get("resourcemap") or []) and
(r["resourcemap"][0].get("location") or []) == ["external"]]
assert names, "fixture must contain an external resource for this test"
for name in names:
line = next(l for l in text.splitlines() if f"name={name} " in l)
assert "external=True" in line, line
def test_dump_renders_mxj_auto_detect_sentinel_not_raw_uint32(full_cluster_bin):
"""The exact bug that motivated this feature: hosts.yaml's `default`
host has `max_slot: -1`, which wire-encodes as HOSTCONFIG_MXJ_AUTO_DETECT

71
src/xceadmin/tests/test_mapping.py
View file

@ -176,6 +176,29 @@ def test_map_list_section_resources_populates_mapped_fields_only(cluster_config)
assert r0.lastReport == 0
def test_map_list_section_resources_builtin_flag(cluster_config):
"""``builtin: true`` (resources.yaml) -> XCE_RESF_BUILTIN (0x01) OR'd
on top of the other bits: the load indices (mem/io/r1m...) and host
attributes (type/model/maxmem...) are produced by hardcoded worker/
controller reporters, and the flag tells them apart from user-declared
resources (the CLI's BUILTIN column renders it). Absent/false must
stay 0 -- the same convention as consumable/release.
"""
items = [
{"name": "mem", "type": "numeric", "interval": 60,
"consumable": True, "builtin": True, "description": "memory"},
{"name": "mylic", "type": "numeric", "consumable": True,
"builtin": False, "description": "user pool"},
{"name": "plain", "type": "numeric"}, # builtin absent
]
mapping.map_list_section(_spec_for("resources"), items, cluster_config)
r0, r1, r2 = cluster_config.resources
assert r0.flags == 0x80 | 0x01 # CONSUMABLE | BUILTIN
assert r1.flags == 0x80 # explicit false -> no BUILTIN bit
assert r2.flags == 0 # absent -> no BUILTIN bit
def test_map_list_section_noop_on_empty_or_missing_items(cluster_config):
mapping.map_list_section(_spec_for("resources"), [], cluster_config)
mapping.map_list_section(_spec_for("resources"), None, cluster_config)
@ -248,6 +271,42 @@ def test_map_list_section_resourcemap_maps_distributions_and_all_flag(cluster_co
assert r4.flags == 0
def test_map_list_section_resourcemap_external_location_flag(cluster_config):
"""resourcemap location==["external"] (external-license-resources): the
resource carries XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL
(0x08|0x04|0x10) -- cluster-wide pool, but its values come only from the
extres daemon's free-count pushes, never from a config distribution or
worker reports. A bare ["external"] (no value) also maps definition-only
-- the daemon owns the values entirely.
"""
items = [
{
"name": "verilog_xl",
"type": "numeric",
"consumable": True,
"resourcemap": [{"location": ["external"]}],
},
{
"name": "extlic_valued",
"type": "numeric",
"resourcemap": [{"location": ["external"], "value": 7}],
},
]
mapping.map_list_section(_spec_for("resources"), items, cluster_config)
r0, r1 = cluster_config.resources
# CONSUMABLE | SHARED | GLOBAL | EXTERNAL
assert r0.flags == 0x80 | 0x08 | 0x04 | 0x10
assert len(r0.distributions) == 1
assert [str(h) for h in r0.distributions[0].hosts] == ["external"]
# valued variant: same flags, the distribution value still maps (the
# controller refuses to seed it -- EXTERNAL skips seed_sharedinst)
assert r1.flags == 0x08 | 0x04 | 0x10
assert r1.distributions[0].value.u.fvalue == 7.0
def test_map_list_section_resourcemap_real_repo_configs(cluster_config):
"""The mapper handles the real src/configs/resources.yaml: every
resourcemap entry lands as a distribution, and the two [all]+value
@ -271,6 +330,18 @@ def test_map_list_section_resourcemap_real_repo_configs(cluster_config):
for name in ("staticres1", "staticres2", "staticres3", "host_quota"):
assert by_name[name].flags & (0x08 | 0x04) == 0
# builtin:true resources (the worker load indices + host attributes the
# repo config declares) carry XCE_RESF_BUILTIN; user-declared shapes
# (staticres*, extlic) do not.
for name in ("slots", "mem", "r1m", "r15m", "r15s", "ut", "pg", "ls", "it",
"swp", "io", "tmp", "type", "model", "cpuf", "maxcpus",
"maxmem", "maxswap", "maxtmp", "ndisks"):
assert by_name[name].flags & 0x01 == 0x01, f"{name} missing BUILTIN"
for name in ("dynres1", "dynres2", "staticres1", "staticres2", "staticres3",
"staticres4", "res1", "sres1", "bigmem", "strres1", "strres2",
"host_quota", "gstring", "extlic"):
assert by_name[name].flags & 0x01 == 0, f"{name} unexpectedly BUILTIN"
# ---------------------------------------------------------------------------
# map_list_section: users.yaml/hosts.yaml -> ugroups/hgroups (named groups)