wolf: mark bug-1973/1974/1975 fixed (commit 2fcacac2)
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
2fcacac2cb
commit
3eb1b9716d
4 changed files with 16 additions and 7 deletions
|
|
@ -63,7 +63,7 @@
|
|||
"timestamp": "2026-10-27T03:27:47Z",
|
||||
"last_seen": "2026-10-27T03:27:47Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).",
|
||||
"error_message": "cluster_build_resources aborted at first EEXIST: new resource defs never registered\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).",
|
||||
"file": "src/controller/cluster.c, src/controller/testsuite/test_config_resource_reconcile.c",
|
||||
"root_cause": "`cluster_build_resources()` looped over the decoded config's resource list\ncalling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)\nreturns NULL with errno=EEXIST when the name is already registered — which is\nthe COMMON case at activation (boot already registered some definitions). The\nold loop treated that rc as fatal and aborted the whole loop, so every\ndefinition AFTER the first already-registered one was never registered.",
|
||||
"fix": "Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under\none write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;\nRES_APPLY_CHANGED updates the registered object's definition fields\n(valtype/ordtype/flags/interval) IN PLACE — none of them touches the\nregistration slot (idx) or accumulated accounting; mirrors\nreconcile_config_queues' in-place refresh precedent (second live bug, see\nbug-1974 — kept here as one loop). Negative rc still propagates.\n\nVerification:\n- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +\n `test_reconcile_resources_hot_applies_changed_definition`\n (src/controller/testsuite/test_config_resource_reconcile.c); RED before the\n fix (lic1 unregistered / flags stayed 0x00), GREEN after.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in\n sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at\n free=1; release re-admitted it.",
|
||||
|
|
@ -76,7 +76,9 @@
|
|||
"related_bugs": [
|
||||
"bug-1854"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"2fcacac2"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1973.md"
|
||||
},
|
||||
{
|
||||
|
|
@ -103,7 +105,9 @@
|
|||
"bug-1854",
|
||||
"bug-1973"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"2fcacac2"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1974.md"
|
||||
},
|
||||
{
|
||||
|
|
@ -116,7 +120,7 @@
|
|||
"timestamp": "2026-10-27T03:40:05Z",
|
||||
"last_seen": "2026-10-27T03:40:05Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xceesd never reconnects after controller restart when its value is unchanged (debounce suppresses pushes)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).",
|
||||
"error_message": "xceesd idle on dead connection after controller restart (stable value: no re-push)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).",
|
||||
"file": "src/esd/src/esd.c, src/lib/client.c, src/lib/client_internal.h",
|
||||
"root_cause": "Three mechanisms conspired:\n1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip\n push) — with a stable value it pushes NOTHING.\n2. A reconnect inside the client library only fires while INSIDE a push's\n failover call — no pushes, no reconnect.\n3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;\n with no push attempt it never runs.",
|
||||
"fix": "`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`\n(bug-1760's pure predicate — the transport's RDHUP handler sets\nXCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.\nThe write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the\nfull re-push contract for the failover path.\n\nVerification:\n- Test: covered by the live contract (client-side predicate already unit\n tested by bug-1760's suite); esd-side loop is integration-shaped (the\n trigger is a transport callback), so live E2E is the proof.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — controller killed and restarted; esd logged\n `connection broken -- forcing a full re-push to reconnect`, rotated to the\n new primary, re-pushed; `xcecli resources distribution list` shows extlic=4\n and a fresh rusage[extlic=3] job RUNS.",
|
||||
|
|
@ -131,7 +135,9 @@
|
|||
"bug-1973",
|
||||
"bug-1974"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"2fcacac2"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1975.md"
|
||||
}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -11,10 +11,11 @@ occurrences: 1
|
|||
files: ["src/controller/cluster.c", "src/controller/testsuite/test_config_resource_reconcile.c"]
|
||||
tags: ["config", "reconcile", "E2", "external-license"]
|
||||
related: ["bug-1854"]
|
||||
summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered on a running controller"
|
||||
fix_commits: ["2fcacac2"]
|
||||
summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered"
|
||||
---
|
||||
|
||||
# bug-1973: cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller
|
||||
# bug-1973: cluster_build_resources aborted at first EEXIST: new resource defs never registered
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ occurrences: 1
|
|||
files: ["src/controller/cluster.c", "src/controller/worker.c", "src/xceadmin/src/xcecommon/mapping.py"]
|
||||
tags: ["config", "flags", "E2", "external-license"]
|
||||
related: ["bug-1854", "bug-1973"]
|
||||
fix_commits: ["2fcacac2"]
|
||||
---
|
||||
|
||||
# bug-1974: config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ occurrences: 1
|
|||
files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"]
|
||||
tags: ["esd", "external-license", "reconnect", "E2"]
|
||||
related: ["bug-1760", "bug-1973", "bug-1974"]
|
||||
fix_commits: ["2fcacac2"]
|
||||
summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
|
||||
title: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue