myxce/.wolf/buglog.json

{
  "version": 1,
  "generated": "by scripts/buglog.py from .wolf/bugs/*.md -- do not edit",
  "bugs": [
    {
      "id": "bug-1971",
      "status": "fixed",
      "status_updated": "2026-10-22",
      "status_note": "two-choke archive in xce-build.sh; test harness green",
      "component": "build",
      "severity": "wrong-result",
      "timestamp": "2026-10-22T07:50:00Z",
      "last_seen": "2026-10-22T07:50:00Z",
      "occurrences": 2,
      "error_message": "next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable\n\n**Observed:** (t2, 2026-10-22) a full `make build` (asan profile) failed on a controller test case; a following `make clean` + `make build` passed 134/134. The failing run's `meson-logs/testlog.{txt,json,junit.xml}` had been destroyed twice over: `setup_build`'s unconditional `rm -rf \"$BUILD_DIR\"` deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites `testlog.*` on every `meson test` invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (`/tmp/test_xcecd_resync.log` and rotations) showed no failure signature, and no `sched_test`/Go test output is retained anywhere.\n\nReconstruction from filesystem timestamps alone (`/tmp/test_*.log` rotations, `meson-log.txt` build options line, `/usr/local/go` ctime, bash history) established the two-run timeline but could not name the failed case.\n\nThe same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.\n\n**Expected:** when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' `/tmp/xce_*` dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.\n\n**Impact:** intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp \"failed cases keep their dirs for post-mortem\" contract, since the next `make build` never looks for those dirs and a clean build gives the appearance of none existing.\n\n**Found by:** user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.",
      "file": "src/tools/xce-build.sh",
      "root_cause": "`setup_build()` in `src/tools/xce-build.sh` unconditionally `rm -rf \"$BUILD_DIR\"` on every non-incremental invocation, and `meson test` overwrites `$BUILD_DIR/meson-logs/testlog.*` on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.",
      "fix": "Two chokes in `src/tools/xce-build.sh` (this commit):\n\n1. `archive_previous_failure()` — runs at script start, before `setup_build`'s\n   wipe. If the marker `.meson_test_failed_rc` (written by the previous run's\n   EXIT trap) exists, the previous run's `meson-logs` are MOVED to\n   `$SRC_DIR/testlogs.archive/<BUILD_DIR>/prev-<stamp>-rc<N>/` — outside the\n   build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e.\n   not a concurrent session's) `/tmp/xce_*` FAILED-case fixture DIRS into\n   `prev-<stamp>-fixtures/`, and caps the archive at the newest 10 entries\n   per build dir.\n2. `finalize_failure_evidence()` — the EXIT trap (replaces the old\n   lock-release trap, which becomes its last statement). On rc≠0 with\n   RUN_TESTS=1, COPIES `meson-logs` to\n   `testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/` and writes the marker.\n   Copy-not-move: meson-logs still belongs to the live build dir for this\n   run's own inspection; the next run moves the copy out at its start.\n   Configure/compile failures (RUN_TESTS unset) archive nothing — those are\n   deterministic and self-explanatory on rerun; it's the testlog-bearing\n   flaky failures that need preservation.\n\nAdditionally `-type d` added to BOTH `/tmp/xce_*` sweeps (this one and\nbug-924's `prune_stale_tmp_fixtures`): mkdtemp fixtures are always\ndirectories (`/tmp/xce_<label>_XXXXXX`), but the bare name also matches\nregular files — the untyped find live-moved daemon debug logs\n(`/tmp/xce_scheduler.log`, `/tmp/xce_strace*.log`) into the archive on\n2026-10-22, and the bug-924 variant `rm -rf`s its matches.\n\nVerification:\n- Test: `src/tools/test-failure-evidence-archive.sh` — 6 scenarios /\n  11 assertions exercising the real function bodies (sourced with surgical\n  stubbing): failed run archives as this-*-rc3 + marker; success archives\n  nothing; next run moves prior failure to prev-*-rc3 + consumes marker;\n  aged fixture dirs move / fresh stay / regular files never touched;\n  retention caps at newest 10; RUN_TESTS=0 archives nothing. All green\n  (11 passed, 0 failed, bash, 2026-10-22).\n- Suites: n/a — this bug lives in the build script, not in C code; no meson\n  suite exercises it. `bash -n` syntax check green.\n- Real wipe path not exercised by the harness test (it stubs meson);\n  validated by inspection: the archive root is `$SRC_DIR/testlogs.archive/`,\n  which `rm -rf \"$BUILD_DIR\"` cannot reach.\n- Live: first t2/sandbox failing run after this commit leaves\n  `testlogs.archive/xce.build/this-*-rc*` with the testlog — that is the\n  evidence this bug asks for.",
      "tags": [
        "test",
        "flaky",
        "observability",
        "build"
      ],
      "related_bugs": [],
      "fix_commits": [
        "68c3ccd7"
      ],
      "record": ".wolf/bugs/bug-1971.md"
    },
    {
      "id": "bug-1972",
      "status": "fixed",
      "status_updated": "2026-10-26",
      "status_note": "",
      "component": "controller",
      "severity": "crash",
      "timestamp": "2026-10-26T09:46:58Z",
      "last_seen": "2026-10-26T09:46:58Z",
      "occurrences": 4,
      "error_message": "test_job_live_gc asserted destruction by reading freed memory\n\n**Observed:**\n`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`\n(test_job_live_gc.c:169):\n\n```\n==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290\nREAD of size 8 at 0x5110000da290 thread T0\n    #0 job_get_refcount ../lib/jobj.c:715\n    #1 test_sweep_purges_expired_done_job test_job_live_gc.c:169\nfreed by:\n    #3 job_dereference ../lib/jobj.c:705\n    #5 job_live_gc_sweep ../controller/job.c:4777\n```\n\n**Expected:** a correct sweep dereferences the purged job down to refcount 0\nand destroys it; the test must verify destruction without touching the freed\nobject.\n\n**Impact:** controller suite aborts (killed by SIGABRT); the whole\n`controller test` meson entry fails. Test-only — the production sweep\n(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)\nis correct.\n\n**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier\nruns all predate that commit).",
      "file": "src/controller/testsuite/test_job_live_gc.c",
      "root_cause": "The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs\nAFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the\nentire RTTI object (header + private + data in one calloc block). Reading the\nrefcount through the dangling `job_t*` is itself the use-after-free: the\nstronger the sweep does its job, the more surely the test crashes. The same\npattern is absent from test_apply_pump.c's refcount test because there the\njob is never destroyed.",
      "fix": "Move the refcount probe BEFORE the sweep and invert its meaning: assert\nrefcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,\nthen verify destruction by absence — `jlg_find_job(81001) == NULL` plus the\nidempotent second sweep (both already present). `expired` is never dereferenced\nafter the sweep; a comment marks the last safe point.\n\nVerification:\n- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic\n  ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero\n  ASAN reports).\n- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);\n  sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,\n  bug-free fixture staleness from fc33a5da's schema change, handled in the same\n  working tree).\n- Live: n/a (test-only defect; production sweep unchanged).",
      "tags": [
        "test-only",
        "uaf",
        "gc"
      ],
      "related_bugs": [],
      "fix_commits": [
        "f780a166"
      ],
      "record": ".wolf/bugs/bug-1972.md"
    },
    {
      "id": "bug-1973",
      "status": "fixed",
      "status_updated": "2026-10-27",
      "status_note": "",
      "component": "controller",
      "severity": "wrong-result",
      "timestamp": "2026-10-27T03:27:47Z",
      "last_seen": "2026-10-27T03:27:47Z",
      "occurrences": 1,
      "error_message": "cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).",
      "file": "src/controller/cluster.c, src/controller/testsuite/test_config_resource_reconcile.c",
      "root_cause": "`cluster_build_resources()` looped over the decoded config's resource list\ncalling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)\nreturns NULL with errno=EEXIST when the name is already registered — which is\nthe COMMON case at activation (boot already registered some definitions). The\nold loop treated that rc as fatal and aborted the whole loop, so every\ndefinition AFTER the first already-registered one was never registered.",
      "fix": "Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under\none write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;\nRES_APPLY_CHANGED updates the registered object's definition fields\n(valtype/ordtype/flags/interval) IN PLACE — none of them touches the\nregistration slot (idx) or accumulated accounting; mirrors\nreconcile_config_queues' in-place refresh precedent (second live bug, see\nbug-1974 — kept here as one loop). Negative rc still propagates.\n\nVerification:\n- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +\n  `test_reconcile_resources_hot_applies_changed_definition`\n  (src/controller/testsuite/test_config_resource_reconcile.c); RED before the\n  fix (lic1 unregistered / flags stayed 0x00), GREEN after.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in\n  sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at\n  free=1; release re-admitted it.",
      "tags": [
        "config",
        "reconcile",
        "E2",
        "external-license"
      ],
      "related_bugs": [
        "bug-1854"
      ],
      "fix_commits": [],
      "record": ".wolf/bugs/bug-1973.md"
    },
    {
      "id": "bug-1974",
      "status": "fixed",
      "status_updated": "2026-10-27",
      "status_note": "",
      "component": "controller",
      "severity": "wrong-result",
      "timestamp": "2026-10-27T03:35:12Z",
      "last_seen": "2026-10-27T03:35:12Z",
      "occurrences": 1,
      "error_message": "config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM\n\n**Observed:**\nController log on t1 after re-pushing a corrected cluster.bin (extlic mapped\nwith `location: [external]` → flags SHARED|GLOBAL|EXTERNAL):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted\n```\n\n**Expected:** update_val_res' SHARED branch routes a host==\"\" resval for a\nSHARED-flagged resource into c->sharedinst; with correct flags the push lands.\n\n**Impact:** the extres daemon's pushes were permanently refused on a controller\nwhose registration predated the flag fix — the stale shape was frozen for the\nprocess lifetime, so re-pushing a corrected config did not heal it.\n\n**Found by:** E2 Phase 1 live E2E on t1.",
      "file": "src/controller/cluster.c, src/controller/worker.c, src/xceadmin/src/xcecommon/mapping.py",
      "root_cause": "Two compounding factors:\n1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/\n   xceadmin-venv) carries a stale mapping.py with NO resourcemap handling\n   (83-line diff vs tree) — cluster bins it builds carry flags=0 for every\n   resource. Ops issue, fixed by building with the tree venv\n   (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs\n   redeployment.\n2. **Controller**: even with a corrected bin, `cluster_build_resources()`\n   treated an already-registered name as \"nothing to do\" (bug-1854's\n   RES_APPLY_CHANGED was reported but never applied), so the stale flags=0\n   shape survived every activation.",
      "fix": "`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the\nregistered object's definition fields (valtype/ordtype/flags/interval) in\nplace under the resmgr write lock — none of them touches the registration\nslot (idx) or accumulated accounting; mirrors reconcile_config_queues'\n\"updated by configuration activation\" in-place refresh precedent. (Same loop\nas bug-1973's fix.) Producer side tracked separately as ops follow-up:\nredeploy the xceadmin venv.\n\nVerification:\n- Test: `test_reconcile_resources_hot_applies_changed_definition`\n  (src/controller/testsuite/test_config_resource_reconcile.c); RED before\n  (flags stayed 0x00), GREEN after (0x1C, idx preserved).\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push\n  landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN.",
      "tags": [
        "config",
        "flags",
        "E2",
        "external-license"
      ],
      "related_bugs": [
        "bug-1854",
        "bug-1973"
      ],
      "fix_commits": [],
      "record": ".wolf/bugs/bug-1974.md"
    },
    {
      "id": "bug-1975",
      "status": "fixed",
      "status_updated": "2026-10-27",
      "status_note": "",
      "component": "controller",
      "severity": "wrong-result",
      "timestamp": "2026-10-27T03:40:05Z",
      "last_seen": "2026-10-27T03:40:05Z",
      "occurrences": 1,
      "error_message": "xceesd never reconnects after controller restart when its value is unchanged (debounce suppresses pushes)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).",
      "file": "src/esd/src/esd.c, src/lib/client.c, src/lib/client_internal.h",
      "root_cause": "Three mechanisms conspired:\n1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip\n   push) — with a stable value it pushes NOTHING.\n2. A reconnect inside the client library only fires while INSIDE a push's\n   failover call — no pushes, no reconnect.\n3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;\n   with no push attempt it never runs.",
      "fix": "`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`\n(bug-1760's pure predicate — the transport's RDHUP handler sets\nXCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.\nThe write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the\nfull re-push contract for the failover path.\n\nVerification:\n- Test: covered by the live contract (client-side predicate already unit\n  tested by bug-1760's suite); esd-side loop is integration-shaped (the\n  trigger is a transport callback), so live E2E is the proof.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — controller killed and restarted; esd logged\n  `connection broken -- forcing a full re-push to reconnect`, rotated to the\n  new primary, re-pushed; `xcecli resources distribution list` shows extlic=4\n  and a fresh rusage[extlic=3] job RUNS.",
      "tags": [
        "esd",
        "external-license",
        "reconnect",
        "E2"
      ],
      "related_bugs": [
        "bug-1760",
        "bug-1973",
        "bug-1974"
      ],
      "fix_commits": [],
      "record": ".wolf/bugs/bug-1975.md"
    }
  ]
}