diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 04237639..ed75cedc 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -15785,14 +15785,14 @@ }, { "id": "bug-1429", - "status_note": "teardown-order fix (2026-08-30) covered 21 sites; test_config_repl.c (~23 sites) + test_term_safety_adapter.c (5) still unverified, and a 300s TIMEOUT recurred 2026-09-24 near test_commit_barrier_rollout.c's jap fixture", - "status": "partial", - "status_updated": "2026-09-24", + "status_note": "574a5f84", + "status": "fixed", + "status_updated": "2026-09-25", "timestamp": "2026-07-10", "error_message": "controller_unit_test suite TIMEOUT (300s, killed by SIGTERM) during /xce-build controller run immediately after uncommitted edits to persist.c, ha_convergence.c, config_repl.c (delaymgr-inline blocking I/O moved to VTASK_LIVE_BLOCK vtasks with inline-fallback-on-failure). Compile succeeded with zero errors (only pre-existing warnings). meson test summary: Ok:0 Fail:0 Timeout:1. Last test output before the hang was test_demotion_replay_yields_committed_only (test_commit_barrier_reply.c) printing overlayfs-SKIP, followed by a burst of HA FSM / persist_apply_pump_run log lines ending in \"HA quorum lost: connected=0 required=1\", then total silence for the remaining ~294s until SIGTERM. No ASAN/crash report -- looks like a genuine hang/deadlock, not a crash.", "file": "src/controller/testsuite/test_commit_barrier_rollout.c, test_demotion_convergence.c, test_convergence_verification.c, test_egress_filter.c, test_apply_pump.c, test_apply_pump_integration.c, test_job_array_fence.c, test_ha_role_lifecycle.c, test_job_array_replay_fence.c, test_commit_durability.c, test_commit_engine.c, test_p2_integration.c (test-fixture teardown ordering, 21 sites across 12 files); src/controller/commit.c (commit_engine_fini, unchanged -- confirms the documented contract); src/controller/persist.c (fsync_worker, unchanged -- confirms the race window)", "root_cause": "ROOT-CAUSED 2026-08-30, LIVE-REPRODUCED with a full gdb thread backtrace (the first time in this bug's history any session captured a live hang instead of just reporting the SIGTERM/timeout). The original delaymgr/VTASK_LIVE_BLOCK hypothesis carried by this bug since 2026-07-10 was WRONG -- a static read of every vtask_enq_classified(..., VTASK_LIVE_BLOCK) call site in ha.c/persist.c/config_repl.c/commit.c found every one of them NULL-checks its pool pointer (or the trampoline is gated behind `c->delaymgr.running`, which test fixtures never set) and falls back inline on failure -- there is no path where a controller_unit_test fixture leaves a non-NULL-but-never-started vtaskmgr_t that a VTASK_LIVE_BLOCK task could be silently swallowed into. That entire subsystem is a red herring for this bug.\n\nThe REAL root cause is the same TOCTOU test-teardown race already root-caused once before, for a DIFFERENT symptom, in bug-1452 (2026-07-19: an ASAN SEGV in test_persist_truncate.c) and partially re-applied once more in bug-1687 (2026-08-25: cbrp_teardown in test_commit_barrier_reply.c). Every affected test fixture calls `commit_engine_fini(&ce)` BEFORE `persist_stop(&persist)` in its teardown helper -- the exact opposite of the contract documented in commit_engine_fini's own comment (commit.c: 'Callers are expected to have stopped the flusher (persist_stop) first'). Because persist_stop had not yet run, the persist_t's live fsync_worker background thread (persist.c:800, started by persist_start) can still be mid-flight calling `p->on_durable_advance(p->durable_cb_ctx)` -- i.e. commit_on_durable_advance(ce) -- with NO lock held (persist.c:866-868), concurrently with commit_engine_fini destroying ce->lock/ce->cond (commit.c:476-477) and freeing ce->continuations. commit_engine_fini's own persist_set_durable_cb(ce->p, NULL, NULL) unhook is 'defensive and cheap' per its own comment but is NOT sufficient on its own: the flusher thread can have already read the OLD non-NULL callback pointer a few instructions before the unhook takes effect. Depending on exact thread interleaving this TOCTOU race manifests as either bug-1452's SEGV (a NULL/freed dereference inside commit_advance_locked) OR -- newly confirmed this session -- a hang: pthread_cond_destroy(&ce->cond) blocking forever because the racing fsync_worker thread is concurrently inside pthread_cond_broadcast(&ce->cond) / holding ce->lock from commit_on_durable_advance, which glibc's cond-destroy implementation cannot safely proceed past. This explains every symptom previously logged against this bug: the varying hang location across sessions (different tests hit their OWN teardown's copy of this same race, at whatever moment the fsync_worker thread happened to be flushing), the total silence after the hang (both racing threads are genuinely blocked in futex/pthread_cond primitives, not spinning), zero CPU time accruing, and \"both threads parked in futex_do_wait\" from the 2026-08-23 sighting.\n\nLIVE REPRO (this session): `meson test -C src/xce.build.bug1429 --suite controller --verbose` hung 40s into the run with 3 threads (main + 2x fsync_worker) all in futex_do_wait, 0 CPU time advancing over a 3s recheck. gdb -p -batch -ex 'thread apply all bt full' (attached BEFORE the 300s SIGTERM, while the process was still alive) showed: Thread 1 (main) stuck in pthread_cond_destroy() called from commit_engine_fini (commit.c:476) <- jap_teardown() (test_commit_barrier_rollout.c:1877) <- test_job_array_submit_payload_size_bounded_at_max_array_idx (test_commit_barrier_rollout.c:2294) <- UnityDefaultTestRun <- test_commit_barrier_rollout() <- main(). Threads 2 and 3 both stuck in pthread_cond_timedwait at fsync_worker (persist.c:827), BOTH with `p = &g_jap` (the same global persist_t) -- confirming jap_teardown's exact commit_engine_fini-before-persist_stop ordering bug (test_commit_barrier_rollout.c:1877-1879, at the time of capture) as the live mechanism. This is the 4th distinct hang location logged against this bug (previously: test_demotion_replay_yields_committed_only, the test_configack_* group, test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs) -- all four are different call sites of the identical teardown-ordering bug, which is why the location never repeated and why every prior static-inference pass aimed at the vtaskmgr/delaymgr subsystem instead never found it.", - "fix": "FIXED 2026-08-30. Test-fixture-only fix -- no production code changed (commit.c and persist.c are exactly as documented; the bug is every affected fixture violating commit_engine_fini's own documented calling contract). Inserted an explicit `persist_stop(&persist);` call immediately before every confirmed-vulnerable `commit_engine_fini(&ce);` call (mirroring bug-1452/bug-1687's established fix pattern exactly), leaving each function's pre-existing later persist_stop()/teardown-helper call in place as a safe, idempotent no-op (persist_stop's own fsync_thread_running and wal_fd>=0 guards make the second call a clean skip). Where a fixture also tears down a real multi-vcpu vtaskmgr_t pool (test_apply_pump.c's tpr_teardown, test_apply_pump_integration.c's tpi_teardown), vtaskmgr_fini() was left in its original position AFTER commit_engine_fini() (a separate, unrelated invariant: no new vtask may be dispatched onto the pool mid-teardown) -- only persist_stop()'s position moved earlier.\n\n21 sites fixed across 12 files, all confirmed vulnerable by verifying (a) the fixture's persist_t is genuinely persist_start()'d (has a live fsync_worker thread) and (b) the teardown called commit_engine_fini() before persist_stop(): test_commit_barrier_rollout.c (3 sites -- g_fce/g_fp inline teardown, cb5_teardown, jap_teardown -- the last being the exact live-reproduced hang site), test_demotion_convergence.c (dc_teardown -- this EXACT site was flagged but explicitly left unfixed by bug-1452's own remediation note as a known follow-up, never actioned until now), test_convergence_verification.c (cv_teardown), test_egress_filter.c (ef_teardown), test_apply_pump.c (2 sites -- tap_teardown, tpr_teardown), test_apply_pump_integration.c (tpi_teardown), test_job_array_fence.c (tf_teardown), test_ha_role_lifecycle.c (lifecycle_cluster_teardown), test_job_array_replay_fence.c (1 site of its 4 persist_start'd fixtures -- the other 3 never wire a commit engine), test_commit_durability.c (3 sites, including one with a comment that incorrectly claimed persist_set_durable_cb's unhook alone was sufficient), test_commit_engine.c (3 sites, all sharing sync_persist_teardown), test_p2_integration.c (3 sites, 2 sharing var `p` + 1 using `g_p9`). Confirmed SAFE and left unchanged: barrier_teardown (test_commit_barrier_rollout.c), engine_teardown (test_commit_engine.c), and the fixtures in test_repl_majority.c/test_commit_admission_pool.c/test_coldstart_lsn0.c/test_commit_resolve.c -- all use a 'stub' persist_t that is never actually persist_start()'d (no live fsync_worker thread), confirmed by grep for persist_start against each fixture's global before touching it.\n\nKNOWN REMAINING EXPOSURE (not fixed this session, explicit follow-up for a future pass, same as bug-1452 left one for this session): test_config_repl.c has ~23 call sites following the identical inline `commit_engine_fini(&c->commit); t42_teardown(c);` pattern (t42_teardown, line 629, confirmed calls persist_stop THEN persist_fini -- so every one of these is the same bug) -- not fixed here because the file is large (~7000 lines) with some sites (e.g. the c_control/c_raced pair around line 2911-2912) not yet individually verified for uniformity, and a bulk edit without that verification was judged too risky for this session's budget. test_term_safety_adapter.c has persist_start'd fixtures (lines 59, 245, 2382) and 5 commit_engine_fini call sites (676, 930, 1776, 1950, 2338) not individually checked. Neither file's exposure has been observed to hang or crash yet (this bug's own history shows the race is intermittent and timing-dependent -- absence of a prior sighting there is not evidence of safety). Recommended next step: a dedicated /xce-audit-invariants or grep-driven sweep of these two files using the exact same verification method used here (confirm persist_start presence, confirm fini-before-stop order, insert persist_stop() immediately before commit_engine_fini()).\n\nVERIFICATION: full `meson test -C xce.build.bug1429 --suite controller` run 3 times back-to-back after the settled fix (no edits during any of these three runs) -- 39.01s / 38.86s / 38.91s, Ok:6 Fail:0 every time, zero hangs, zero timeouts (previously: TIMEOUT 300.08s-300.30s on effectively every attempt this session before the fix, reproduced live on the very first try). No regression test added (per the task's own documented exception for a suite-level timing-dependent teardown race): the race requires a live fsync_worker thread mid-flush at the exact moment of teardown, which is not reliably triggerable in a bounded, fast unit test without reintroducing the same flakiness this fix removes; the 3x full-suite re-run is the verification method this project's own bug-1452 precedent already established for this exact bug class.", + "fix": "FIXED 2026-08-30. Test-fixture-only fix -- no production code changed (commit.c and persist.c are exactly as documented; the bug is every affected fixture violating commit_engine_fini's own documented calling contract). Inserted an explicit `persist_stop(&persist);` call immediately before every confirmed-vulnerable `commit_engine_fini(&ce);` call (mirroring bug-1452/bug-1687's established fix pattern exactly), leaving each function's pre-existing later persist_stop()/teardown-helper call in place as a safe, idempotent no-op (persist_stop's own fsync_thread_running and wal_fd>=0 guards make the second call a clean skip). Where a fixture also tears down a real multi-vcpu vtaskmgr_t pool (test_apply_pump.c's tpr_teardown, test_apply_pump_integration.c's tpi_teardown), vtaskmgr_fini() was left in its original position AFTER commit_engine_fini() (a separate, unrelated invariant: no new vtask may be dispatched onto the pool mid-teardown) -- only persist_stop()'s position moved earlier.\n\n21 sites fixed across 12 files, all confirmed vulnerable by verifying (a) the fixture's persist_t is genuinely persist_start()'d (has a live fsync_worker thread) and (b) the teardown called commit_engine_fini() before persist_stop(): test_commit_barrier_rollout.c (3 sites -- g_fce/g_fp inline teardown, cb5_teardown, jap_teardown -- the last being the exact live-reproduced hang site), test_demotion_convergence.c (dc_teardown -- this EXACT site was flagged but explicitly left unfixed by bug-1452's own remediation note as a known follow-up, never actioned until now), test_convergence_verification.c (cv_teardown), test_egress_filter.c (ef_teardown), test_apply_pump.c (2 sites -- tap_teardown, tpr_teardown), test_apply_pump_integration.c (tpi_teardown), test_job_array_fence.c (tf_teardown), test_ha_role_lifecycle.c (lifecycle_cluster_teardown), test_job_array_replay_fence.c (1 site of its 4 persist_start'd fixtures -- the other 3 never wire a commit engine), test_commit_durability.c (3 sites, including one with a comment that incorrectly claimed persist_set_durable_cb's unhook alone was sufficient), test_commit_engine.c (3 sites, all sharing sync_persist_teardown), test_p2_integration.c (3 sites, 2 sharing var `p` + 1 using `g_p9`). Confirmed SAFE and left unchanged: barrier_teardown (test_commit_barrier_rollout.c), engine_teardown (test_commit_engine.c), and the fixtures in test_repl_majority.c/test_commit_admission_pool.c/test_coldstart_lsn0.c/test_commit_resolve.c -- all use a 'stub' persist_t that is never actually persist_start()'d (no live fsync_worker thread), confirmed by grep for persist_start against each fixture's global before touching it.\n\nKNOWN REMAINING EXPOSURE (not fixed this session, explicit follow-up for a future pass, same as bug-1452 left one for this session): test_config_repl.c has ~23 call sites following the identical inline `commit_engine_fini(&c->commit); t42_teardown(c);` pattern (t42_teardown, line 629, confirmed calls persist_stop THEN persist_fini -- so every one of these is the same bug) -- not fixed here because the file is large (~7000 lines) with some sites (e.g. the c_control/c_raced pair around line 2911-2912) not yet individually verified for uniformity, and a bulk edit without that verification was judged too risky for this session's budget. test_term_safety_adapter.c has persist_start'd fixtures (lines 59, 245, 2382) and 5 commit_engine_fini call sites (676, 930, 1776, 1950, 2338) not individually checked. Neither file's exposure has been observed to hang or crash yet (this bug's own history shows the race is intermittent and timing-dependent -- absence of a prior sighting there is not evidence of safety). Recommended next step: a dedicated /xce-audit-invariants or grep-driven sweep of these two files using the exact same verification method used here (confirm persist_start presence, confirm fini-before-stop order, insert persist_stop() immediately before commit_engine_fini()).\n\nVERIFICATION: full `meson test -C xce.build.bug1429 --suite controller` run 3 times back-to-back after the settled fix (no edits during any of these three runs) -- 39.01s / 38.86s / 38.91s, Ok:6 Fail:0 every time, zero hangs, zero timeouts (previously: TIMEOUT 300.08s-300.30s on effectively every attempt this session before the fix, reproduced live on the very first try). No regression test added (per the task's own documented exception for a suite-level timing-dependent teardown race): the race requires a live fsync_worker thread mid-flush at the exact moment of teardown, which is not reliably triggerable in a bounded, fast unit test without reintroducing the same flakiness this fix removes; the 3x full-suite re-run is the verification method this project's own bug-1452 precedent already established for this exact bug class.\n\nUPDATE (2026-09-24): 2026-09-24 ROOT CAUSE of the residual 300s TIMEOUT (live-captured with gdb under a 6-way parallel stress harness, unshare -rn per instance): any failing TEST_ASSERT longjmps out and SKIPS the fixture teardown; the fixture's commit engine sweep thread (and persist fsync_worker) stay asleep on the static engine's ce->cond / p->fsync_cond; the next test re-inits the same static struct underneath them (pthread_cond_init on a condvar with a live waiter), corrupting the waiter count; a later teardown then blocks forever in pthread_cond_destroy (commit_engine_fini / persist_fini) or pthread_join (cb5_teardown) -- or, for a stack-local engine, the sweep thread reads a dead frame (ASAN stack-use-after-return in commit_sweep_worker, test_persist_truncate.c). The failing assertion itself was never seen because Unity's stdout was fully buffered and lost on SIGTERM/abort. Triggering flaky assertions found once visible: test_job_array_submit_one_entry_one_continuation_across_sizes (the test in flight when the 11:34 meson run hung; continuation resolved on the fsync_worker thread, test read a plain int immediately), test_pump_mixed_facts_and_commands_catchup_original_order (reply count read once before trailing continuations), test_persist_flush_fires_durable_advance_callback (fsync_worker credited a concurrent persist_flush advance to itself -> double fire). FIX (uncommitted): (1) controller_test.c line-buffers stdout; (2) XCE_TESTING registries: tearDown() calls persist_test_reap_unstopped() then commit_test_reap_unfinalized_engines(), stopping/joining whatever a failed test leaked (clean full run reaps 0; injected failure now = 1 FAIL in 2s instead of a hang); (3) persist.c flusher_flush_once reports only its own advance (fsync_worker no longer double-fires; regression test test_flusher_pass_does_not_claim_concurrent_flush_advance); (4) the two flaky tests wait (bounded) for cross-thread resolution.", "tags": [ "FIXED", "VTASK_LIVE_BLOCK", @@ -27868,14 +27868,14 @@ }, { "id": "bug-1859", - "status": "investigating", - "status_updated": "2026-09-24", - "status_note": "chaotic soak: worker-resync job stuck PEND (ctrl) vs RUN (sched) after overlapping failover; 1 of 2 chaotic runs", + "status": "fixed", + "status_updated": "2026-09-25", + "status_note": "444fd504", "timestamp": "2026-09-24T13:45:31Z", "error_message": "xce-test-ha --chaotic FAIL: chaotic: 1/4 worker-resync job(s) never reconciled after the loop settled: 0-2-1790256202-546579(last=PEND). scheduler.log: 'JOBS_PUSH: refusing resync state 0x1 for jobid ... -- existing job is RUN with a real allocation on record (existing state 0x2 kept)'", "file": "src/scheduler/src/core_events.c,src/controller/job.c", "root_cause": "Not yet confirmed. Timeline (2026-09-24 chaotic soak, 6 rounds): job dispatched on primary ctrl-1 (term 6) at 13:23:23.74, commit_advance 80->81 at 13:23:23.79; worker started it and finished it at 13:23:43 (retained outcome exit_status=1); ctrl-1 killed in round 5, overlapping kills follow; after settle the cluster reports the job PEND while the scheduler keeps RUN+alloc and refuses the PEND resync (core_events.c:984), so neither side converges. Also: round 6 'scheduler restart did not confirm join within 20s (bug-1760 workaround)'. Candidates: dispatch/RUN not in the surviving committed prefix while the scheduler already applied it, or worker's retained finish never re-delivered to the new primary (cf. bug-1741 fire-and-forget finish notify). Same chaotic soak passed earlier the same day (21 jobs, 6 resync jobs reconciled), so intermittent.", - "fix": "", + "fix": "UPDATE (2026-09-24): Frequency 2026-09-24: 1 failure in 3 chaotic 6-round soaks (pass 12:5x, FAIL 13:2x, pass 13:5x on 393a07c2). Failing run's logs kept at scratchpad ha-chaos-bug1859-run1 (session-local).\n\nUPDATE (2026-09-24): ROOT CAUSE (from preserved logs of the failing run): job 546579 dispatched+committed (LSN 81) on ctrl-1 term 6; worker finished it 13:23:43 and sent the fire-and-forget finish notify (xcecli_notify = single-shot send, rc ignored: 'FIXME: add error handling when msg send failure') into a dead connection during overlapping failovers; worker could not reach any primary for 84s; its retained finished outcome expired at finish_time + 60s (default window; xce.yaml sets no worker_resync_window_ms, controller pushes 0 -> default) DURING the outage; at rejoin (13:25:07) ctrl-2's resync query (op 43) got 'unrecognized' -> record_job_reconcile_pending_fact requeued it PEND while the scheduler was disconnected; scheduler rejoined 13:25:11 and its JOBS_PUSH bug-1812 guard refused the PEND (RUN+alloc on record) -> permanent divergence. FIX (uncommitted): worker retention window no longer counts outage time -- xcecd_on_reconnect() records the reconnect time and expiry is measured from max(finish_time, last_reconnect); nothing expires while checking_connection()'s hello probe reports unhealthy. Reported finish_time_ms unchanged. Tests: test_finished_job_retention.c test_reconnect_restarts_window_for_outcome_finished_before_outage (84s outage) + test_unhealthy_connection_suspends_expiry, both red first; worker suite 10/10. The scheduler-side divergence (controller requeue while scheduler disconnected is refused by bug-1812 guard) logged separately.", "tags": [ "ha", "chaotic-soak", @@ -27887,6 +27887,70 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-09-24T13:45:31Z" + }, + { + "id": "bug-1860", + "status": "open", + "status_updated": "2026-09-24", + "status_note": "bug-1812 guard blocks authoritative PEND after disconnected requeue; needs per-job freshness in JOBS_PUSH", + "timestamp": "2026-09-24T14:38:03Z", + "error_message": "Scheduler keeps a job RUN (with allocation) forever after the controller requeued it to PEND while the scheduler was disconnected: scheduler.log 'JOBS_PUSH: refusing resync state 0x1 for jobid ... -- existing job is RUN with a real allocation on record' on rejoin; job never redispatched (seen in bug-1859's chaotic soak)", + "file": "src/scheduler/src/core_events.c,src/controller/sched.c,src/controller/resync.c", + "root_cause": "core_events.c JOBS_PUSH bug-1812 guard refuses any non-terminal resync state that would drop a real allocation, to protect against a committed snapshot racing the scheduler's own fresh dispatch. It cannot tell that race apart from a genuine, committed controller-side requeue (e.g. resync 'unrecognized' -> record_job_reconcile_pending_fact) that happened while the scheduler was disconnected, and JOBS_PUSH carries no per-job freshness (commit LSN/version) to decide. Needs a design decision (e.g. per-job last-modified LSN in the push).", + "fix": "", + "tags": [ + "scheduler", + "controller", + "resync", + "ha", + "divergence", + "design" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-09-24T14:38:03Z" + }, + { + "id": "bug-1861", + "status": "fixed", + "status_updated": "2026-09-25", + "status_note": "574a5f84", + "timestamp": "2026-09-24T15:05:56Z", + "error_message": "config_repl.c do_upgrade_config_with_bytes: temp decode file /tmp/xce_upgrade_cfg..tmp -- the name is identical across processes (parallel controller_unit_test instances failed handle_upgrade_config with -EINVAL, 'a majority-acked upgrade must succeed') and is opened O_CREAT|O_TRUNC without O_EXCL in world-writable /tmp", + "file": "src/controller/config_repl.c", + "root_cause": "pthread_self() is a thread-descriptor address, not a unique id: the main thread's value repeats across processes (esp. with ASLR off under ASAN), so two processes on one host share and clobber the file; a predictable name opened without O_EXCL in /tmp is also a symlink-attack target (write/truncate through an attacker's link).", + "fix": "", + "tags": [ + "controller", + "security", + "tmpfile", + "config-upgrade" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-09-24T15:05:56Z" + }, + { + "id": "bug-1862", + "status": "open", + "status_updated": "2026-09-24", + "status_note": "resync query (vtask thread) walks worker list while main loop frees entries; no lock", + "timestamp": "2026-09-24T15:24:57Z", + "error_message": "xcecd_virtual ASAN heap-use-after-free in find_worker_by_jobid (xcecd_worker_utils.c:702) <- xcecd_resync_classify_job (xcecd_resync.c:33) <- xcecd_resync_build_report <- handle_jobstatequery (controller_connector.c:1662) on a vtask thread; freed by remove_worker (xcecd_worker_utils.c:283) <- cleanup_worker_list <- main loop (main.c:652). Worker aborted mid chaotic HA soak (2026-09-24, run 1 of 3), so later rounds lost worker-resync coverage.", + "file": "src/worker/src/xcecd_resync.c,src/worker/src/xcecd_worker_utils.c,src/worker/src/controller_connector.c,src/worker/src/main.c", + "root_cause": "The worker's live job list is mutated by the main loop (add_worker / cleanup_worker_list -> remove_worker) with no lock, while the controller's OPCode_jobStateQuery is answered on a vtask thread (myreq_handler) that walks the same list. Any rejoin resync overlapping a job reap reads a freed node.", + "fix": "", + "tags": [ + "worker", + "asan", + "race", + "use-after-free", + "resync", + "ha" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-09-24T15:24:57Z" } ] }