docs(raft-ha-core-safety): 5.7 verified (election resonance fixed); round-4 re-roots to bug-1298 UAF (task 5.8)
5.7 election-timeout resonance FIXED + soak-confirmed: 6-round chaotic soak now elects with bounded terms (rounds 1-3 at terms 4/5/6, not 25-30+), zero job loss. The election-liveness layer is done. Round-4 stall re-roots to a NON-election crash: ctrl-3 ASAN heap-use-after-free in endpoint teardown — io_worker (endpoint_killer, cluster.c:813) frees an endpoint while the vtask dispatcher reads ep->busy (endpoint_is_task_blocked, netio.c:1008). ctrl-3 dies -> cluster drops to 1/3 -> no quorum -> no primary. Same async-io dispatch surface as bug-1282/1197; UPSTREAM-owned (raft-drive-async / libxce). Logged bug-1298; tracked as task 5.8.
This commit is contained in:
parent
74784b2b18
commit
5be165312a
3 changed files with 249 additions and 4 deletions
|
|
@ -647,3 +647,75 @@ two survivors desynchronize and one's term stops racing the other's reply. Prese
|
|||
— the freshest survivor is granted but its term out-races the grant. New follow-up.
|
||||
Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every
|
||||
completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn.
|
||||
|
||||
---
|
||||
|
||||
## Task 5.7 fix VERIFIED + round-4 blocker re-rooted (2026-06-30)
|
||||
|
||||
**5.7 (election-timeout resonance) — FIXED, commit 74784b2.**
|
||||
|
||||
The 5.6 note guessed the 5.7 cause as "term re-escalates faster than the grant
|
||||
can land." Reading the code pinned the *mechanism* precisely: the per-CAMPAIGN
|
||||
election watchdog (`raft_start_election` → `on_schedule_timeout` →
|
||||
`raft_cb_schedule_timeout`, ha.c) was armed with a **FIXED** `election_timeout_ms`
|
||||
— **no randomization**. Only the *startup-discovery* path (`ha_wait_for_heartbeat_or_timeout`)
|
||||
jittered; the recurring campaign watchdog did not. raft.c:157 passes the bare
|
||||
`s->election_timeout_ms` straight through.
|
||||
|
||||
A fixed deadline + a roughly-constant cold-connection grant RTT **RESONATE**: if
|
||||
ctrl-3's grant consistently lands a hair *after* a constant deadline, every
|
||||
campaign loses by the *same* margin forever → the campaign times out, the term
|
||||
re-escalates, and the now-stale grant is dropped by `DropStaleResponse`
|
||||
(raft.c:453, which is *correct* Raft). Terms climb 25→30+. The bug was never
|
||||
"RTT genuinely > 5 s"; it was that the deterministic cadence locks the grant
|
||||
permanently just outside every window.
|
||||
|
||||
**Fix (Ongaro thesis §3.4 — randomized election timeout, re-drawn each campaign):**
|
||||
new `ha_jittered_election_delay(ha, base)` returns `base + rand[0, base]` (the
|
||||
canonical [T, 2T] interval), wired into `raft_cb_schedule_timeout` so the watchdog
|
||||
is re-randomized per campaign. Done in the HA adapter so the pure, sim-tested raft
|
||||
library stays deterministic. Independent of `election_jitter_max_ms` (which tunes
|
||||
only the sub-second startup/stagger jitter) so it spreads even at jmax=200; never
|
||||
below the floor so genuine-loss failover is not sped up past `election_timeout_ms`.
|
||||
|
||||
Unit: `test_5_7_campaign_timeout_is_randomized` (CORE) + `test_5_7_floor_preserved_and_zero_safe`
|
||||
(PIN). RED verified by stubbing the jitter to 0 → the randomization test fails with
|
||||
the resonance message while the PIN stays green. `test_ha_core_safety` 24 Tests / 0
|
||||
Failures. (Aside: the full aggregate `controller_unit_test` binary cannot reach the
|
||||
ha_core_safety suite because an EARLIER suite, `test_ha_vep_teardown`, SIGSEGVs in
|
||||
`iomgr_fini`→`SSL_CTX_free` under openssl 3.6 — unrelated to this change; run the
|
||||
suite in isolation with `-t test_ha_core_safety`.)
|
||||
|
||||
**Soak confirmation:** a fresh ASAN 6-round chaotic soak now elects with **BOUNDED
|
||||
terms** — rounds 1→3 elect ctrl-3/ctrl-2/ctrl-1 at terms **4→5→6** (not 25→30+).
|
||||
The unbounded escalation is gone; zero job loss on every completed round (3/6/9/12
|
||||
tracked jobs all present on each new primary). The election-liveness layer is DONE.
|
||||
|
||||
**Round-4 blocker RE-ROOTS to a non-election crash (task 5.8 / bug-1298).**
|
||||
Round 4 still stalls, but no longer for any election reason. ctrl-3 **ABORTs** on
|
||||
an ASAN heap-use-after-free:
|
||||
|
||||
```
|
||||
UAF READ size 4 endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3)
|
||||
vtask_eligible vtaskmgr.c:255 ← dataq_deq_first_match ← vdispatcher_entry
|
||||
FREED BY (T2) endpoint_killer cluster.c:813 (io_worker)
|
||||
handle_endpoint_error cluster.c:108 ← handle_error ← io_worker
|
||||
ALLOC endpoint_get netio.c:748 ← handle_connection cluster.c:741
|
||||
```
|
||||
|
||||
io_worker (T2) tears down a connection on error and `free()`s the endpoint via
|
||||
`endpoint_killer` while the vtask dispatcher (T3) concurrently reads `ep->busy` on
|
||||
that just-freed endpoint during its dataq eligibility scan. The endpoint is freed
|
||||
while still referenced by a pending vtask; nothing drains its queued vtasks before
|
||||
the free and no refcount keeps it alive across the dispatcher read. ctrl-3 dies →
|
||||
the cluster drops to 1/3 (the round-4 survivors are the just-restarted ctrl-2 and a
|
||||
now-dead ctrl-3) → no quorum → "NO new primary within window." ctrl-2's logs show
|
||||
only PreVote-round timeouts (`prospective term=6`, never wins) — the textbook
|
||||
symptom of a lost quorum peer, not an election defect.
|
||||
|
||||
Same netio surface and routing precedent as task 5.4/bug-1282: an async-io dispatch
|
||||
concurrency bug, **UPSTREAM-owned** by raft-drive-async / core libxce. Likely a
|
||||
latent UAF that the bug-1282 lockless `ep->busy` read unmasked (the dispatcher now
|
||||
reaches the read without contending on `ep->lock`). Logged as **bug-1298**; tracked
|
||||
as task 5.8. Consensus safety (Phase-2/5.2/5.3/5.5/5.6/5.7) is sound; elections and
|
||||
job safety hold for every round that completes.
|
||||
|
|
|
|||
|
|
@ -146,14 +146,23 @@
|
|||
- _Depends: 5.5_
|
||||
- _PARTIAL FIX (commit 81cb09a): the RDHUP-entry now drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress (don't preempt an in-flight election). Unit RED→GREEN (22 Tests/0 Failures); reduces the term churn. But the soak still stalls at round 4 → a deeper 5th layer (task 5.7): the freshest survivor is GRANTED but its term out-races the grant (vote_reply_stale_drop). Detail in `research.md` §"Task 5.6 fix VERIFIED"._
|
||||
|
||||
- [ ] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6)
|
||||
- _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. A just-restarted ctrl-2 has COLD peer connections (each real-vote opens a fresh endpoint_connect+SSL) and races connect attempts to the dead ctrl-1, so the grant RTT exceeds the window before ctrl-2 re-escalates. election_timeout_ms=5000, jitter=200ms._
|
||||
- Fix direction: (a) do not bump the election term while a vote round-trip for the current term is still outstanding (let the grant land before re-escalating); and/or (b) warm/reuse the peer vote connection so the grant RTT < election_timeout (avoid per-vote connection setup); and/or (c) widen election-timeout jitter so the two survivors desynchronize. Preserve Election Safety (DropStaleResponse is correct Raft — fix the escalation cadence, not the drop), genuine-loss liveness, 5.2 suppression, and the 5.6 no-preempt guard. RED: a just-restarted freshest survivor + a willing grantor with a cold connection; assert it wins within a bounded number of election terms (term does not escalate unbounded).
|
||||
- Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate
|
||||
- [x] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6)
|
||||
- _FIXED + VERIFIED (commit 74784b2, `research.md` §"Task 5.7 fix VERIFIED"). Actual root cause was narrower than the 5.6 note guessed: the per-CAMPAIGN election watchdog was armed with a FIXED `election_timeout_ms` (`raft_start_election`→`on_schedule_timeout`→`raft_cb_schedule_timeout`); only the startup-discovery path jittered. A fixed deadline + a roughly-constant cold-connection grant RTT RESONATE — the single needed grant lands a hair past every deadline forever, the campaign times out, the term re-escalates, and the now-stale grant is dropped (DropStaleResponse correctly). Fix (Ongaro §3.4): re-randomize the watchdog per campaign to the canonical [T, 2T] interval, injected in the HA adapter so the pure (sim-tested) raft library stays deterministic; independent of `election_jitter_max_ms`; never below the floor. New `ha_jittered_election_delay()` + 2 unit tests (RED verified — stubbing jitter to 0 fails `test_5_7_campaign_timeout_is_randomized` with the resonance message; PIN guards floor + modulo-by-zero). `test_ha_core_safety` 24 Tests / 0 Failures. SOAK-CONFIRMED: a fresh 6-round chaotic soak now elects with BOUNDED terms (rounds 1→3 at terms 4→5→6, NOT 25→30+); zero job loss across completed rounds. The election-liveness layer is DONE._
|
||||
- _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. election_timeout_ms=5000, jitter=200ms._
|
||||
- Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate ✅ (election layer met; 7.2 now blocked on a SEPARATE non-election crash, task 5.8)
|
||||
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
|
||||
- _Boundary: Election Liveness_
|
||||
- _Depends: 5.6_
|
||||
|
||||
- [ ] 5.8 Endpoint teardown-vs-dispatch heap-use-after-free crash (NEW — the round-4 7.2 blocker after 5.7, UPSTREAM-owned)
|
||||
- _Root cause (`research.md` §"Task 5.7 fix VERIFIED" + bug-1298): with 5.7's election fix in, the 6-round chaotic soak's round-4 stall RE-ROOTS away from elections entirely. ctrl-3 ABORTs on an ASAN heap-use-after-free: the vtask dispatcher (T3) reads `ep->busy` via `endpoint_is_task_blocked` (netio.c:1008, in `vtask_eligible`→`dataq_deq_first_match`) on an endpoint that the io_worker (T2) just `free()`d via `endpoint_killer` (cluster.c:813, from `handle_endpoint_error`→`zarray_del`). The endpoint is freed while still referenced by a pending vtask in the dataq — no drain of its queued vtasks before the free, no refcount across the dispatcher read. ctrl-3 dies → cluster drops to 1/3 → no quorum → "no new primary within window." NOT an election bug; elections + job safety hold for every round that completes (rounds 1-3: zero job loss, bounded terms)._
|
||||
- **Ownership = UPSTREAM raft-drive-async / core libxce async-io dispatch** (vtaskmgr/dataq/netio/cluster teardown) — same surface and same routing precedent as task 5.4/bug-1282 (this spec's Non-Goals exclude the async-io layer). Same family as bug-1282 (ABBA, ep->busy made lockless) and the bug-1197 dispatch_inflight precedent; the bug-1282 lockless busy-read likely unmasked this latent UAF by letting the dispatcher reach the read sooner._
|
||||
- Fix direction (upstream): drain/cancel an endpoint's queued vtasks from the dataq BEFORE `endpoint_killer` frees it, OR hold a refcount/epoch on `endpoint_t` across the dispatcher eligibility read so the free defers until no dispatcher references remain (mirror bug-1197 `dispatch_inflight`). Must keep the bug-1282 lockless busy read and not reintroduce the ABBA ep->lock/q->lock inversion.
|
||||
- Observable completion: the 6-round chaotic soak runs all rounds with NO controller crash — the green 7.2 gate
|
||||
- _Requirements: 6.1, 6.2, 6.3_
|
||||
- _Boundary: Async-IO dispatch (UPSTREAM)_
|
||||
- _Depends: 5.7_
|
||||
|
||||
- [ ] 6. Integration: single-seam invariant end-to-end
|
||||
- [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
|
||||
- Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
|
||||
|
|
|
|||
|
|
@ -21394,6 +21394,170 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T09:15:12.861Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1291",
|
||||
"timestamp": "2026-06-30T09:32:54.664Z",
|
||||
"error_message": "Missing error handling in unknown",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
|
||||
"fix": "Added try/catch block",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"error-handling",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T09:32:54.664Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1292",
|
||||
"timestamp": "2026-06-30T09:33:08.918Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Had `1+2*njobs`",
|
||||
"fix": "Changed to `c = ep->owner`",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T09:33:08.918Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1293",
|
||||
"timestamp": "2026-06-30T11:15:05.373Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "5 lines replaced/restructured",
|
||||
"fix": "Rewrote 11→19 lines (5 removed) | Also: /* ha-term-safety task 1.2: advertise our Raft (la; * — identical sourcing to the real-vote send path ",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-30T11:15:18.266Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1294",
|
||||
"timestamp": "2026-06-30T11:42:35.428Z",
|
||||
"error_message": "Missing error handling in unknown",
|
||||
"file": "src/controller/testsuite/test_ha_core_safety.c",
|
||||
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
|
||||
"fix": "Added try/catch block",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"error-handling",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T11:42:35.428Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1295",
|
||||
"timestamp": "2026-06-30T13:19:42.706Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "No early return/throw for edge case: jmax > 0",
|
||||
"fix": "Added guard clause: if (jmax > 0)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:19:42.706Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1296",
|
||||
"timestamp": "2026-06-30T13:21:00.000Z",
|
||||
"error_message": "10-round chaotic soak FAIL at round 7: all 3 nodes in RDHUP→PreVote→timeout→repeat livelock — zero pre_vote_reply_in events (not denials, silence). ctrl-3 sent 20+ PreVote rounds with NO reply received; ctrl-1 and ctrl-2 same loop. Rounds 1-6 PASS clean (all elections + job-verification succeeded); round 7 kills ctrl-2 (term=9 primary), restarts ctrl-3 (no convergence wait — chaotic). 2 surviving nodes (ctrl-1, ctrl-3) both start PreVote; neither processes the other's request → mutual timeout → RDHUP fires again → repeat forever.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "TASK 5.6 dueling-candidate PreVote cross-denial — DISTINCT from the earlier PreVote-WINS + real-election split-vote (which had both nodes winning PreVote then deadlocking in RequestVote). Here PreVote itself starves: each node's PreVote handler likely drops/ignores inbound PreVote requests while in its own PreVote round (no reply sent, not even denial). The task 5.6 RDHUP guard (commit 25e741d+) only prevents restarting failover while in CTRL_ELECTING — it does NOT fix the core problem where two nodes enter simultaneous PreVote rounds and each starves the other's reply. The priority stagger (bug-606) helps when 3 nodes are alive (one grants to break symmetry), but with only 2 survivors (the third restarting without convergence in chaotic mode), both deny each other and the stagger is insufficient.",
|
||||
"fix": "NOT YET FIXED. Two approaches: (A) In the PreVote request handler, if we are in our own PreVote round and the incoming candidate has a FRESHER log (higher lastLogIndex/lastLogTerm), grant the PreVote and abort our own round (let the more-complete node lead). (B) Add randomized election-timeout jitter on the failover re-election path (currently only applied at cold-start via ha_wait_for_heartbeat_or_timeout) so the two survivors de-synchronize and one grants before starting its own round. Approach (A) is preferred for determinism.",
|
||||
"tags": [
|
||||
"raft-ha-core-safety",
|
||||
"election-liveness",
|
||||
"prevote-cross-denial",
|
||||
"dueling-candidates",
|
||||
"task-5.6",
|
||||
"chaotic-soak",
|
||||
"round-7-stall"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1281",
|
||||
"bug-1282",
|
||||
"bug-1286",
|
||||
"bug-1292"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:21:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1297",
|
||||
"timestamp": "2026-06-30T13:28:01.170Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "No early return/throw for edge case: jmax > 0",
|
||||
"fix": "Added guard clause: if (jmax > 0)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:28:01.170Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1298",
|
||||
"timestamp": "2026-06-30T13:36:00-07:00",
|
||||
"error_message": "AddressSanitizer: heap-use-after-free READ size 4 at endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3) — endpoint freed by endpoint_killer cluster.c:813 (io_worker T2); ctrl-3 ABORTs mid-chaotic-soak round 3→4, dropping the cluster to 1/3 (no quorum) so round 4 'NO new primary within window'",
|
||||
"file": "src/lib/netio.c / src/lib/vtaskmgr.c / src/controller/cluster.c",
|
||||
"root_cause": "Teardown-vs-dispatch race: io_worker (T2) tears down a connection on error — handle_endpoint_error → zarray_del → endpoint_killer free()s the endpoint_t — while the vtask dispatcher (T3) is concurrently scanning the dataq (dataq_deq_first_match → vtask_eligible → endpoint_is_task_blocked) and reads ep->busy on that just-freed endpoint. The endpoint is freed while still referenced by a pending vtask in the dataq; nothing drains/guards the ep's queued vtasks before the free, and no refcount keeps it alive across the dispatcher read. Same netio surface as bug-1282 (ABBA, ep->busy made lockless) and the bug-1197 dispatch_inflight precedent; the bug-1282 lockless read removed the (incidental) lock contention so the dispatcher reaches the read faster, exposing the latent UAF.",
|
||||
"fix": "NOT FIXED — proposed: drain/cancel an endpoint's queued vtasks from the dataq BEFORE endpoint_killer frees it, or hold a refcount/epoch on endpoint_t across the dispatcher eligibility read so the free is deferred until no dispatcher references remain (mirror bug-1197 dispatch_inflight). Must preserve the bug-1282 lockless busy read and not reintroduce the ABBA ep->lock/q->lock inversion.",
|
||||
"tags": [
|
||||
"netio",
|
||||
"vtaskmgr",
|
||||
"use-after-free",
|
||||
"asan",
|
||||
"endpoint-teardown",
|
||||
"concurrency",
|
||||
"raft-ha-core-safety",
|
||||
"soak",
|
||||
"quorum",
|
||||
"bug-1282-family"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1282",
|
||||
"bug-1197",
|
||||
"bug-1286"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:36:00-07:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1299",
|
||||
"timestamp": "2026-06-30T13:39:09.194Z",
|
||||
"error_message": "Function not marked async",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Function uses await but wasn't declared async",
|
||||
"fix": "Added async modifier",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"async-fix",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:39:09.194Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue