fix: reduce db worker memory retention #13775

Merged
tiensonqin opened 1:11pm on October 16, 2026 wanted to merge 587 Ξ” into logseq/logseq dev from
logseq/reduce-db-worker-memory

Pull Request Overview

  • Opened on October 16, 2026
  • Status Merged
  • Commit count 2 with first commit October 16, 2026

Total Delta

587 Total Diff Delta

Open Days

Open 4 weekdays

Test Delta

348 Diff Delta in Test Files
Breakdown by Phase

How long has this pull request spent in each phase of its lifecycle?

Data pending calculation for pull request

Author avatar

fix: reduce db worker memory retention

Opening a graph or editing repeatedly retains decoded index nodes and obsolete values in the Electron DB worker. Search indexing also retains entity caches across batches and creates excessive temporary allocations, while the native runtime keeps empty heap pools mapped after collection.

Restore worker indexes with weak references, remove the unbounded decoded-payload cache while keeping the current root metadata and pending writes, and materialize search entities per batch. Preserve accent-replacement ordering with bucketed matching, and release native heap capacity after five idle seconds when at least 32 MiB of major allocations have occurred.

Uses the native reference-policy fix merged in https://github.com/logseq/datascript-ocaml/pull/45.

Validation



  • opam exec -- dune build bin/main.exe and opam exec -- dune runtest passed in deps/db-worker, including native retention, snapshot/reopen, persisted weak-reference, Unicode normalization, allocation, and search-build tests; worker/desktop JavaScript tests and multi-client sync simulations passed.

  • Actual Electron open, index, query, edit, close, and reopen checks used disposable SQLite backups of a 25,548-entity graph. Edited content survived a new worker process and entity counts stayed unchanged.


  • git diff --check and decision-document validation passed. Authenticated live remote sync was not exercised.

Electron benchmark

The original strong-cache comparison reduced live heap after full major collection from 53.67 to 4.80 MiB. A separate final pair below measures the search and idle-release changes against a baseline that already includes the weak-reference/storage-cache fix; no diagnostic GC ran in this pair.

Measurement
Before follow-up
After follow-up

Footprint after indexing and idle
108.7 MiB
31.9 MiB

Footprint after 3,000 title transactions and idle
153.6 MiB
73.7 MiB

Startup/index CPU time
19.13 s
15.93 s

Mean point pull, 2,000 requests after 100 warmups
0.267 ms
0.285 ms

3,000 fixed-set title transactions
1.904 s
1.878 s

Startup/index allocations fell approximately 56%. These are development-profile measurements on macOS arm64 with OCaml 5.5.0; the final pair does not establish small throughput differences conclusively. Sustained requests did not trigger idle compaction; subsequent releases took approximately 21–23 ms. The remaining post-edit footprint includes approximately 45 MiB of malloc fragmentation/empty capacity, alongside a 9.1 MiB managed heap. Weak caches trade retention for additional reads, and incoming work can still encounter a GC pause.

The investigation, earlier three-pair benchmark, and evidence references are recorded in docs/agent-guide/implemented/bugfix/2026-10-16-db-worker-memory-growth.md.

Comparison with master CLJS

Three controlled Electron pairs compared master 4cf26891c1 with this native worker using fresh backups, the same existing page, development builds, and natural GC. All runs returned 25,548 UUID entities, produced 17,902 search rows, and verified the final title write. Idle measurements followed 15 seconds without probe requests. Medians:

Measurement
Master CLJS
Fixed native

Startup/index peak footprint
313.5 MiB
45.5 MiB

Footprint after indexing and idle
312.9 MiB
28.5 MiB

Footprint after transactions and idle
192.5 MiB
64.1 MiB

First index build completed
16.53 s
18.85 s

Mean point pull, 2,000 requests after 100 warmups
0.216 ms
0.237 ms

3,000 fixed-set title transactions
5.885 s
1.761 s

Native memory is lower, with a 14% slower first index build, approximately 10% slower mean point pulls, and 70% less transaction time in this workload. Post-transaction idle footprint varied across runs (CLJS 186.0–278.4 MiB; native 48.5–70.1 MiB). These worker HTTP RPC measurements do not establish release-build performance or keyboard-edit latency. The controlled page differs from the earlier journal-view benchmark; compare values within each table.

PR was closed without comments.