LMDB readers: a stable snapshot can exhaust a fixed map

The same 128 KiB of values, different space pressure

Each environment contained eight keys with 16 KiB values. We repeatedly overwrote one key per committed transaction, keeping the key count and value lengths fixed. The only treatment was the lifetime of a read transaction opened after seeding: end it, keep it active, or reset it while retaining its handle.

The active-reader cases stopped after 30 and 64 commits at map sizes of 1 and 2 MiB respectively. The original reader still returned every original byte. A fresh reader saw every successfully committed change, with no partial failed update. After ending the active reader, all 64 further overwrites succeeded at the unchanged map size.

The committed high-water extent did not decrease after release. Progress came from reuse, not file shrinking. Both controls completed all 256 initial updates and 64 follow-up updates. These constructed cases are not independent samples of production incidents; no failure probability follows.

Not blocking the writer is not the same as releasing space

Chu's 2011 paper describes copy-on-write B+ trees and a separate tree of retired pages, grouped by transaction. A writer can reuse a group only when its retirement predates every active reader. Thus an old snapshot can hold back reclamation even though reads do not take the writer mutex. We examine this mechanism, not the paper's LDAP throughput results. [1, section 4.2]

In the pinned implementation, mdb_find_oldest() scans reader transaction IDs and mdb_page_alloc() excludes free-list records that are too recent. If suitable reusable pages are unavailable, allocation seeks new pages within the map bound. This is a source-code inference for LMDB 0.9.33, not a universal MVCC rule. [3]

The boundary is a transaction ID, not a list of pages explicitly fetched by the reader. Our byte checks establish snapshot consistency for this workload, not a per-page liveness census. Compare the different SQLite WAL checkpoint boundary: both retain historical views, but their reclamation mechanisms differ.

A live old reader is not a stale dead-process slot

At map exhaustion, mdb_reader_check() returned success and cleared zero slots. Another write still returned MDB_MAP_FULL. The reader was alive and intentionally holding a snapshot. We did not kill a process or test stale-slot recovery.

The reviewed implementation checks process liveness for stale reader entries; it is not a transaction-age timeout. Separately, mdb_txn_reset()ends a read transaction but retains a renewable handle. Our reset control kept that handle without retaining the old snapshot. Reading through it again requires mdb_txn_renew(). [2] [3]

Python LMDB warns that cleanup of an abandoned transaction reference can depend on the interpreter's garbage collector. Explicit lifetime management is easier to audit. Our experiment uses the C API and does not measure a binding's cleanup behavior. [4]

Reproduce the observation

We compiled LMDB 0.9.33 at commit 3a29a24777c82a0165de813ae696a5068b5add30, verifying source SHA-256 hashes first. The runtime reported 4,096-byte pages. All 18 environments used private temporary directories, removed after use. No existing database was opened.

Calls were sequential on one thread with overlapping transaction lifetimes.MDB_NOTLS permits the read handle and writer in that arrangement. We did not enable MDB_NOSYNC, MDB_NOMETASYNC,MDB_WRITEMAP or MDB_NOLOCK. Payloads were deterministic.

The harness checks key count, value lengths and all bytes, records extent after each commit, checks the old snapshot, aborts failed put transactions, retries after reader checking, and checks values after release. The runner requires identical outcomes across three repetitions per configuration.

Download the JSON observations, C harness and Python runner. Obtain LMDB at the pinned commit, keep the harness files together, then run:

python3 run-lmdb-readers.py --lmdb /path/to/lmdb --output observations.json

Requires a C compiler, pthreads, Python 3 and a Unix-like temporary filesystem. The JSON records the platform and compiler. Exact counts depend on page size, map geometry, build and workload. This is not a benchmark, crash-consistency test, hardware-fault experiment or field failure-rate estimate.

Capacity, reclamation and compaction are different questions

MDB_MAP_FULL concerns the environment's map-capacity bound, not necessarily exhausted filesystem space. The transaction API separately lists disk-space and I/O errors. [2]

Our recommendation: record map size, committed extent, oldest active transaction and the owning request before changing capacity. Avoid keeping a snapshot across unrelated network waits or unbounded report generation. End or reset it only when its view is no longer needed; renewal obtains a new view.

A bigger map provides headroom but does not remove a retained reader. Ending a reader permits later reuse; it promises neither immediate file shrinking nor secure erasure. We deliberately did not resize, compact or crash an environment.

Corpus: LMDB and LMDB reader table. The SQLite experiment separately examines stale values and transaction retries.

Primary sources

1. Howard Chu. MDB: A Memory-Mapped Database and Backend for OpenLDAP. LDAPCon 2011, sections 3.2 and 4.2.
2. LMDB 0.9.33 API: transaction lifetime, reset/renew, map capacity and reader checks
3. LMDB 0.9.33 implementation: mdb_find_oldest, mdb_page_alloc and mdb_reader_check0
4. Python LMDB documentation: transaction management and long-lived readers