→ ↺ λ

LMDB Manual

Table of Contents

[in package LMDB]

← ↑ → ↺ λ

1 The lmdb ASDF System

← ↑ → ↺ λ

2 Links

Here is the official repository and the HTML documentation for the latest version.

← ↑ → ↺ λ

3 Introduction

LMDB, the Lightning Memory-mapped Database, is an ACID key-value database with MVCC. It is a small C library ("C lmdb" from now on), around which lmdb is a Common Lisp wrapper. lmdb covers most of C lmdb's functionality, has a simplified API, much needed Safety checks, and comprehensive documentation.

Compared to other key-value stores, lmdb's distuingishing features are:

Other notable things:

Do read the Caveats, though. On the Lisp side, this library will not work with virtual threads because lmdb's write locking is tied to native threads.

Using lmdb is easy:

(with-temporary-env (*env*)
  (let ((db (get-db "test")))
    (with-txn (:write t)
      (put db "k1" #(2 3))
      (print (g3t db "k1")) ; => #(2 3)
      (del db "k1"))))

More typically, the environment and databases are opened once so that multiple threads and transactions can access them:

(defvar *test-db*)

(unless *env*
  (setq *env* (open-env "/tmp/lmdb-test-env/" :if-does-not-exist :create))
  (setq *test-db* (get-db "test" :value-encoding :utf-8)))

(with-txn (:write t)
  (put *test-db* 1 "hello")
  (print (g3t *test-db* 1)) ; => "hello"
  (del *test-db* 1))

Note how :value-encoding sneaked in above. This was so to make g3t return a string instead of an octet vector.

lmdb treats keys and values as opaque byte arrays to be hung on a B+ tree, and only requires a comparison function to be defined over keys. lmdb knows how to serialize the types (unsigned-byte 64) and string (which are often used as keys so sorting must work as expected). Serialization of the rest of the datatypes is left to the client. See Encoding and decoding data for more.

← ↑ → ↺ λ

4 Design and implementation

← ↑ → ↺ λ

4.1 Safety

The lmdb C API trusts client code to respect its rules. Being C, managing object lifetimes is the biggest burden. There are also rules that are documented, but not enforced. This Lisp wrapper tries to enforce these rules itself and manage object lifetimes in a safe way to avoid data corruption. How and what it does is described in the following.

Environments
Transactions
Databases
Cursors
Signal handling

The C lmdb library handles system calls being interrupted (eintr and eagain), but unwinding the stack from interrupts in the middle of lmdb calls can leave the in-memory data structures such as transactions inconsistent. If this happens, their further use risks data corruption. For this reason, calls to lmdb are performed with interrupts disabled. For SBCL, this means sb-sys:without-interrupts. It is an error when compiling lmdb if an equivalent facility is not found in the Lisp implementation. A warning is signalled if no substitute is found for sb-sys:with-interrupts because this makes the body of with-env, with-txn, with-cursor and similar uninterruptible.

Operations that do not modify the database (g3t, cursor-first, cursor-value, etc) are async unwind safe, and for performance they are called without the above provisions.

Note that the library is not reentrant, so don't call lmdb from signal handlers.

← ↑ → ↺ λ

4.2 Deviations from the C lmdb API

The following are the most prominent deviations and omissions from the C lmdb API in addition to those listed in Safety.

Environments
Transactions
Databases

← ↑ → ↺ λ

5 Library versions

← ↑ → ↺ λ

6 Environments

An environment (class env) is basically a single memory-mapped file holding all the data, plus some flags determining how we interact it. An environment can have multiple databases (class db), each of which is a B+ tree within the same file. An environment is like a database in a relational db, and the databases in it are like tables and indices. The terminology comes from Berkeley DB.

← ↑ → ↺ λ

6.1 Environments reference

← ↑ → ↺ λ

6.2 Opening and closing environments

← ↑ → ↺ λ

6.3 Miscellaneous environment functions

← ↑ → ↺ λ

7 Transactions

The lmdb environment supports transactional reads and writes. By default, these provide the standard ACID (atomicity, consistency, isolation, durability) guarantees. Writes from a transaction are not immediately visible to other transactions. When the transaction is committed, all its writes become visible atomically for future transactions even if Lisp crashes or there is power failure. If the transaction is aborted, its writes are discarded.

Transactions span the entire environment (see env). All the updates made in the course of an update transaction - writing records across all databases, creating databases, and destroying databases - are either completed atomically or rolled back.

Write transactions can be nested. Child transactions see the uncommitted writes of their parent. The child transaction can commit or abort, at which point its writes become visible to the parent transaction or are discarded. If the parent aborts, all of the writes performed in the context of the parent, including those from committed child transactions, are discarded.

← ↑ → ↺ λ

7.1 Nesting transactions

When with-txns are nested (i.e. one is executed in the dynamic extent of another), we speak of nested transactions. Transaction can be nested to arbitrary levels. Child transactions may be committed or aborted independently from their parent transaction (the immediately enclosing with-txn). Committing a child transaction only makes the updates made by it visible to the parent. If the parent then aborts, the child's updates are aborted too. If the parent commits, all child transactions that were not aborted are committed, too.

Actually, the C lmdb library only supports nesting write transactions. To simplify usage, the Lisp side turns read-only with-txns nested in another with-txns into noops.

(with-temporary-env (*env*)
  (let ((db (get-db "test" :value-encoding :uint64)))
    ;; Create a top-level write transaction.
    (with-txn (:write t)
      (put db "p" 0)
      ;; First child transaction
      (with-txn (:write t)
        ;; Writes of the parent are visible in children.
        (assert (= (g3t db "p") 0))
        (put db "c1" 1))
      ;; Parent sees what the child committed (but it's not visible to
      ;; unrelated transactions).
      (assert (= (g3t db "c1") 1))
      ;; Second child transaction
      (with-txn (:write t)
        ;; Sees writes from the parent that came from the first child.
        (assert (= (g3t db "c1") 1))
        (put db "c1" 2)
        (put db "c2" 2)
        (abort-txn)))
    ;; Create a top-level read transaction to check what was committed.
    (with-txn ()
      ;; Since the second child aborted, its writes are discarded.
      (assert (= (g3t db "p") 0))
      (assert (= (g3t db "c1") 1))
      (assert (null (g3t db "c2"))))))

commit-txn, abort-txn, and reset-txn all close the active transaction (see open-txn-p). When the active transaction is not open, database operations such as g3t, put, del signal lmdb-bad-txn-error. Furthermore, any Cursors created in the context of the transaction will no longer be valid (but see cursor-renew).

An lmdb parent transaction and its cursors must not issue operations other than commit-txn and abort-txn while there are active child transactions. As the Lisp side does not expose transaction objects directly, performing Basic operations in the parent transaction is not possible, but it is possible with Cursors as they are tied to the transaction in which they were created.

ignore-parent true overrides the default nesting semantics of with-txn and creates a new top-level transaction, which is not a child of the enclosing with-txn.

Nesting a read transaction in another transaction would be an lmdb-bad-rslot-error according to the C lmdb library, but a read-only with-txn with ignore-parent nil nested in another with-txn is turned into a noop so this edge case is papered over.

← ↑ → ↺ λ

8 Databases

← ↑ → ↺ λ

8.1 The unnamed database

lmdb has a default, unnamed database backed by a B+ tree. This db can hold normal key-value pairs and named databases. The unnamed database can be accessed by passing nil as the database name to get-db. There are some restrictions on the flags of the unnamed database, see lmdb-incompatible-error.

← ↑ → ↺ λ

8.2 dupsort

A prominent feature of lmdb is the ability to associate multiple sorted values with keys, which is enabled by the dupsort argument of get-db. Just as a named database is a B+ tree associated with a key (its name) in the B+ tree of the unnamed database, so do these sorted duplicates form a B+ tree under a key in a named or the unnamed database. Among the Basic operations, put and del are equipped to deal with duplicate values, but g3t is too limited, and Cursors are needed to make full use of dupsort.

When using this feature the limit on the maximum key size applies to duplicate data, as well. See env-max-key-size.

← ↑ → ↺ λ

8.3 Database API

← ↑ → ↺ λ

9 Encoding and decoding data

In the C lmdb library, keys and values are opaque byte vectors only ever inspected internally to maintain the sort order (of keys and also duplicate values if dupsort). The client is given the freedom and the responsibility to choose how to perform conversion to and from byte vectors.

lmdb exposes this full flexibility while at the same time providing reasonable defaults for the common cases. In particular, with the key-encoding and value-encoding arguments of get-db, the data (meaning the key or value here) encoding can be declared explicitly.

Even if the encoding is undeclared, it is recommended to use a single type for keys (and duplicate values) to avoid unexpected conflicts that could arise, for example, when the UTF-8 encoding of a string and the :uint64 encoding of an integer coincide. The same consideration doubly applies to named databases, which share the key space with normal key-value pairs in the default database (see The unnamed database).

Together, :uint64 and :utf-8 cover the common cases for keys. They trade off dynamic typing for easy sortability (using the default C lmdb behaviour). On the other hand, when sorting is not concern (either for keys and values), serialization may be done more freely. For this purpose, using an encoding of :octets or nil with cl-conspack is recommended because it works with complex objects, it encodes object types, it is fast and space-efficient, has a stable specification and an alternative implementation in C. For example:

(with-temporary-env (*env*)
  (let ((db (get-db "test")))
    (with-txn (:write t)
      (put db "key1" (cpk:encode (list :some "stuff" 42)))
      (cpk:decode (g3t db "key1")))))
=> (:SOME "stuff" 42)

Note that multiple db objects with different encodings can be associated with the same C lmdb database, which declutters the code:

(defvar *cpk-encoding*
  (cons #'cpk:encode (alexandria:compose #'cpk:decode #'mdb-val-to-octets)))

(with-temporary-env (*env*)
  (let ((next-id-db (get-db "test" :key-encoding *cpk-encoding*
                                   :value-encoding :uint64))
        (db (get-db "test" :key-encoding *cpk-encoding*
                           :value-encoding *cpk-encoding*)))
    (with-txn (:write t)
      (let ((id (or (g3t next-id-db :next-id) 0)))
        (put next-id-db :next-id (1+ id))
        (put db id (list :some "stuff" 42))
        (g3t db id)))))
=> (:SOME "stuff" 42)
=> T

← ↑ → ↺ λ

9.1 Overriding encodings

Using multiple db objects with different encodings is the recommended practice (see the example in Encoding and decoding data), but when that is inconvenient, one can override the encodings with the following variables.

← ↑ → ↺ λ

10 Basic operations

← ↑ → ↺ λ

11 Cursors

← ↑ → ↺ λ

11.1 Positioning cursors

The following functions position or initialize a cursor while returning the value (a value with dupsort) associated with a key, or both the key and the value. Initialization is successful if there is the cursor points to a key-value pair, which is indicated by the last return value being t.

← ↑ → ↺ λ

11.2 Basic cursor operations

The following operations are similar to g3t, put, del (the Basic operations), but g3t has three variants (cursor-key-value, cursor-key, and cursor-value). All of them require the cursor to be positioned (see Positioning cursors).

← ↑ → ↺ λ

11.3 Miscellaneous cursor operations

← ↑ → ↺ λ

12 Conditions

← ↑ → ↺ λ

12.1 Conditions for C lmdb error codes

The following conditions correspond to C lmdb error codes.

← ↑ ↺ λ

12.2 Additional conditions

The following conditions do not have a dedicated C lmdb error code.