Benchmark of Transact 0.3.7 LMDB and experimental SQLite backends
Summary
Transact provides a number of core features used by Splinter, such as transaction execution and state storage. As these features play an important role in the operation of a splinter network, it is important to measure the performance of these subsystems to determine where optimizations can be made.
The benchmarks included here cover the performance of one of the lowest-level areas: Transact’s database abstraction, and two of its back-end implementations, LMDB and SQLite (currently an experimental feature). These benchmarks have been run on several file systems: Elastic Block Store (EBS), Elastic File System (EFS), and APFS (Apple File System).
EFS has a noticeable performance impact, with a 18x slowdown for LMDB and a 99x slowdown for SQLite. These results strongly imply that with the current implementations, LMDB would be the preferred choice for a back-end implemented over a file on EFS.
Environment
The environments where these benchmarks have been run is in AWS as well as on the developer’s local system. In AWS, two different file system types are used: Elastic Block Store (EBS) and Elastic File System (EFS).
t2.medium
-
2 vCPU
“T2 instances are backed by the latest Intel Xeon processors with clock speeds up to 3.3 GHz during burst periods.”
- 4GB RAM
- OS: ubuntu-bionic-18.04-amd64-server-20201026
- EBS (for a single node configuration) and EFS
Local Development Environment
- MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports)
- Processor: 2.3 GHz Quad-Core Intel Core i7
- Memory: 32 GB 3733 MHz LPDDR4X
- OS: macOS Catalina 10.15.7
Setup
The tests have been compiled in release mode, and include the experimental feature “sqlitedb” enabled:
cargo build --manifest-path libtransact/Cargo.toml \
--tests \
--release \
--features sqlite-db
In these benchmark runs, the integration tests compiled to a binary
mod-5b692c58f25d0ea4
on AWS. This includes all of the tests that cover the
Merkle Trie overlayed on multiple back-ends: LMDB, SQLite, BTree (i.e. Memory),
and Redis (not exercised in these benchmarks).
On macOS, the compiled binary is mod-a47d5999419a955f
. This particular
instance has been built against SQlite version
3.34.0 2020-12-01 16:14:00 a26b6597e3ae272231b96f9982c3bcc17ddec2f2b6eb4df06a224b91089fed5b
This is a newer version than the default sqlite3 installation provided by the OS.
For each benchmark, the test runs are limited to those specific to a given
back-end and prefixed with merkle_trie_
. In some back-ends, the test
merkle_trie_update
has been renamed, though the behavior remains the same.
Results
AWS B-Tree Baseline
Running the tests against the B-tree back-end. This back-end is memory-only, and therefore not dependent on file system characteristics.
$ time ./target/release/deps/mod-5b692c58f25d0ea4 \
-- btree::merkle_trie
running 10 tests
test state::merkle::btree::merkle_trie_empty_changes ... ok
test state::merkle::btree::merkle_trie_delete ... ok
test state::merkle::btree::merkle_trie_pruning_parent ... ok
test state::merkle::btree::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::btree::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::btree::merkle_trie_root_advance ... ok
test state::merkle::btree::merkle_trie_pruning_successors ... ok
test state::merkle::btree::merkle_trie_update_same_address_space ... ok
test state::merkle::btree::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::btree::merkle_trie_update ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m0.843s
user 0m0.822s
sys 0m0.028s
EBS
LMDB
For LMDB tests, merkle_trie_update
has been named
merkle_trie_update_multiple_entries
.
$ time ./target/release/deps/mod-5b692c58f25d0ea4 \
-- lmdb::merkle_trie
running 10 tests
test state::merkle::lmdb::merkle_trie_empty_changes ... ok
test state::merkle::lmdb::merkle_trie_delete ... ok
test state::merkle::lmdb::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::lmdb::merkle_trie_pruning_parent ... ok
test state::merkle::lmdb::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::lmdb::merkle_trie_root_advance ... ok
test state::merkle::lmdb::merkle_trie_pruning_successors ... ok
test state::merkle::lmdb::merkle_trie_update_same_address_space ... ok
test state::merkle::lmdb::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::lmdb::merkle_trie_update_multiple_entries ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m0.850s
user 0m0.825s
sys 0m0.036s
SQLite
The merkle_trie_update
test has been named,
merkle_trie_update_with_wal_mode
, which configures sqlite with a fast
configuration. The additional tests merkle_trie_update_with_sync_full_wal_mode
and merkle_trie_update_atomic_commit_rollback
have been ignored, as these run
the same test assertions as merkle_trie_update
, but with slower performance
configurations.
$ time ./target/release/deps/mod-5b692c58f25d0ea4 \
--skip atomic \
--skip sync_full \
-- sqlitedb::merkle_trie
running 10 tests
test state::merkle::sqlitedb::merkle_trie_empty_changes ... ok
test state::merkle::sqlitedb::merkle_trie_delete ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_parent ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_successors ... ok
test state::merkle::sqlitedb::merkle_trie_root_advance ... ok
test state::merkle::sqlitedb::merkle_trie_update_same_address_space ... ok
test state::merkle::sqlitedb::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::sqlitedb::merkle_trie_update_with_wal_mode ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m6.737s
user 0m1.510s
sys 0m0.527s
EFS
In order to run the tests on EFS, a temp directory is created on the EFS
partition. In these tests, it has been set to /efs/benchmark
.
$ mkdir /efs/benchmark
$ export TMPDIR=/efs/benchmark
$ export TMP=$TMPDIR
$ export TEMP=$TMPDIR
LMDB
$ time ./target/release/deps/mod-5b692c58f25d0ea4 \
-- lmdb::merkle_trie
running 10 tests
test state::merkle::lmdb::merkle_trie_empty_changes ... ok
test state::merkle::lmdb::merkle_trie_delete ... ok
test state::merkle::lmdb::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::lmdb::merkle_trie_pruning_parent ... ok
test state::merkle::lmdb::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::lmdb::merkle_trie_pruning_successors ... ok
test state::merkle::lmdb::merkle_trie_root_advance ... ok
test state::merkle::lmdb::merkle_trie_update_same_address_space ... ok
test state::merkle::lmdb::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::lmdb::merkle_trie_update_multiple_entries ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m15.440s
user 0m1.139s
sys 0m0.064s
This represents a 18x slowdown.
SQLite
$ time ./target/release/deps/mod-5b692c58f25d0ea4 \
--skip atomic \
--skip sync_full \
-- sqlitedb::merkle_trie
running 10 tests
test state::merkle::sqlitedb::merkle_trie_empty_changes ... ok
test state::merkle::sqlitedb::merkle_trie_delete ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_parent ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_successors ... ok
test state::merkle::sqlitedb::merkle_trie_root_advance ... ok
test state::merkle::sqlitedb::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::sqlitedb::merkle_trie_update_same_address_space ... ok
test state::merkle::sqlitedb::merkle_trie_update_with_wal_mode ... test state::merkle::sqlitedb::merkle_trie_update_with_wal_mode has been running for over 60 seconds
test state::merkle::sqlitedb::merkle_trie_update_with_wal_mode ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 11m3.621s
user 0m4.483s
sys 0m7.279s
This represents a 99x slowdown. There are several possible improvements that are being explored, though many are only safe to apply in single-threaded, single-process environments.
Several options include disabling the journal, and disabling file synchronization.
macOS
B-Tree
$ time target/release/deps/mod-a47d5999419a955f \
-- btree::merkle_trie
running 10 tests
test state::merkle::btree::merkle_trie_empty_changes ... ok
test state::merkle::btree::merkle_trie_delete ... ok
test state::merkle::btree::merkle_trie_root_advance ... ok
test state::merkle::btree::merkle_trie_pruning_parent ... ok
test state::merkle::btree::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::btree::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::btree::merkle_trie_pruning_successors ... ok
test state::merkle::btree::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::btree::merkle_trie_update_same_address_space ... ok
test state::merkle::btree::merkle_trie_update ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m0.746s
user 0m0.744s
sys 0m0.018s
LMDB
$ time target/release/deps/mod-a47d5999419a955f \
-- lmdb::merkle_trie
running 10 tests
test state::merkle::lmdb::merkle_trie_empty_changes ... ok
test state::merkle::lmdb::merkle_trie_delete ... ok
test state::merkle::lmdb::merkle_trie_root_advance ... ok
test state::merkle::lmdb::merkle_trie_pruning_parent ... ok
test state::merkle::lmdb::merkle_trie_pruning_successors ... ok
test state::merkle::lmdb::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::lmdb::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::lmdb::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::lmdb::merkle_trie_update_same_address_space ... ok
test state::merkle::lmdb::merkle_trie_update_multiple_entries ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m1.646s
user 0m0.881s
sys 0m0.812s
SQLite
$ time target/release/deps/mod-a47d5999419a955f \
--skip atomic \
--skip sync_full \
-- sqlitedb::merkle_trie
running 10 tests
test state::merkle::sqlitedb::merkle_trie_empty_changes ... ok
test state::merkle::sqlitedb::merkle_trie_root_advance ... ok
test state::merkle::sqlitedb::merkle_trie_delete ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_parent ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_successor_duplicate_leaves ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_duplicate_leaves ... ok
test state::merkle::sqlitedb::merkle_trie_pruning_successors ... ok
test state::merkle::sqlitedb::merkle_trie_update_same_address_space_with_no_children ... ok
test state::merkle::sqlitedb::merkle_trie_update_same_address_space ... ok
test state::merkle::sqlitedb::merkle_trie_update_with_wal_mode ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out
real 0m3.152s
user 0m1.416s
sys 0m0.826s
Conclusions
The benchmarks show several interesting results. First, LMDB on EBS has very little overhead compared to the baseline B-Tree implementation. On macOS, it has a higher overhead, most likely due poorer memory-mapped file support.
EFS has a noticeable performance impact, with a 18x slowdown for LMDB and a 99x slowdown for SQLite. These results strongly imply that with the current implementations, LMDB would be the preferred choice for a back-end implemented over a file on EFS.
In both file systems, a number of changes may be required for the SQLite implementation to improve performance in general, as well as single-threaded environments. Some of these changes will most likely be implemented before the feature is stabilized, as they will require changes to the SQLite database builder API.