BangDB vs LevelDB - Performance Comparison

This post is about performance comparison for BangDB vs LevelDB. Following are high level overview of the dbs.

LevelDB

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Leveldb is based on LSM (Log-Structured Merge-Tree) and uses SSTable and MemTable for the database implementation. It's written in C++ and availabe under BSD license. LevelDB treats key and value as arbitrary byte arrays and stores keys in ordered fashion. It uses snappy compression for the data compression. Write and Read are concurrent for the db, but write performs best with single thread whereas Read scales with number of cores

BangDB

BangDB is a high performance multi-flavored distributed transactional nosql database for key value store. It's written in C++ and available under BSD license. BangDB treats key and value as arbitrary byte arrays and stores keys in both ordered fashion using BTREE and un-ordered way using HASH. Write, Read are concurrent and scales well with the number of cores. BangDB used here is the embedded version as LevelDB is also an embedded db, but BangDB is also available in other flavors like client/server, clustered and Data Fabric(upcoming)

Following commodity machine ($400 commodity hardware) used for the test;

• Model: 4 CPU cores, Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz, 64bit
• CPU cache : 6MB
• OS : Linux, 3.2.0-54-generic, Ubuntu, x86_64
• RAM : 8GB
• Disk : 500GB, 7200 RPM, 16MB cache
• File System: ext4

Following are the configuration that are kept constant throughout the analysis (unless restated before the test)

• Assertion: OFF
• Compression for LevelDB: ON
• Write and Read: Sequential and Random as mentioned before the test
• Access method: Tree/Btree
• Key Size: 10 bytes
• Value Size: 16 bytes

The tests are designed to cover following;

Performance of the dbs for

1. sequential write and read for 100M keys using 1 thread
2. sequential write and read for 75M keys using 4 threads
3. random write and read for 75M keys using 1 thread
4. random write and read for 75M keys using 4 threads
5. sequential write and read for 1 Billion keys using 1 thread