It’s a lot faster and there are more cool options
We’re releasing v0.22 today! The last release was on [TR], and the OrbitDB team has been working hard since then to make sure that this next release makes OrbitDB faster, easier, and more delightful to work with. Here’s the tl;dr:
- Significant performance improvements (by significant, we mean 10x)
- Better concurrency across the board
- Lots of small bug fixes and a few option improvements
This wouldn’t have been possible without our community. So, thank you to:
- And anyone else we’ve unintentionally left out, who has helped out or been on the Gitter or supported us in any way.
The rest of this post will be fairly technical. Here’s a ToC:
- Performance Improvements
- Other Options
- Other Stuff
$ node benchmarks/benchmark-add.js 47 queries per second, 47 queries in 1 seconds (Oplog: 47) 39 queries per second, 86 queries in 2 seconds (Oplog: 86) 43 queries per second, 129 queries in 3 seconds (Oplog: 129) 38 queries per second, 167 queries in 4 seconds (Oplog: 167) 38 queries per second, 205 queries in 5 seconds (Oplog: 205) 35 queries per second, 240 queries in 6 seconds (Oplog: 241) 35 queries per second, 275 queries in 7 seconds (Oplog: 275) 39 queries per second, 314 queries in 8 seconds (Oplog: 314) 40 queries per second, 354 queries in 9 seconds (Oplog: 354) --> Average of 39.1 q/s in the last 10 seconds
$ node benchmarks/benchmark-add.js 431 queries per second, 431 queries in 1 seconds (Oplog: 431) 471 queries per second, 902 queries in 2 seconds (Oplog: 902) 476 queries per second, 1378 queries in 3 seconds (Oplog: 1378) 492 queries per second, 1870 queries in 4 seconds (Oplog: 1870) 493 queries per second, 2363 queries in 5 seconds (Oplog: 2363) 496 queries per second, 2859 queries in 6 seconds (Oplog: 2859) 492 queries per second, 3351 queries in 7 seconds (Oplog: 3351) 498 queries per second, 3849 queries in 8 seconds (Oplog: 3849) 504 queries per second, 4353 queries in 9 seconds (Oplog: 4353)
Here’s how we did this.
Better concurrency in
Certain expensive operations in
ipfs-log have been refactored to be concurrent instead of serial.
We’ve adjusted the default concurrency during traversal to balance load times with write performance.
Introducing the OrbitDB Storage Adapter
OrbitDB keeps local-only data stores for its keystore and cache. OrbitDB uses
level for this, but OrbitDB is now compatible with any library that implements abstract leveldown.
Our automated tests currently run with the following leveldown implementations
Things the reader should know
- A Cache schema migration is required for 0.21 -> 0.22. The migration should happen automatically with no user action, but if you have problems create an issue in orbitdb/orbit-db.
- This is not required, but we are moving toward a first-principle of OrbitDB being a singleton orchestrator for an entire node.
- Custom Sort function - You can now pass in
orbitdb.createto provide your own, custom sorting algorithm.
- Timeout - The Timeout option is now passed through all
ipfs-logfetches and traversals.
- skipManifest - This option allows the user to skip the manifest step when using a custom access controller.
orbit-db-ionow supports ProtoBuf, CBOR, and Raw formats when creating DAG nodes.
Adding timeout option to ipfs-log
Concurrent Browser Tab support:
Browser users can now utilize multiple tabs with the same OrbitDB ID and expect deterministic sorting
Further refactoring of our codebase to allow the entryIndex to be more swappable and pluggable in
We’ve extended the Lamport Clock model so that not only does it match by “time” and fall back to “ID”, but now there’s a third fallback layer - the “content ID”:
So if the same ID writes the same time from two different devices, it will compare the hashes to finally complete the sort deterministically.
js-ipfs 0.37 support
We now support js-ipfs 0.37 across the board. We’re also working on supporting 0.38, and already have that enabled in the
develop branch in orbitdb/orbit-db.
Refactor: identity.id instead of identity.key
We’re now using
identity.id instead of
identity.key in the access controllers.
New formats in ipfs-io
You can now write the arguments for orbit-db-io directly, without needing to stringify.
A metafield in the manifest
Users now have the ability to add additional data to the db manifest. This could be useful when wanting to package information with the db that should not change during the lifetime of the db. An example could be adding a public ECDH key that the owner of the db controls. (Thanks, @tabcat!)
General bugfixes and updates:
To save you reading endless issues you may have already known about, here is a general list of PRs with bugfixes and updates. The tl:dr of this is that we’ve made OrbitDB better, and you shouldn’t have to change anything for these to take effect.
And of course, we also have merged lots of tests and doc fixes:
And that’s it! As always, if you have any questions, reach out to us on GitHub at any of the relevant repositories, or on our Gitter.