Server Performance

Updated: 3 Jan, 2022

Sparrow Wallet depends on the Electrum server protocol for retrieving and sending transaction information. This document serves to provide a reasonably up-to-date performance benchmark for different full index Electrum server implementations running on commonly used hardware. It also contains a discussion on two different approaches taken by implementations based on these findings.

Background

An Electrum server is an example of a Bitcoin address index. In simple terms, you provide it an arbitrary address, and it returns transactions associated with that address. The Bitcoin reference implementation does not (and probably will never) support this functionality. Note that you can also obtain this information using compact block filters, but this method does not support mempool transactions and is much less performant.

Other examples of address indexes exist, but the Electrum server protocol is by far the most widely used Bitcoin address index protocol. This benchmark focuses on full address indexes. A full address index is important privacy-wise because it does not store details about any particular wallet on the server. For true cold storage, all details about the wallet should be contained in the wallet file and not left on the server once the wallet has been closed. This excludes projects such as BWT and EPS which must store the addresses for any wallet provided to them.

Motivation

A previous performance report was written by Jameson Lopp in July 2020, and is worth reading.

This page serves to build on that work in two ways:

  1. A Raspberry Pi 4 is used instead of an AWS server
  2. Up-to-date builds of the projects are used, and retested whenever important changes are made

As such, Sparrow users who are looking to run their own Electrum server on a single board computer may find it useful to compare and contrast different implementations for their own needs.

Hardware

In order to mimic the situation of most privacy-conscious Sparrow users, a single board computer (SBC) is used for all these tests. Currently the most common SBC is a Raspberry Pi 4. This benchmark uses the 8Gb iteration of that board running Raspbian Buster.

The Bitcoin blockchain, and any indexes built off of it, are fairly large in data size (currently around 0.5 Tb). For this reason, it is generally recommended to prefer an SSD over a normal HDD for data storage. This test uses a 1Tb external USB SSD.

Projects

ElectrumX

ElectrumX is the second iteration of Electrum server implementations after the original Electrum server project was abandoned in favour of it in 2017. After the original author of ElectrumX decided not to support the Bitcoin blockchain, the Electrum developers forked the project. It is now maintained at https://github.com/spesmilo/electrumx.

The chief difficulty in running ElectrumX is initially building the index. Building the index on the hardware used for this test takes many weeks. Note however that the index can be built on a more powerful computer and transferred to an SBC.

  • Current database size: 75 Gb
  • Requires txindex enabled on Bitcoin Core
  • Tested version: ElectrumX 1.16

Electrs

While ElectrumX was designed with public server use in mind, Electrs is designed for personal use. As such, it has lower storage requirements (but higher CPU usage as we will see later). It is maintained at https://github.com/romanz/electrs.

In contrast to ElectrumX, building the index takes 12-24 hours on the hardware used for this test. This has led to it being preferred on all prebuilt node packages.

  • Current database size: 31 Gb
  • Does not require txindex on Bitcoin Core
  • Tested version: Electrs 0.9.4

Electrs-esplora

This is a fork of the Electrs project which builds a number of additional indexes to improve performance for enterprise use. However given the high data requirements (> 0.5Tb) this implementation was not considered suitable for the hardware used in this test.

addrindexrs

This is another fork of the Electrs project used by the Samourai Dojo backend to retrieve historical transaction data. There are no significant performance-related changes to warrant inclusion as a separate implementation.

Indexing

Before proceeding to the performance testing it’s important to consider initial indexing. ElectrumX and Electrs differ hugely on this point.

At the time of testing, ElectrumX had taken 53d 15h 23m to build it’s index on the test hardware - almost two months! In addition, a memory allocation bug caused ElectrumX to crash after every hour or so of indexing, meaning it needed to be frequently restarted. Any restarts (caused by crash or normal shutdown) meant that any work since the last database flush was lost (and in fact needed to be deleted first, which took even more time).

Electrs, on the other hand, builds its initial index with 24 hours.

This vast difference cannot be explained by the size of index. If the both servers were simply IO bound, ElectrumX would take roughly twice the time. Instead, the indexing algorithm on the test hardware is CPU bound - in other words, bounded by the speed of the processor. Profiling the indexing of ElectrumX reveals that most of the time is spent parsing blocks, or (to a lesser extent) waiting for blocks to be received from bitcoind. Both ElectrumX and Electrs must do this work - parsing transaction inputs and outputs - in order to create their indexes. The difference therefore lies in the implementation. Some of this may be explained in that ElectrumX is written in Python which is single-threaded, while Electrs is written in Rust and utilizes multiple threads when indexing.

In any event, this difference means that ElectrumX is unlikely to be used in a prebuilt node package. It does however remain a reasonable option when the database is built on a faster machine. Even when the architecture of that machine differs, the ElectrumX database can be transferred to the Raspberry Pi which can keep it up to date with relative ease.

The Test

In order to provide meaningful numbers, a large wallet (2000+ used addresses) was used.

This benchmark serves to test two common (but quite different) server loads from Sparrow:

  1. Initial loading of a wallet. Since an existing Sparrow wallet already contains much of the wallet data (transactions and blocks) which does not need to be re-retrieved, this test focuses on address subscriptions. Address subscriptions allow Sparrow to “subscribe” an address so that the server will provide it with updates whenever transactions for an address change. In addition, a subscription request returns a hash of all of the transaction ids (txids) and block heights affecting that address. This second requirement is key to understanding server performance. The test measures how long subscribing to all the addresses in the wallet takes.
  2. Refreshing a wallet. In ideal circumstances, wallet refreshes should never be necessary - the wallet should only need to be updated incrementally. However, communication and server issues can lead to bad data, which is solved by refreshing the wallet in Sparrow (View menu). This test measures only the time to retrieve all wallet data (transactions and blocks), since addresses are already subscribed to when a refresh takes place.

Both Electrs and ElectrumX support batching of requests. For this test, a batch page size of 50 was used.

Results

Test 1: Initial load

Test Cold Start Run 1 Run 2 Run 3 Average
ElectrumX 4247 ms 4118 ms 3985 ms 3952 ms 4076 ms
Electrs 322386 ms 302511 ms 305844 ms 303903 ms 308661 ms

Test 2: Wallet refresh

Test Cold Start Run 1 Run 2 Run 3 Average
ElectrumX 16363 ms 12580 ms 12834 ms 12445 ms 13556 ms
Electrs 10780 ms 28363 ms 16131 ms 20965 ms 19082 ms

Discussion

The results of this test are at first surprising for Test 1. There is a vast difference in performance between ElectrumX and Electrs (4 secs vs 5 mins to produce the same data!). This is primarily because Electrs does not store all the data required to find the transactions associated with an address.

As can be seen from the Electrs database schema, only the block height for an address is stored. To be more specific, the ScriptPubKey is hashed and the first 8 bytes of this ‘script hash’ are stored as a key where the value is the confirmed block height for a transaction associated with that script hash. This means that block must be retrieved from Bitcoin Core using the P2P interface and parsed for transactions that have outputs matching the address. For a deep wallet such as that used in this test, this can be a lot of data that must be reparsed for every wallet load. For this particular wallet, 3.5Gb of blocks needed to be fetched from Bitcoin Core and parsed every time the wallet is loaded!

This presents a significant CPU load for an SBC, both for the Bitcoin Core and Electrs processes involved. The most significant problem this causes is that Electrs is sometimes so busy it fails to respond to any other requests, or to notify Sparrow when one of its subscribed addresses has new transactions during this period. This can lead to outdated or bad data in any wallet should a loading wallet have significant depth. If you have ever had cause to use the Refresh Wallet function in Sparrow when using Electrs, this is probably the reason.

ElectrumX on the other hand does store the data required for a full address index lookup. It’s history database is currently 49Gb and contains a summarised blockchain necessary to perform a lookup for the transactions of an address internally.

It does so in an interesting way. Similar to Electrs, the history database stores the first 11 bytes of the script hash as a key (this slightly longer key increases data storage but reduces ‘false positive’ incorrect lookups). Contrary to Electrs however, the value stored against this key is a list containing the ordered numbers (called tx_num) of its associated transactions in the blockchain. The tx_num can be described as follows: Each transaction is ordered within a block, so a cumulative number for a transaction can be determined by looking at its order in the entire blockchain. This is effectively a shorthand for the txid, which not only reduces data storage but also allows for fast block height lookups.

For example, the txid is determined from a logical file containing all txids in blockchain order, where the offset in the file is simply the tx_num * 32 bytes (the size in bytes of a SHA256 hash). Similarly, ElectrumX keeps an array in memory where each index corresponds to the blockheight, and the value at that index contains the cumulative number of transactions that existed at that height. This means from the list of tx_nums retrieved from a script hash, ElectrumX can quickly lookup both the txid and the block height for each tx_num. This is how it achieves such fast performance in comparison to Electrs. The cost to this performance is increased data storage - an additional 21Gb of storage is needed for the logical file to perform the tx_num to txid lookup.

Conclusion

We can be relatively sure of two trends continuing into the next few years - wallet depths will increase, and storage costs will decrease. For this reason, it may make sense for current and future implementations to consider a larger, more inclusive index rather than a dependence on Bitcoin Core’s database for address-to-transaction lookups.

So long as care is taken to ensure initial indexing can be completed in reasonable time, such an address index will be better able to scale to meet the demands of Bitcoiners in the future.