V25 Daric development update - formal testing begins!

We are very excited to announce that the first release candidate for V25 Daric is now ready for formal testing! V25 is a complementary update to the recently released V24 Siglos and completes this two-pronged update to the nano node software. This version includes several enhancements that improve the performance and security of the nano network.

Behind the scenes, a significant amount of work and cleanup has gone into adding the V25 features. These changes have improved testing, modularity, and maintainability of the node codebase and will make future updates easier to validate and merge.

Firstly, a quick note to our node operators. Thank you for sharing our effort and enthusiasm in continually working to improve nano. Please remember, release candidate versions are subject to rapid change, so representative nodes shouldn’t update until the official release. We’ll let you know exactly when testing is complete and it’s time to upgrade. 

What's in V25?

So, what has changed with V25? And what do we expect to see? Let’s dive in!

Ascending bootstrap:

One of the significant features added in V25 is the Ascending Bootstrapper which reduces the time it takes for new nodes to join and sync with the network. The bootstrap process of the node has largely been unchanged from the original design and bringing it up to date required an overhaul of multiple parts of the node.

Of significance, the new ascending bootstrapper:

• No longer uses the disk (unchecked table) for storing blocks while working
• Communicates blocks through existing realtime sockets rather than needing to open separate bootstrap sockets. This makes the bootstrap process firewall and NAT friendly.
• Scores bootstrap peers and prioritises requests to the most responsive nodes
• Makes smaller requests to more nodes rather than larger requests to a few
• Improves general performance of the bootstrap process as a whole

The new mechanism uses stateless bootstrap messages to send bootstrap query/response through the node's real-time socket, allowing nodes behind a firewall to participate in bootstrapping. Additionally, the Ascending Bootstrap mechanism reduces the consumption disk resources by lightening the burden on the data structure that holds the "unchecked" blocks, resulting in improved efficiency.

Optimistic elections:

Optimistic elections are another major feature that speeds up the cementing process, especially during the bootstrapping phase. The feature introduces a new type of election scheduler, the optimistic scheduler, that operates alongside the default election scheduler. The optimistic scheduler randomly samples accounts with unconfirmed blocks and activates an election for the account head block, reducing the time required to confirm all blocks in an account chain.

Continuous Backlog Population

The Continuous Backlog Population feature improves the scanning process of all accounts in the ledger for uncemented blocks by queuing and prioritising them in proper buckets. This reduces the consumption of node resources and improves efficiency.

Node handshake

The node handshake, responsible for establishing a connection between two nodes, has been upgraded to add an additional layer of security to the network by including salt and genesis block hash fields in the node ID handshake exchange process. The upgrade is backward compatible, and legacy handshake support will be phased out gradually over time.

Test infrastructure

Over the past few months, our team has continued to enhance the nano network's testing infrastructure. These improvements aim to enable faster development by catching bugs early and facilitating proactive tweaking. Special thanks to Gr0vity and Ricki for volunteering to write and run this testing framework.

To evaluate new features going forward, we have introduced new tools that spin up an arbitrary number of nodes on the beta network. These nodes serve as bootstrapping nodes, putting pressure on the network and allowing us to observe its performance under various conditions. To evaluate the impact, we perform CPS throughput tests against the Beta network, both with and without the bootstrapping nodes. 

Another component of the test infrastructure is a series of automated tests that spin up nano networks on GitHub runners. These tests run automatically for each new commit and pull request, ensuring our code is rigorously tested and potential issues are promptly identified and addressed.

Finally, over the past few weeks, we have developed a new tool that enables the creation of configurable nano networks in the cloud. This advancement allows us to better understand network performance under various real-world conditions and ensures more accurate testing.

The benefits of this tool include:

• Gaining more reliable test results due to real resource limitations, such as CPU and bandwidth constraints.
• Refining and enhancing local test parameters by running our existing test cases against the cloud network and comparing the results for improved accuracy.
• Simulating our live network to better understand its behaviour under different scenarios and conditions.

We have designed a cloud network consisting of 30 representative nodes with vote weight and geographical location according to nodes on our live network. This tool enables us to simulate live network behaviour in various situations, such as the impact of network throughput when nodes upgrade to a specific version. This helps us understand the implications of fine-grained changes and upgrading individual nodes.

The following shows a simulation to test network recovery performance with different percentages of nodes running on V25:

As we move forward, our goal is to further integrate these tools into our automated testing pipeline, ensuring a more robust and reliable testing process.

Code hygiene

We’ve improved several parts of the node build process: The code has been upgraded to C++20 and the docker image has been upgraded to Ubuntu 22.04 LTS. Boost was upgraded to 1.81 and is now pulled in directly as a submodule rather than needing to be built externally. A lot of work has been done splitting files and documenting classes.

Expectations for V25

The enhancements introduced in V25 significantly improve the efficiency of the bootstrapping process by optimising node resource usage and increasing the speed of block cementing. The Ascending Bootstrap mechanism, stateless bootstrap messages, and Continuous Backlog Population feature improve the node's resource usage in disk and bandwidth, while optimistic elections increase the speed of block cementing during bootstrapping. These features work together to provide a smoother, more efficient user experience on the nano network.

What comes next?

Plans for V26 are already underway. Subject to change, current V26 plans include: refactoring of the network code, splitting the block table, improving election scheduler bucket bounding, and refactoring of the block processing pipeline. These changes will improve the network’s flow control and response under heavy load.

We can’t wait to formally release V25 - an important step on the road towards commercial grade. We want to thank everyone on the dev team as well as all of nano’s community contributors for their commitment and hard work. Remember, there are many ways to help maintain and improve the nano network with C++ and non C++ contributions. 

If you have the skills and drive to help, join our development Discord!