Firedancer: A Solution for Solana’s Challenges in the Crypto World
⚠️ This is a rabbit hole article: To avoid boring you with unnecessary details, this post is divided into sections based on the level of detail you desire (similar to a rabbit 🕳️)
LEVEL I : FIREDANCER – AN INTRODUCTION
1️⃣ Solana’s Challenges – And where Firedancer Fits In
Solana’s network relies on a single validator client used by all third-generation blockchain nodes to support the network.
To make the blockchain less prone to failure, introducing a second validator client designed for performance is an exciting evolution in Solana’s promise as a layer one chain in the crypto world.
Solana has faced various challenges, including
- code errors that developers have yet to fix and
- bots that overload the network with up to 400,000 transactions per second
This makes it impossible for regular users to conduct transactions. Network stability has become a top priority, with a recent drop in performance caused by a validator misconfiguration.
However, a new scaling solution called Firedancer is in the works by Jump Crypto and Solana co-founder, Anatoly Yakovenko, stated that a final solution will be found.
The implementation should make transaction processing more efficient. The community version of SafeCoin has already shown room for improvement, with founder Jeff Galloway’s team enhancing efficiency and security, resulting in the world’s most energy-efficient blockchain.
It’s unclear if Firedancer can match SafeCoin’s performance, but Yakovenko believes it can correct the programming errors made by his team, which other developers haven’t made.
Therefore, the community can be confident that past problems are unlikely to recur.
2️⃣ What is Firedancer?
Firedancer is a new validator client for the Solana blockchain made by the Jump Crypto team.
It aims to improve Solana’s current and only validator client by enhancing performance and scalability while adding diversity to its ecosystem.
- During a live demo in November 2022, a single validator using Firedancer processed 1.2 million transactions per second, indicating improved performance and redundancy.
- Furthermore, Firedancer is designed to be more efficient, which may reduce the cost of running Solana nodes and lower operating costs for node operators.
- However, the total cost of running a node depends on various factors, including hardware and infrastructure expenses.
3️⃣ History of Firedancer
Jump, a well-known trading and crypto company, has developed Firedancer, an independent validator for Solana. Jump is a strong player in trading, and they want Firedancer to be the most secure and fast trading environment in the world.
Jump Labs started in 2015 as an intern project at UIUC’s Research Park, and it developed into an HFT Trading Firm.
Jump Crypto has expanded beyond trading to become active in decentralized finance, investing, governance voting, and community participation. It even provided a DeFi bailout of $320 million for the Wormhole cross-chain bridge.
Kanav Kariya, who worked as an intern at Jump Trading Group’s digital asset division, became Jump Crypto’s president in September 2021. Under Kariya’s leadership, Jump developed tools like Pyth, providing real-time asset pricing data for DeFi apps. Jump also agreed to develop upgrades for Solana and create a new validator client.
Jump Crypto has approximately 140 employees, including over 100 developers. Half of the developers work in trading, while the other half works on protocol code. Jump has invested in core crypto infrastructure, such as Serum, Pyth, and Wormhole, and stays ahead of market structure changes, according to Kariya.
4️⃣ What Does Firedancer Do?
Currently, Solana’s software inefficiencies (not hardware) limit its throughput. Firedancer aims to solve this issue and enhance decentralization and performance on the network.
Some PoS consensus methods have a fatal flaw of being overthrown by the majority of stakeholders. With Firedancer, just over a third of stakers running the validator client can prevent a network supermajority, and having a second validator eliminates a single point of failure.
Firedancer’s core code is written in C/C++ programming languages, commonly used for low-level computer hardware access. Using these languages means shorter operating times, higher throughput, and faster transaction speeds.
How does it help?
Firedancer will :
- Improve network efficiency.
- Help scale the network.
- Make it cheaper to run a validator node.
- Make the network more decentralized.
5️⃣ How Could Firedancer Improve Solana’s Efficiency?
Firedancer will be designed to use hardware efficiently.
It will avoid redundant data copying, inefficient locking, and NUMA-unaware memory allocation. These techniques are common in trading but not commonly used in standard software development, including Solana Labs’ current implementation.
Firedancer will use zero-copy networking to bypass most of the operating system’s kernel network stack and include its queues and concurrency primitives. The development team has decades of experience writing highly efficient software.
6️⃣ Programming Languages on Firedancer
Firedancer is developed in C and C++, programming languages known for hardware performance and reliability.
- They’re fast and memory-efficient, ideal for building high-performance software.
- Early releases of Firedancer will have some compatibility with Rust, which is commonly used in smart contracts.
- However, the project will mainly focus on C/C++-based codebase as it ages to align with the systems underlying language and logic.
7️⃣ Firedancer vs. Current Solana Node Client
A) What are the current Solana node client’s limitations?
Clients of Solana validator face several restrictions that can impact their performance and scalability. The majority of these limitations are caused by software rather than hardware. These are some of the limitations:
1) Transaction Processing in Concurrent Mode
Because Solana validators can only process a limited number of transactions simultaneously, transaction processing can be slowed, and bottlenecks can occur. This could become an issue as the Solana network expands and the number of transactions increases.
2) Sharding Support as a Result
Solana’s current validator clients do not support sharding, a technique for scaling a blockchain horizontally by dividing it into smaller pieces (shards). As the network grows, this can lead to a need for more scalability.
B) What are the improvements of the new Firedancer node client?
The Firedancer client aims to address existing limitations and improve on current Solana validator clients in a variety of ways, including enhanced concurrent transaction processing, sharding support, optimized P2P communication, and improved consensus protocol support.
1) Concurrent Transaction Processing Enhancement
The Firedancer client is designed to handle more concurrent transactions, resulting in faster transaction processing times and fewer bottlenecks. This is accomplished through C and C++, known for their high performance and dependability.
Additionally, upon launch, compatibility with Rust-based decentralized applications. Concurrency codebases can port to the new validator and reap the efficiency benefits.
2) Sharding Assistance
In addition to improved concurrent transaction processing, the Firedancer client supports sharding. As the network grows, this can improve scalability and allow the Solana blockchain to scale horizontally.
3) Networking and peer-to-peer communication
To improve the efficiency and speed of block and transaction propagation, the Firedancer client includes optimized networking and P2P communication protocols. This is accomplished by employing techniques like compression and batching, which can reduce the data that must be transmitted across the network.
4) Enhanced Consensus Protocol
The Firedancer client employs a modified version of the Solana consensus protocol, a proof-of-stake (PoS) protocol that enables validators to participate in the consensus process and earn rewards for their network contributions. Firedancer’s modified version is intended to be more efficient and reliable than the original protocol. Firedancer is an addition to the already active Solana validator client, not a replacement.
Why does 1 million TPS matter?
A blockchain with 1 million TPS will have several unique benefits:
- It enables a blockchain to process many transactions quickly, which is necessary for the industry to reach 1 billion users.
- It improves user experience. A low TPS rate can hinder the user experience because it takes longer to complete a token swap, buy an NFT, or use a dApp.
- It improves security. A high TPS can also make a blockchain less susceptible to spam attacks. Currently, if spam bots request more transactions than the network can handle, it can cause the network to shut down for extended periods, as has been observed with Solana in recent years.
- It unlocks new use cases, like a 24/7 decentralized stock exchange.
After deduplication in a synthetic load test, the Firedancer client processed 1.2 million raw TPS and 600,000 TPS.
LEVEL II : LET’S DIVE DEEPER
1️⃣ Jump Ecosystem
Firedancer may be Jump’s most ambitious project yet. Their goal is to drastically increase Solana’s networking throughput, efficiency, and resiliency over the next 12 months, with a significantly greater result within 24 months.
These are some of Jump’s projects :
A) Cyclone
Zero Knowledge proofs are an important building block for blockchains, providing scalability and privacy.
Efficient proof generation is crucial for their success, and Jump has used their hardware, FPGA design, and crypto expertise to speed up the core components of ZK systems.
Their code is open-source and aimed at contributing to the future of ZK.
- Jump used well-known techniques in elliptic curve cryptography, including the bucket method (Pippenger algorithm) for bucket accumulation, aggregation, and final result. They initially intended to do bucket accumulation in hardware and aggregation on the host, but it was too slow, so they ended up doing bucket aggregation in the FPGA.
- Jump also used extended Twisted Edwards coordinates for elliptic curve arithmetic because BLS12-377 supports them. They incorporated a trick that enables a curve addition with just seven field multiplications, which reduces FPGA resource requirements.
- For field arithmetic at 377 bits, they tried different methods, including using Montgomery representation, CIOS, and various multiplication techniques such as schoolbook, Karatsuba, and Toom-Cook methods.
RESULT 1 : Jump created a fully pipelined elliptic curve adder in extended Twisted Edwards coordinates that operates at 250MHz and uses seven field multipliers, resulting in a 2 orders of magnitude faster addition.
RESULT 2 : Delayed Scheduler
Jump introduced the concept of a scheduler in hardware that reorders the points to maximize the use of the adder pipeline, which is more efficient than processing the points in the exact order given in the input.
The same scheduler can also solve a software problem of implementing MSM with affine coordinates via batch inversion, a well-known technique in theory but has never been successfully implemented in libraries.
B) Silo
When working with large teams of people on blockchains, it’s important to keep digital assets secure. Custody solutions are needed that can be customized to meet the specific needs of different groups and institutions. Project Silo was launched by Jump to create such solutions that are open-source and meet industry standards.
One issue that Silo helps address is the need for increased security for larger teams that want to interact with the DeFi ecosystem. Teams use browser wallets like MetaMask to interact with the Web3, but these wallets can be vulnerable to hacks. Hardware wallets like Ledger can provide added security, but they have limitations such as :
- They are single, physical devices that can be lost, destroyed, or stolen.
- They are challenging to use in a team since only one person can use a Ledger.
- There is no policy checking or multi-approval functionality natively built in.
MetaSilo
Jump created MetaSilo, an air-gapped integration for Silo with browser wallets like MetaMask to mitigate risks and provide flexibility to teams.
- With MetaSilo, teams can use existing browser wallets while still incorporating Silo for secure signing.
- Jump is working on an open-source custody solution that is decentralized, secure, transparent, and convenient. They will release open-source components as they mature and write about them.
2️⃣ The Jump’s Pit
What is the Pit?
Jump says “The Pit is a space for the smartest, most passionate talent out there. In the Pit, devs are the heartbeat, and we have only one mission – to get shit done.”
The Jump’s Pit is a unique and inspiring space designed for web3 development. It’s a community of talented individuals who come together to collaborate and push the boundaries of what’s possible in web3. The Pit provides the necessary tools and resources to support builders of tomorrow, fostering creativity and collaboration. It’s a testament to the Jump’s commitment to supporting the web3 community.
Polygon at the Pit 2023 saw 150+ developers, speakers, and industry leaders come together in Singapore for two consecutive weeks of discussions and workshops surrounding the future of web3.
3️⃣ Jump vs. the Speed of Light
Jump operates globally and to be competitive, it requires a physical presence near each exchange. Unlike traditional supercomputers that are purchased all at once and replaced entirely, Jump’s systems evolve in stages.
New technologies are introduced, existing technologies are maintained, and obsolete technologies are phased out. Misbehaving systems can have severe legal and financial consequences, so Jump’s systems need to be fault-tolerant, have low bug rates, and real-time monitoring to detect errant behaviors.
Traders frequently adopt Jump’s systems for their quick time-to-market and competitive implementations of their strategies. Jump’s systems for production trading and quantitative research now operate at the planetary scale limits of physics and information theory.
Solana is experiencing similar difficulties to Jump, and the second validator, Firedancer, will utilize the latest algorithms, software, hardware, and networking technologies to increase transaction throughput and lower the cost per transaction.
4️⃣ Why Jump and Not Someone Else to Take on Firedancer?
Jump has deep experience scaling networks and building highly performant software systems.
This allows them to apply our comprehensive research and development prowess to solving the scale issue for the Solana blockchain.
As said best by Yakovenko, Co-Founder of Solana,
“By adding more core contributors like Jump Crypto, the network can maintain its status as the best place to build in web3 while scaling to billions of users. I’m excited for Jump’s engineers to bring a new perspective to the network and help improve network resiliency and efficiency.”
5️⃣ Kevin Bowers as a Leader of the Firedancer Project
Kevin Bowers has a background in computational physics and has spent his entire career pushing supercomputers to their limits.
- He dealt creatively with interesting bottlenecks like the speed of light while at Jump Trading.
- This has enabled him and his team to develop high-performance systems emphasizing robustness and resilience.
- His background as a systems architect and scientist at companies such as Jump Trading, D.E. Shaw Research, Los Alamos National Lab, Bell Labs, and Berkeley makes him an ideal candidate to help strengthen Solana’s core infrastructure.
6️⃣ Solana’s Four Historical Outages
- 2021-09-14: 17h outage caused by a transaction flood (peer-to-peer interface).
- 2022-04-30: 7h outage caused by a transaction flood (peer-to-peer interface).
- 2022-06-01: 4.5h outage caused by a chain split bug (execution layer).
- 2022-09-30: 9h outage caused by a fork choice rule bug (consensus layer).
Jump’s final stop on the road to reliability is the consensus and execution layers.
Every public blockchain, including Solana, uses a variation of the following steps.
- Ordering: Pack incoming transactions into blocks.
- Consensus: Determine the canonical fork (attestation/voting on PoS or mining on PoW).
- Execution: Apply each transaction and validate the resulting state.
These steps must happen deterministically across the network.
Simply put, every step must produce the same results on every node. One of the innovations of public blockchains is the ability to independently verify any data without relying on trust assumptions. As a result, any diverging behavior results in a chain split by design.
Fortunately, Solana can avoid network outages. The vulnerability classes are well known, and Jump has sufficient engineering resources to identify and address them. Most chains split bugs were discovered during testing before reaching a live network.
Jump has created extensive testing frameworks for its core trading infrastructure and plans to apply what it has learned to Firedancer.
Firedancer is working to produce specification documents to define the Solana protocol. Ultimately, one should create a Solana validator by looking at the documentation, not the Rust validator code.
Here’s a mind-map of major Solana network components:
LEVEL III : FULL KNOWLEDGE
1️⃣ The Partnerships
Jump has frequently been approached to be a catalyst for nascent platforms and projects as one of the few global firms in the space with robust technology, capital, and an innovative spirit.
- Platforms and projects frequently lacked the liquidity and opportunity set to incentivize early organic participation, but they held a lot of promise in the long run.
- This generally entailed viewing engagement as a partnership rather than a transaction and evaluating the upside over much longer time horizons than a typical trading decision.
- While not entirely natural for a trading group, Jump has been able to use its thesis as a guiding light in evaluating these opportunities. Jump collaborated with several incredible communities to create alignment with the space.
Jump Crypto partners
2️⃣ Community Work
- Terra Governance proposals: Some of the most exciting and successful projects in DeFi use complex economic machinery. Terra’s stability mechanism is an excellent example. Jump has experience analyzing complex systems and can contribute to the discussion on critical market parameters to balance safety and elasticity.
- Solana vesting contract grant: Collaboration is a big part of DeFi’s success. Jump hopes that tools like the vesting contract will help builders focus on unique elements. Bonfida delivered a clean UI, and the product has been audited and is ready for use.
- Wormhole network’s guardian: Wormhole is a decentralized network that lets high-value blockchains communicate with each other. Jump is proud to be one of the guardians that ensure network security by helping tackle cross-chain message passing among chains that can’t run light clients of each other.
- Serum: The Central Limit Order Book is how market interest is coordinated in financial markets. Serum can export that function to Solana and build an orderbook application. Trust in the base layer is the default brick in every blockchain application and should not be overlooked in discussions of composability.
3️⃣ Build
Certus One acquisition – Jump recently brought the exceptionally talented Certus One team on board. Their arrival has supercharged efforts to grow the communities Jump is involved in.
Here are a few of the things they bring, with much more to follow:
- Deep security is built into the Proof of Stake and custody stacks at every level. The team has made significant progress with their Mongon operating system, and the SignOS distributed key management solution is built on top of it.
- Concentrate on validator quality and open development. In this early stage of the evolution of Proof of Stake systems, open, uncensored discussion of the validator and network quality is critical. Discourse that includes the right, transparently tracked metrics at every level of the stack will likely save the industry a lot of trouble in the future. Certus has been one of the most vocal and influential voices in this space, and Jump hopes to assist them in being a beacon of robust and secure decentralization.
- Lido Validator – As one of the select validators behind the staked Eth project, they strongly support the Lido network. Staked PoS assets will play a significant role in the future of composable DeFi, and Lido is a frontrunner in the space.
- Pyth Network – One of the space’s white whales has long been the oracle problem. The blockchain enables the trustless execution of arbitrary pieces of logic on the state of the ledger. One of the major barriers to leveraging this capability for a wide range of applications has been the inability to fruitlessly include off-chain (or so-called real-world) information in this state.
4️⃣ Jump Crypto Investment Categories
- DeFi: 54.5%
- Blockchain Infrastructure: 18.2%
- GameFi: 18.2%
- NFT: 4.5%
- Blockchain Service: 4.5%
5️⃣ Jump Crypto Co-Investors
6️⃣ ‘Jump Crypto Trading Firm Strikes Back: Recovers $140 Million Stolen Crypto from Wormhole Hack’
In early 2022, a hacker stole 120,000 ETH tokens worth more than $321 million from the Wormhole cross-chain bridge, making it the fourth-largest crypto theft ever.
Jump managed to counter-exploit the Wormhole protocol hacker and retrieve $140 million worth of tokens.
A collaborative effort between Jump Crypto and Oasis, which develops multi-signature wallet software, resulted in the recovery of certain assets associated with the Wormhole exploiter’s wallet address.
- While Oasis stated that a Whitehat group assisted the project, blockchain data suggests that the other party could be Jump Crypto, as ownership of the wallets involved in the counter-exploit has been traced back to the company.
- On February 21, Oasis revealed that it received an order from the High Court of England and Wales requiring it to take steps to recover certain stolen assets.
- A Whitehat group reached out to the team with a plan that showed it would be possible to retrieve the assets and provided a Proof of Concept on how it could be achieved.
7️⃣ Chiliz and Jump Crypto
Chiliz, a blockchain-based sports token platform, is launching Chiliz Labs, a $50 million incubator program aimed at early-stage projects in the sports and entertainment industries. Jump Crypto, the investment arm of Jump Trading Group in Chicago, supports the program.
Chiliz Labs will invest in Web3 projects based on Chiliz’s layer 1 blockchain. Chiliz expects to announce between eight and ten projects based on the network soon, according to an email obtained by CoinDesk.
8️⃣ ‘Jump Crypto is the Unnamed Firm That Made $1.28B From Do Kwon’s Doomed Terra Ecosystem?’
When US regulators sued Do Kwon and Terraform Labs for the spectacular implosion of the terraUSD stablecoin and related LUNA token, they left a huge question unanswered:
Who was the trading partner who booked $1.28 billion in profits before Terra’s $40 billion ecosystem collapsed?
According to sources familiar with the situation, it was Chicago-based Jump Crypto.
- Jump Crypto actively participated in the Terra ecosystem, frequently posting governance proposals and investing heavily in the project, including constructing a Terra cross-chain bridge and co-leading a $1 billion capital raise to seed the Luna Foundation Guard.
- Jump Crypto President Kanav Kariya was also a member of the Luna Foundation Guard, which was in charge of Terra’s multibillion-dollar bitcoin reserve treasury.
- According to the SEC complaint, the reserves were depleted in May 2022 in a failed attempt to restore UST’s dollar peg and also siphoned to a Swiss bank account controlled by Kwon.
Summary
Jump Crypto actively participates in developing software infrastructure and tooling for blockchain ecosystems, as well as trading and market-making activities that help make global crypto markets more efficient.
Jump Crypto’s team has been actively creating decentralized, open-source tooling that seeks to solve real-world problems in the public domain for more than six years. Jump’s wealth of systems engineering talent and quantitative acumen has given us a significant head start on Jump’s crypto effort.
However, Jump Crypto is still in its early stages. Still, its developers and community members are energized by the industry’s growth and driven to continue pushing it forward.
Jump’s Firedancer client is a next-generation validator client for the Solana blockchain that aims to improve upon existing validator clients in several ways.
It includes enhanced concurrent transaction processing, sharding support, and other optimizations and enhancements to improve its overall performance and reliability.
While it is still in development, the Firedancer client can be a game-changer for the Solana network.