Report on The Disruption of The Solana Mainnet Beta
At 05:46:16 UTC 2023-02-25, the Solana Mainnet Beta experienced a severe decline in performance which resulted in the validator community deciding to restart the network. The source of this issue is still being investigated.
Validator nodes shifted into vote-only mode, a safety system that revives the network when data is inaccessible, as consensus continued but no user transactions were processed. Even though the reason was unclear at that time, without help from validators, the network could not be restored.
The engineers troubleshooting the problems suggested rebooting the network. The validator group collectively concurred and determined to go back to the prior reliable version, v1.13.6, simultaneously to diminish restart chance, as 1.14 had just been released.
The first attempt to restart the network was cancelled in order to provide more time for a comprehensive review of the data and guarantee no user activity would be adversely affected. After further review, all involved parties agreed to restart the system with an earlier slot than initially planned. There were no confirmed cases of user transactions being reversed or negatively impacted.
On February 26, 2023 at 01:28 UTC, the Solana Mainnet Beta was restarted and resumed processing user transactions without any issues.