- The network has had an issue affecting one of its two Polygon PoS chain layers, Heimdall
- Polygon’s deployment on OpenSea was affected, making users unable to complete NFT transactions
Speed and cost-focused Ethereum layer two scaling solution Polygon overnight suffered a recent outage. The network was down for more than 13 hours, leaving users worried and wondering why it was taking the developers so long to fix the said bug.
The Issue
The Polygon team informed users yesterday at 4.20 pm UTC that it anticipated the network would be down starting around 5:50 pm UTC, as one of its three consensus layers needed maintenance. It noted that the proof of stake users would be the casualties.
Also, since the last time Polygon had a major bug, $24 billion of user funds were left at risk, the Polygon team was keen to assure the community that all user holdings remained safe.
“The team is currently looking into an issue with the Tendermint implementation used by one of the two layers of Polygon PoS chain. All user funds and data remain fully secure, but there is a potential threat of Polygon PoS users experiencing downtime sporadically,” Polygon developers wrote on Twitter.
The Cause
The Polygon developers said that the issue had resulted from a recent update affecting Heimdall, a PoS verified layer on the network. Heimdall is essentially a series of PoS nodes that participate in network consensus, and in this case, a bug had plagued Heimdall validators in effect, causing them to start employing different chain versions.
“We suspect that there may have been a bug in the update that affected the consensus and caused different Heimdall validators to be on different versions of the chain, thus not reaching a 2/3 consensus,” the developers said.
As anticipated, the network was down by 5:55 pm UTC, but as time went by and Polygon didn’t get back up, users became unsettled. Words of worry spread across the community, with users then saying the downtime had them feel more than just a pinch.
Action
Polygon didn’t bring an update until at least 7 hours after the network went down. The 1.45 am update briefed users that developers were issuing a temporary hotfix as the team worked on a long-term solution. Even then, the ‘fix’ took longer to deploy. Polygonscan data shows that the network is back up with new validated blocks.