Last night BitPay's servers started
huffing and puffing. At first we thought we might have had a database
corruption issue, but on further inspection we realized that one of our Bitcoin
nodes seemed to be out of sync with the rest of the network. It had gone
into what is called "safe mode" and was refusing to provide details
about transactions. This was affecting real time notifications of
transactions as purchases were made.
We started looking around on IRC and
forums and discovered that others had noticed problems as well. This led
us to conclude that it was not likely an issue specific to Bitpay (likely not
data corruption). Several of the developers in the #bitcoin-dev chatroom
soon realized that the issue was due to pre 0.8 versions of Bitcoin-QT not
recognizing recently mined blocks (due to a bug in the pre 0.8 releases).
This meant that the block chain had forked. Bitcoin-QT 0.8 nodes
and pre 0.8 nodes we're on a different version of the block chain.
Although it does make the network more vulnerable to a double spend attack, this actually isn't all that bad to the average user because their transactions would most likely be confirmed in both forks, however it's very bad news if you're a miner. Ultimately one of the forks would survive and the other one would die out. If you're a miner on the fork that dies out, it means your reward for finding the blocks will be worthless. Miners needed to quickly decide which fork to support and then all get on the same page. They decided to direct all their mining efforts on the pre 0.8 fork.
This decision was made because the alternative would have required everyone to upgrade to 0.8 or for someone to develop a patch that fixed the pre 0.8 software. Mining on 0.8 would be more risky by the fact that many people might not choose to upgrade, creating a more long lasting fork where the ultimate winner is not as clear.
Although it does make the network more vulnerable to a double spend attack, this actually isn't all that bad to the average user because their transactions would most likely be confirmed in both forks, however it's very bad news if you're a miner. Ultimately one of the forks would survive and the other one would die out. If you're a miner on the fork that dies out, it means your reward for finding the blocks will be worthless. Miners needed to quickly decide which fork to support and then all get on the same page. They decided to direct all their mining efforts on the pre 0.8 fork.
This decision was made because the alternative would have required everyone to upgrade to 0.8 or for someone to develop a patch that fixed the pre 0.8 software. Mining on 0.8 would be more risky by the fact that many people might not choose to upgrade, creating a more long lasting fork where the ultimate winner is not as clear.
Once we ascertained that the pre 0.8 fork
would survive, we took the affected Bitpay node out of safe mode and full
service was restored.
- Customers that placed orders during the time of the network issues may have noticed that their transactions were not immediately recognized. These transactions will eventually get credited to those invoices and confirmed by the network.
- If you're a merchant, you may have received warnings about unconfirmed invoices, those invoices will eventually be confirmed and credited to your account.
- We've checked all payments into BitPay and at this point there are no instances where a transaction has failed to confirm due to this incident.
If you would like more technical details
on the issue, see the following page at bitcoin.org:
http://bitcoin.org/chainfork.html
Kudos to the dev team, miners and pools
operators for their quick diagnosis and correction of the problem. It is
actually a great demonstration of how Bitcoin is more than just the software
and the protocol it implements. It's also the network of users, miners,
developers and service providers that can quickly react to situations such as
this to preserve the integrity of the system.
