Nonce collisions
Nonce collisions were another mysterious thing we’ve encountered when trying to scale the number of Geth nodes in order to cover the case when one node crashes. It turns out that
We used a simple load balancer before the three Geth nodes, which was sending each transaction to one of the three nodes. The problem was that each time we submitted many transactions at once, some of those transactions were mysteriously disappearing. It took a day or two until we finally figured out that this was a problem with nonce collisions.
When you are submitting raw transactions to the network you are fine, because you keep track of nonce numbers yourself. In this case you just need a node to publish raw transactions to the network. But in the case you are using an account unlocking mechanism built into the node and do not specify the nonce when publishing transactions (with web3 or so), the node tries to pick the appropriate nonce value itself and then signs a transaction.
Because of the network delays, in the case two nodes receive the same transaction publishing request, they can generate the same nonce value. At the moment of receiving the transaction publishing request they don’t know that they both received a transaction with the same nonce. Thus, when propagating these transactions through the network, one of them will eventually be dropped because its “transaction nonce is too low”.