About 19 hours ago, I had liquidations in two Mango accounts as I was trying, retrying, and re-re-re-retrying to make deposits. Because of Solana’s “degraded performance”. In one account, I managed to get in a deposit after many minutes of trying; according to the Mango GUI, the deposit happened 17 seconds after the liquidation. In the other account, I gave up repeatedly retrying deposits after I noticed my coins were gone. Both deposits had already been delayed by transactions needing dozens of retries while I was preparing the funds.
People should be warned about this: Not only will your transactions fail, but a liquidator bot’s will succeed in liquidating you while yours fail, fail, and fail again.
Depositors are covered by insurance against bankrupt accounts. Where is the insurance on the other side, for Solana network breakage outside the borrower’s control? Of course, there is none. But borrowers are depositors, too. The collateral on deposit is always at risk of being dumped, maybe at the very bottom of a dip (or in a flash-crash, as I gather Mango already got hit with). The borrower’s risk includes Solana’s network unreliability preventing the borrower from taking action to protect the collateral.
On a technical level, this problem cuts to the heart of any blockchain’s reason for existence: Establishing the order of transactions. Cryptographically signed notes were a solved problem decades ago. Transaction ordering required a trusted central authority, until Bitcoin was invented. Transaction ordering is a blockchain’s one and only job. Solana’s recognition of this as a distributed database problem of transaction histories is what first drew me in.
If the borrower’s deposit transaction gets in first, his account health is above 0%; and the liquidator’s transaction will fail. If the liquidator’s transaction gets in first, the borrower’s coins were just market-dumped at a terrible price, with a liquidation penalty. If the borrower was frantically trying to make a deposit before the liquidation, and Solana places the liquidator’s transaction first (or drops the borrower’s transaction altogether), then Solana has failed its one and only job.
For some reason, in my case, the liquidator bot’s transactions confirmed, while mined time out repeatedly. I doubt that it is only a matter of luck. My hypothesis: If liquidators run their own high-performance nodes with bots reading their own nodes’ private RPC, and write their transactions via UDP to the current leader with a very tight timeout for retries, they have a huge timing advantage over people who are trying to make deposits through the GUI with the public RPC server. Or it could be very bad luck — twice in a row.
I noticed that the timeout seems to have increased for making transactions through Mango’s GUI. I do not know if this is a Mango issue, or an RPC server issue; and I have not looked through the GH repo for relevant commits. Wherever the change was done, I strongly urge that this should be reverted. If I had been able to retry more frequently, I may have been able to get deposits through before liquidations.
I know that I am taking a risk, doing whatever I’m doing with leverage. I accept that risk that the market price may suddenly plunge while I am not monitoring the situation (and I cloned Mango’s SDK with intent to write a bot for this). What I did not anticipate is the risk that Solana’s network fails exactly the wrong way, at the critical moment; and of course, due to the flood of trading transactions, Solana is most likely to suffer “degraded performance” at precisely the time when the market is crashing hard.
I wrote this up because Solana’s “degraded performance” is no mere annoyance: It caused me a significant loss of money. It could do the same to others. Thus, others should be made aware that such things have happened.
Ironically, what I lost was most of my recently-started investment in the Solana defi ecosystem. On paper, Serum and complementary projects built on Serum are almost exactly what I have been wanting for years! Mango Markets is, in itself, a generally excellent site. But in practice, all these things rely on Solana to be not broken. For users, mainnet breakage equals financial loss. Bitcoin has never failed me; that is why I trust it with real money. Trust is easily damaged. I have lost confidence in Solana — I want it to work, but I now learned the hard way that it doesn’t.
I wish I had more constructive suggestions for how Mango could handle this; but as far as I can see, what it ultimately boils down to is, “the underlying financial network must never, ever do this to people”. I do not blame Mango for what happened to my accounts, but I do blame Solana.