Please give a stronger warning: Solana's "degraded performance" causes financial loss

About 19 hours ago, I had liquidations in two Mango accounts as I was trying, retrying, and re-re-re-retrying to make deposits. Because of Solana’s “degraded performance”. In one account, I managed to get in a deposit after many minutes of trying; according to the Mango GUI, the deposit happened 17 seconds after the liquidation. In the other account, I gave up repeatedly retrying deposits after I noticed my coins were gone. Both deposits had already been delayed by transactions needing dozens of retries while I was preparing the funds.

People should be warned about this: Not only will your transactions fail, but a liquidator bot’s will succeed in liquidating you while yours fail, fail, and fail again.

Depositors are covered by insurance against bankrupt accounts. Where is the insurance on the other side, for Solana network breakage outside the borrower’s control? Of course, there is none. But borrowers are depositors, too. The collateral on deposit is always at risk of being dumped, maybe at the very bottom of a dip (or in a flash-crash, as I gather Mango already got hit with). The borrower’s risk includes Solana’s network unreliability preventing the borrower from taking action to protect the collateral.


On a technical level, this problem cuts to the heart of any blockchain’s reason for existence: Establishing the order of transactions. Cryptographically signed notes were a solved problem decades ago. Transaction ordering required a trusted central authority, until Bitcoin was invented. Transaction ordering is a blockchain’s one and only job. Solana’s recognition of this as a distributed database problem of transaction histories is what first drew me in.

If the borrower’s deposit transaction gets in first, his account health is above 0%; and the liquidator’s transaction will fail. If the liquidator’s transaction gets in first, the borrower’s coins were just market-dumped at a terrible price, with a liquidation penalty. If the borrower was frantically trying to make a deposit before the liquidation, and Solana places the liquidator’s transaction first (or drops the borrower’s transaction altogether), then Solana has failed its one and only job.

For some reason, in my case, the liquidator bot’s transactions confirmed, while mined time out repeatedly. I doubt that it is only a matter of luck. My hypothesis: If liquidators run their own high-performance nodes with bots reading their own nodes’ private RPC, and write their transactions via UDP to the current leader with a very tight timeout for retries, they have a huge timing advantage over people who are trying to make deposits through the GUI with the public RPC server. Or it could be very bad luck — twice in a row.

I noticed that the timeout seems to have increased for making transactions through Mango’s GUI. I do not know if this is a Mango issue, or an RPC server issue; and I have not looked through the GH repo for relevant commits. Wherever the change was done, I strongly urge that this should be reverted. If I had been able to retry more frequently, I may have been able to get deposits through before liquidations.


I know that I am taking a risk, doing whatever I’m doing with leverage. I accept that risk that the market price may suddenly plunge while I am not monitoring the situation (and I cloned Mango’s SDK with intent to write a bot for this). What I did not anticipate is the risk that Solana’s network fails exactly the wrong way, at the critical moment; and of course, due to the flood of trading transactions, Solana is most likely to suffer “degraded performance” at precisely the time when the market is crashing hard.

I wrote this up because Solana’s “degraded performance” is no mere annoyance: It caused me a significant loss of money. It could do the same to others. Thus, others should be made aware that such things have happened.

Ironically, what I lost was most of my recently-started investment in the Solana defi ecosystem. On paper, Serum and complementary projects built on Serum are almost exactly what I have been wanting for years! Mango Markets is, in itself, a generally excellent site. But in practice, all these things rely on Solana to be not broken. For users, mainnet breakage equals financial loss. Bitcoin has never failed me; that is why I trust it with real money. Trust is easily damaged. I have lost confidence in Solana — I want it to work, but I now learned the hard way that it doesn’t.

I wish I had more constructive suggestions for how Mango could handle this; but as far as I can see, what it ultimately boils down to is, “the underlying financial network must never, ever do this to people”. I do not blame Mango for what happened to my accounts, but I do blame Solana.

2 Likes

Update with support request:

I essentially got my long-term savings wiped on Mango, with Solana’s so-called “degraded performance” being a highly significant contributory factor.

In January, asset losses from the wrongful liquidations described in OP were part of what got me stuck in a bad position, from which I could not cleanly unwind and exit. Those liquidations were near the local bottom in the market, when I was trying to make deposits to avoid realized losses—the worst kind of liquidations. At that time, after the same Solana-wide failure, Solend partly or fully refunded the liquidation penalties of those who had been affected on their platform; I did not know of this, when I posted OP in this thread.

In May and June, my then-remaining assets in Mango were repeatedly liquidated right before my eyes as Solana ignored my orders to sell smaller amounts, to avoid liquidation. That was after I had spent months pouring in money to deleverage and shore up my position.

My total direct losses from penalties in wrongful liquidations are $3,278.500907 (USDC), as calculated from the numbers listed in “Liquidations” in Mango’s GUI. My indirect losses proximately caused by these Solana network failures are difficult to quantify, but surely much higher.

This excludes other liquidations that clearly fall into the category of “I messed up”. Solana and its popular wallets do not retain information about timed-out transactions; I have done my earnest best to distinguish “I messed up” from “Solana wrecked me”. Needless to say, this figure entirely excludes general losses from adverse market conditions, and from poor trading decisions. I have lost much >$100k in total assets on Mango—thus stating only the order of magnitude.

I despise people who whine about their trading losses. Upon long and careful reflection, and looking to Solend’s example, I realize that I am being unfair to myself.

This has now long been an been ongoing problem with Solana; but new users don’t know that, until they are stuck in a position where it is hard to get out. Vague warnings are given—which ordinary reasonable people will presume refer simply to usual defi risks of unknown bugs or hacks. Solana has a general reputation for unreliability—but those who have not experienced this specific scenario won’t realize just how it plays out, until it hits them repeatedly. Neither Solana nor Solana ecosystem projects have adequately safeguarded users from a systemic problem well-known to developers, which effectively makes liquidation impossible to avoid. If anything, I think that the passage of time with no fix puts the onus much more on projects to protect their users.

Accordingly, I request that Mango should at least refund to me the penalties incurred in wrongful liquidations.

Thanks in advance for doing right by someone who, as a new Solana user, was never warned that Solana habitually fails at the worst possible time: When the market is crashing, and the liquidator bots are closing in. Other than that, Solana works great!


In the foregoing, I omit some relevant details out of concern for my own financial privacy. Although it’s all public on the blockchain and in Mango’s GUI, I am not exactly thrilled at the idea of exposing all this on a public forum, too. I invite moderators or developers to DM me, whereupon I would be happy to provide a list of the relevant Mango accounts.

2 Likes

Sorry to hear about your losses. I think all Mango users have experienced these challenges to some extent and it is frustrating. It’s obvious now Solana should not be trusted for highly levered positions until the network stabilizes but that was not apparent in January.

However, I don’t think Mango should reimburse the liquidator fee. To be clear, that fee goes to liquidators with none of it to Mango. The liquidator fee is critical to ensure the solvency of Mango. Many users were liquidated during this period and there is no ability to discern who was attempting to top it up. Therefore, Mango would be setting a precedent to reimburse all users who were liquidated and this would be a substantial drain on the treasury’s assets.

Perhaps there is a compromise where reimbursements are capped up to some dollar amount per account to help out smaller users. If you want to do an analysis on liquidated accounts and what the totals would be that could be interesting. Still not sure I would support but more likely than a one-off payment that is not replicable for all users.

If you are really committed to seeing this proposal go to a vote then DM me and I can delegate some MNGO to you for initiating the proposal. I suspect the vote would fail though only one way to find out.

Sorry again for your losses.

I’m really sorry for this situation, fully aware that you and other users are going through this pain and can assure you that everyone I know, that works on Mango, feels ashamed when they read, what your experience was.

Neither Solana nor Solana ecosystem projects have adequately safeguarded users from a systemic problem well-known to developers, which effectively makes liquidation impossible to avoid. If anything, I think that the passage of time with no fix puts the onus much more on projects to protect their users.

Here is a rough list of what was done to address the issue:

  1. someone added warnings specifically for this scenario to the UI, it mentions liquidations and warns from using high leverage
  2. mango DAO developers contribute to solana’s code base in order to improve network stability
  3. mango DAO rents excessive amounts of RPC servers with built in spamming techniques from triton to give UI users the same level of access as trading bots
  4. reimbursements were discussed but rejected, more on that below
  5. shutting down the UI to protect new users from a bad experience was discussed, but rejected as well

In general it is very hard to verify who tried to save their position, but couldn’t, due to transactions not hitting the chain. It just looks very similar to someone who doesn’t care and just walked away from their position or simply had bad risk management. Repeatedly toping up from CEX might just mean that someone is transferring pnl between two hedged positions.

Agree with brian on what the next steps could be.

2 Likes