Please give a stronger warning: Solana's "degraded performance" causes financial loss

About 19 hours ago, I had liquidations in two Mango accounts as I was trying, retrying, and re-re-re-retrying to make deposits. Because of Solana’s “degraded performance”. In one account, I managed to get in a deposit after many minutes of trying; according to the Mango GUI, the deposit happened 17 seconds after the liquidation. In the other account, I gave up repeatedly retrying deposits after I noticed my coins were gone. Both deposits had already been delayed by transactions needing dozens of retries while I was preparing the funds.

People should be warned about this: Not only will your transactions fail, but a liquidator bot’s will succeed in liquidating you while yours fail, fail, and fail again.

Depositors are covered by insurance against bankrupt accounts. Where is the insurance on the other side, for Solana network breakage outside the borrower’s control? Of course, there is none. But borrowers are depositors, too. The collateral on deposit is always at risk of being dumped, maybe at the very bottom of a dip (or in a flash-crash, as I gather Mango already got hit with). The borrower’s risk includes Solana’s network unreliability preventing the borrower from taking action to protect the collateral.


On a technical level, this problem cuts to the heart of any blockchain’s reason for existence: Establishing the order of transactions. Cryptographically signed notes were a solved problem decades ago. Transaction ordering required a trusted central authority, until Bitcoin was invented. Transaction ordering is a blockchain’s one and only job. Solana’s recognition of this as a distributed database problem of transaction histories is what first drew me in.

If the borrower’s deposit transaction gets in first, his account health is above 0%; and the liquidator’s transaction will fail. If the liquidator’s transaction gets in first, the borrower’s coins were just market-dumped at a terrible price, with a liquidation penalty. If the borrower was frantically trying to make a deposit before the liquidation, and Solana places the liquidator’s transaction first (or drops the borrower’s transaction altogether), then Solana has failed its one and only job.

For some reason, in my case, the liquidator bot’s transactions confirmed, while mined time out repeatedly. I doubt that it is only a matter of luck. My hypothesis: If liquidators run their own high-performance nodes with bots reading their own nodes’ private RPC, and write their transactions via UDP to the current leader with a very tight timeout for retries, they have a huge timing advantage over people who are trying to make deposits through the GUI with the public RPC server. Or it could be very bad luck — twice in a row.

I noticed that the timeout seems to have increased for making transactions through Mango’s GUI. I do not know if this is a Mango issue, or an RPC server issue; and I have not looked through the GH repo for relevant commits. Wherever the change was done, I strongly urge that this should be reverted. If I had been able to retry more frequently, I may have been able to get deposits through before liquidations.


I know that I am taking a risk, doing whatever I’m doing with leverage. I accept that risk that the market price may suddenly plunge while I am not monitoring the situation (and I cloned Mango’s SDK with intent to write a bot for this). What I did not anticipate is the risk that Solana’s network fails exactly the wrong way, at the critical moment; and of course, due to the flood of trading transactions, Solana is most likely to suffer “degraded performance” at precisely the time when the market is crashing hard.

I wrote this up because Solana’s “degraded performance” is no mere annoyance: It caused me a significant loss of money. It could do the same to others. Thus, others should be made aware that such things have happened.

Ironically, what I lost was most of my recently-started investment in the Solana defi ecosystem. On paper, Serum and complementary projects built on Serum are almost exactly what I have been wanting for years! Mango Markets is, in itself, a generally excellent site. But in practice, all these things rely on Solana to be not broken. For users, mainnet breakage equals financial loss. Bitcoin has never failed me; that is why I trust it with real money. Trust is easily damaged. I have lost confidence in Solana — I want it to work, but I now learned the hard way that it doesn’t.

I wish I had more constructive suggestions for how Mango could handle this; but as far as I can see, what it ultimately boils down to is, “the underlying financial network must never, ever do this to people”. I do not blame Mango for what happened to my accounts, but I do blame Solana.

2 Likes

Update with support request:

I essentially got my long-term savings wiped on Mango, with Solana’s so-called “degraded performance” being a highly significant contributory factor.

In January, asset losses from the wrongful liquidations described in OP were part of what got me stuck in a bad position, from which I could not cleanly unwind and exit. Those liquidations were near the local bottom in the market, when I was trying to make deposits to avoid realized losses—the worst kind of liquidations. At that time, after the same Solana-wide failure, Solend partly or fully refunded the liquidation penalties of those who had been affected on their platform; I did not know of this, when I posted OP in this thread.

In May and June, my then-remaining assets in Mango were repeatedly liquidated right before my eyes as Solana ignored my orders to sell smaller amounts, to avoid liquidation. That was after I had spent months pouring in money to deleverage and shore up my position.

My total direct losses from penalties in wrongful liquidations are $3,278.500907 (USDC), as calculated from the numbers listed in “Liquidations” in Mango’s GUI. My indirect losses proximately caused by these Solana network failures are difficult to quantify, but surely much higher.

This excludes other liquidations that clearly fall into the category of “I messed up”. Solana and its popular wallets do not retain information about timed-out transactions; I have done my earnest best to distinguish “I messed up” from “Solana wrecked me”. Needless to say, this figure entirely excludes general losses from adverse market conditions, and from poor trading decisions. I have lost much >$100k in total assets on Mango—thus stating only the order of magnitude.

I despise people who whine about their trading losses. Upon long and careful reflection, and looking to Solend’s example, I realize that I am being unfair to myself.

This has now long been an been ongoing problem with Solana; but new users don’t know that, until they are stuck in a position where it is hard to get out. Vague warnings are given—which ordinary reasonable people will presume refer simply to usual defi risks of unknown bugs or hacks. Solana has a general reputation for unreliability—but those who have not experienced this specific scenario won’t realize just how it plays out, until it hits them repeatedly. Neither Solana nor Solana ecosystem projects have adequately safeguarded users from a systemic problem well-known to developers, which effectively makes liquidation impossible to avoid. If anything, I think that the passage of time with no fix puts the onus much more on projects to protect their users.

Accordingly, I request that Mango should at least refund to me the penalties incurred in wrongful liquidations.

Thanks in advance for doing right by someone who, as a new Solana user, was never warned that Solana habitually fails at the worst possible time: When the market is crashing, and the liquidator bots are closing in. Other than that, Solana works great!


In the foregoing, I omit some relevant details out of concern for my own financial privacy. Although it’s all public on the blockchain and in Mango’s GUI, I am not exactly thrilled at the idea of exposing all this on a public forum, too. I invite moderators or developers to DM me, whereupon I would be happy to provide a list of the relevant Mango accounts.

2 Likes

Sorry to hear about your losses. I think all Mango users have experienced these challenges to some extent and it is frustrating. It’s obvious now Solana should not be trusted for highly levered positions until the network stabilizes but that was not apparent in January.

However, I don’t think Mango should reimburse the liquidator fee. To be clear, that fee goes to liquidators with none of it to Mango. The liquidator fee is critical to ensure the solvency of Mango. Many users were liquidated during this period and there is no ability to discern who was attempting to top it up. Therefore, Mango would be setting a precedent to reimburse all users who were liquidated and this would be a substantial drain on the treasury’s assets.

Perhaps there is a compromise where reimbursements are capped up to some dollar amount per account to help out smaller users. If you want to do an analysis on liquidated accounts and what the totals would be that could be interesting. Still not sure I would support but more likely than a one-off payment that is not replicable for all users.

If you are really committed to seeing this proposal go to a vote then DM me and I can delegate some MNGO to you for initiating the proposal. I suspect the vote would fail though only one way to find out.

Sorry again for your losses.

1 Like

I’m really sorry for this situation, fully aware that you and other users are going through this pain and can assure you that everyone I know, that works on Mango, feels ashamed when they read, what your experience was.

Neither Solana nor Solana ecosystem projects have adequately safeguarded users from a systemic problem well-known to developers, which effectively makes liquidation impossible to avoid. If anything, I think that the passage of time with no fix puts the onus much more on projects to protect their users.

Here is a rough list of what was done to address the issue:

  1. someone added warnings specifically for this scenario to the UI, it mentions liquidations and warns from using high leverage
  2. mango DAO developers contribute to solana’s code base in order to improve network stability
  3. mango DAO rents excessive amounts of RPC servers with built in spamming techniques from triton to give UI users the same level of access as trading bots
  4. reimbursements were discussed but rejected, more on that below
  5. shutting down the UI to protect new users from a bad experience was discussed, but rejected as well

In general it is very hard to verify who tried to save their position, but couldn’t, due to transactions not hitting the chain. It just looks very similar to someone who doesn’t care and just walked away from their position or simply had bad risk management. Repeatedly toping up from CEX might just mean that someone is transferring pnl between two hedged positions.

Agree with brian on what the next steps could be.

4 Likes

Thanks for your responses, @brian_smith_0 and @mschneider. I apologize for the delayed reply. I somehow missed this due to personal circumstance. (Please note that my forum account is unable to send direct messages, as Mr. Smith suggested upthread.)

I like your ideas: A general solution to help others. I’ve been mulling this, trying to think up an optimal approach.

MNGO holders will wonder what’s in it for Mango. I think that bolstering trust and confidence in the Mango platform should be worthwhile, from their perspective. To a financial platform, reputation is an irreplaceable asset; and some best-effort compensation by Mango for Solana problems would surely engender such long-term goodwill as would dwarf the short-term cost. I will write that up in a positive way, if/when it comes to a concrete proposal to put to a vote.

Unfortunately, I lack sufficient knowledge of the centralized parts of Mango to know where to draw data for analysis.

Aside, I code...

A few months ago, I did spend some time perusing the code for Mango’s Rust on-chain program and JavaScript SDK. I spent much more time studying Serum, and Solana itself—mostly the program runtime, but I had some preliminary ideas about what the problem may really be with the TPU. The behavior that I observe as a user “smells” to me like the sort of lock contention issues for which operating system kernels usually have a menagerie of different locking mechanisms; Solana’s r/rw account locking model seems simplistic by comparison. I had thought to dig into that; and I also have some much easier ideas for helping to increase Mango user safety in these scenarios. The losses that I suffered admittedly cooled my earlier excitement about potentially building things, or contributing to development efforts for Mango or Solana.

Is there any documentation of what data Mango logs for the Mango-operated GUI, and how it could be accessed for analysis? I could write a bot to crawl as much Mango transaction data as may be available from Triton. But even if I were to write code for that, I doubt I even have enough disk space here—and the on-chain data would need to be correlated with data on Solana performance hiccups, if Mango logs such issues.

Oracle price data are also important here. Besides the direct relevance of price data, Solana predictably, almost inevitably stalls when BTC makes a big red candlestick. (I should probably mention this in Github, since they only seem to discuss performance stalls on NFT drops.) If performance data are not logged, performance degradation could potentially be inferred from price data. If oracle prices are not logged, they could be reconstructed from oracle update transactions; but that itself would be a development task.

I think that the first concrete objective is to estimate how many wrongful liquidations there may have been, and to develop criteria for some goodwill compensation of penalties by the Mango treasury.

The difficulty of establishing such a claim is one reason why I was reluctant even to inquire. These events have given me some ideas about how timestamped evidence could be logged—but that rather obviates the purpose of a blockchain.

If a user made (a) deleveraging transactions (b) during a severe market crash (c) immediately before or after a liquidation that occurred during a period of severe Solana underperformance—or if a user had a clear pattern of making such transactions—then I would take that as evidence indicating that a liquidation may have been caused by Solana dropping a deleveraging transaction. By “deleveraging transaction”, I mean either a deposit or a sell order, in a sufficient amount to avoid liquidation.[1]

I cannot guess how many or how few people may meet all of these criteria. If I could send you a DM, I would be happy to tell you the addresses of relevant Mango accounts. Even excluding price and network performance data, I think that my publicly visible transaction histories make it obvious in some instances that I should at least have been able to avoid a liquidation penalty. Some of my confirmed transactions would be completely nonsensical, unless I had been struggling to get a transaction through. Please see below for a brief description of one example.

A small note on something that I should probably raise in more of a development context: A liquidator bot is at an extraordinary advantage here, because some types of deleveraging transactions are not idempotent.

Based on observed behavior, I hypothesize that there is a TPU issue where spamming redundant transactions with different signatures may give an advantage over simply firing off many UDP packets with the same transaction. A liquidator bot can always do this, without any significant risk: The Mango program will only allow so much liquidation, with no penalty to the liquidator on error. A user sell order (or in some circumstances, a deposit) cannot be similarly retried.

I got burned several times while waiting multiple minutes between retries to make sure that the Recent Blockhash had timed out. I am aware that Mango was trying to spam the cluster leader and next leader with my transactions—thanks!

I sometimes successfully saved my account by manually putting as many as ten or twenty different deposit transactions (with different signatures) in flight at once: Most would time out, one would succeed, and a few would return program errors due to insufficient wallet funds. In a few instances, I could not do this where redundant deposits would have been financially adverse to me—if funds immediately needed elsewhere could be trapped below the “initial health” threshold. I obviously could not do it when selling to avoid liquidation.

In one instance, I got doubly burned because my sell order confirmed immediately after liquidation: I sold some under pressure, the bot dumped much more, and I was charged a penalty. I had been trying for awhile, awaiting Recent Blockhash timeouts between retries, before my account became eligible for liquidation. (This is one of those instances where my transactions would be bizarre, if I had not started trying to save the account while it was still above “Maintenance Health”.)


The supportiveness shown in this thread indicates to me that maybe I should have gotten involved in Mango much earlier. A trust-minimized, permissionless decentralized exchange with 80% of the user-facing features used by 80% of Binance users is a considerable achievement. Unfortunately, catastrophic platform failures in market crashes are a kind of a dealbreaker for anything that involves debt.

I would still much recommend Mango’s spot interface to people who simply want to buy or sell supported coins, without leverage. Before the negative experiences reported in this thread, I pleasantly used Mango for that purpose myself—including a 1 BTC cash purchase, now all gone with much else.


A note on leverage:

Since “high leverage” was mentioned several times, I should mention that my leverage did not start out so terribly high in the account where I suffered the worst losses. The way that leverage works on a curve, any non-negligible leverage could result in an extreme-risk position very suddenly in the face of black-swan events. That’s what happened to me in May–June. I suffered my largest liquidations in the Terra BTC dump, and then got wiped out almost to zero after the U.S. CPI numbers panicked every market at once. Quite probably, others were hit similarly.

The only way absolutely to avoid such possibilities is never to use any leverage that cannot be fully, immediately backstopped with other assets. Most famously nowadays, MicroStrategy can survive a BTC drop all the way down to zero. But of course, they are not in a situation where an attempt to cover could fail while a robot market-dumps their assets.


  1. I meet all of my own criteria; but I think that the criteria are objectively fair. I also admit that I suffered several liquidations not caused by Solana performance problems; I absolutely would not try to claim anything for those. ↩︎