Real-time WS market data API for Mango

Hi, I’d like to apply for a grant for a CEX like real-time market data API server for Mango, it would offer:

  • familiar experience for centralized exchanges APIs users
    • WebSocket API with Pub/Sub flow - subscribe to selected channels and markets and receive real-time data as easy to parse JSON messages that can be consumed from any language supporting WebSocket protocol
    • incremental L2 order book updates - instead of decoding Mango market asks and bids accounts for each account change in order to detect order book updates, receive initial L2 snapshot and incremental updates as JSON messages real-time over WebSocket connection
    • tick-by-tick trades - instead of decoding eventQueue account data which is quite large (> 1MB) and in practice it’s hard to consume real-time directly from Solana RPC node due to it’s size, receive individual trade messages real-time over WebSocket connection
    • real-time L3 data - receive the most granular updates on individual order level: open, change, fill, done for every order that Mango DEX processes
  • decreased load and bandwidth consumption for Solana RPC nodes hosts - by providing real-time market data API server instead of RPC node directly, hosts can decrease substantially both CPU load and bandwidth requirements as only this API server will be direct consumer of RPC API when it comes to market data accounts changes and will efficiently normalize and broadcast small JSON messages to all connected clients

I’ve built serum-vial (GitHub - tardis-dev/serum-vial: Real-time WebSocket market data API for Serum) for Serum spot markets and I’m pretty sure similar project can be done for Mango DEX and would be very useful for Mango API users.

I know there is mango-vial (GitHub - microwavedcola1/mango-vial: Real-time WebSocket market data API for Mango Markets v3) already which is a copy of serum-vial, but I’m proposing full implementation (not only L2 data) and ongoing support, bug-fixing and running demo/community server with this API.

I’d like to also add support for WS based feeds for liquidation data (I think it can be parsed from eventQueue account for a given market) as well as funding rates etc if possible.

For compensation I propose $60k in locked (for a year?) MNGO tokens (~180k tokens), project would be completed most likely by the end of this year (end of Jan 2022 in worst case).

7 Likes

I think this makes a lot of sense, specially the ongoing bugfixing, maintenance and hosting a live version to play with . I have also been asked by a couple of people who used mango-vial about liquidation, L3, and trade data. I think the proposal covers all of those points. Also serum-vial from which mango-vial has been forked is an excellent codebase. I support this proposal/author.

One nice-to-have from my side would be a python client. Most traders I have talked to use python (and I have heard 1-2 use scala) and would have liked to integrate mango-vial into there existing codebases.

Side note: my gut feeling is that for anybody who trades serious money is more interested in the intention (or information they don’t have) of the rest of the traders and live orders are sufficient to observe this, and less so in executed trades. Hence the original purpose of mango-vial was to focus on the orders and not on trades, but I could be wrong.

We did have great success in mimicking the FTX API, would it be possible to make both serum-vial as well as mango-vial to replicate the behaviour as close as possible, to the point where people can easily switch without changing code?

Surprisingly, a lot of institutional traders have requested trades because their systems/backoffice requires it.

Please do this. You have my vote. Many people like serum-vial and would greatly appreciate this.

What do you plan to do with the codebase that microwavedcola already produced? Is it insufficient? I don’t know as much about it.

2 Likes

@mschneider you mean mimicking FTX WebSocket API for market data? I’m not convinced tbh, their API is ok, but not really state of the art, serum-vial provides full order book data (not only top 100 levels as FTX), L3 data etc and JSON message format is flat (so easy to write deserializers for it for compiled languages).
Idea if having the same API contract for FTX and Mango is great, but I’d say it would be a leaky abstraction there’s just too much of a difference how those exchanges work.

@daffy I plan to start from scratch (from serum-vial a a starting point), most likely won’t use microwavedcola code directly, but definitely will review it and see what could be reused.

1 Like

I think this would be a valuable add. I support you doing this.

After how many requests we get for this, my only hesitation is that end of Jan 2022 is too far into the future. Something like FTX api with the standard docs website that every exchange uses would be extremely valuable. Obviously all functions won’t and can’t mirror the CEX, but it should be close enough. Not because those CEXes have the best APIs but because that’s what would make it easy for new professional traders to integrate into mango.

In my case I would like to be able to record trades to calibrate my market making simulation engine and to get some metrics like the expected toxicity of the flow. On top of this you might want to monitor toxicity in real time and adjust quotes accordingly.

I’ll do my best to have basic version fully working in December, then will add mango specific features to it like WebSocket liquidations info based on event queue updates & instrument info like funding rates etc.

3 Likes

First version is ready at GitHub - tardis-dev/mango-bowl

2 Likes