HangukQuant Research

HangukQuant Research

Share this post

HangukQuant Research
HangukQuant Research
Clock + Event-based market making + testing
Copy link
Facebook
Email
Notes
More

Clock + Event-based market making + testing

HangukQuant's avatar
HangukQuant
Oct 09, 2024
∙ Paid
4

Share this post

HangukQuant Research
HangukQuant Research
Clock + Event-based market making + testing
Copy link
Facebook
Email
Notes
More
Share

A few days ago we added improvements to the market maker to fix bugs and allow for clock-based triggering of order actions.

Big improvements to the market-maker backtests

HangukQuant
·
October 6, 2024
Big improvements to the market-maker backtests

Read full story

With this addition, we are able to both trade AND test with the same code base strategies that involve tick data. Together with the simulator (Alpha, GeneticAlpha) suite and our new hft (feed, oms, mock) modules, we are able to trade on exchanges working with both OHLCV bars and tick data.

This post will be a demonstration of the tick data features and graphics that we have added. The full code is provided.

quantpylib is our community github repo for annual readers:

HangukQuant Community Github Repo

HangukQuant Community Github Repo

HangukQuant
·
November 24, 2023
Read full story

You can get the repo pass here, alternatively:

https://hangukquant.thinkific.com/courses/quantpylib

Moving on, we will go right into the demo. We will demonstrate the multi-exchange compatibility and portability of our logic. There is no alpha in the quoter - it is for illustrative purposes only.

Typically, a market-maker needs to be able to track their portfolio states: order states (pending/open/cancelled/rejected), open positions, account equity/balance and so on. In general, a market maker action triggers include but are not limited to internal clock cycles, trade arrival, orderbook delta updates pushed and a number of variable proprietary logic. We may choose to act on these data immediately upon arrival (a onTick behavior) or store it in some shared state that is later used to compute optimal market quotes. We will explore all of these options.

Let us make some imports:

import os 
import asyncio

from pprint import pprint
from decimal import Decimal
from dotenv import load_dotenv
load_dotenv()

import quantpylib.standards.markets as markets

from quantpylib.hft.oms import OMS
from quantpylib.hft.feed import Feed
from quantpylib.gateway.master import Gateway 
from quantpylib.utilities.general import _time
from quantpylib.utilities.general import save_pickle, load_pickle

exchanges = ['bybit','hyperliquid']
configs = {
    "binance" : {
        "tickers":["CELOUSDT"],
    },
    "bybit" : {
        "tickers":["CELOUSDT","AERGOUSDT","DYDXUSDT"],
    },
    "hyperliquid" : {
        "tickers":["GALA","RDNT","DYDX"],
    },
}

config_keys = {
    "binance" : {
        "key":os.getenv('BIN_KEY'),
        "secret":os.getenv('BIN_SECRET'),    
    },
    "bybit" : {
        "key":os.getenv('BYBIT_KEY'),
        "secret":os.getenv('BYBIT_SECRET'),    
    },
    "hyperliquid" : {
        "key":os.getenv('HPL_KEY'),
        "secret":os.getenv('HPL_SECRET'),    
    },
}

gateway = Gateway(config_keys=config_keys)
buffer_size = 10000
l2_feeds = {exc : {} for exc in exchanges}
trade_feeds = {exc : {} for exc in exchanges}
order_value = 50

Define the keys for the exchanges you would like to integrate - in our demo, we would use bybit and hyperliquid. The keys are passed into the gateway. For demonstration, we will use a fixed order size of fifty dollars.

The gateway is the connector to the relevant exchanges - which is passed into the oms and feed objects. The oms handles order/position tracking, recovery, execution and auxiliary tasks. The feed does tick data subscription, storing, and retrieval.

We can instantiate the feed and oms and set up how long we want to run our quotes:

oms = OMS(gateway=gateway)
feed = Feed(gateway=gateway)
play = lambda : asyncio.sleep(60 * 10 * 3)
time = lambda : _time()

For clock based trading agents, we can register a callback to the oms - this callback coroutine is called every specified interval. This could be as simple as 'submit/cancel quotes every 500ms' and so on. We will demonstrate an event-based quoter for now, but for demonstration - we will just print the exchange balances every 5 seconds - no action is taken. The actual quoter is done in a make coroutine that runs asynchronously for each ticker in each exchange:

async def clock_event():
    pprint(await oms.get_all_balances())

async def main():
    await gateway.init_clients()
    await oms.init()

    await oms.add_clock_callback(
        callback=clock_event,
        interval_ms=5000
    )

    quote_coros = []
    for exchange in exchanges:
        for ticker in configs[exchange]["tickers"]:
            quote_coros.append(make(exc=exchange,ticker=ticker))
    await asyncio.gather(*quote_coros)

    #save data after finish quoting (can ignore), cleanup
    l2_data = {
        exc:{ticker:lob.buffer_as_list() for ticker,lob in l2_feed.items()}
        for exc,l2_feed in l2_feeds.items()
    }
    trade_data = {
        exc:{ticker:trades.get_buffer() for ticker,trades in trade_feed.items()}
        for exc,trade_feed in trade_feeds.items()
    }
    save_pickle('hft_data.pickle',(l2_data,trade_data))
    await oms.cleanup()

async def make(exc,ticker):
    #implement maker logic 
    return 

if __name__ == "__main__":
    asyncio.run(main())

Since we called the make function, we have to implement the maker logic. First, we will get a trade feed with no handler (this is just done for show, we won't use the trade data here). We will, however, register a handler for the orderbook ticks:

async def make(exc,ticker):
    trade_feed = await feed.add_trades_feed(
        exc=exc,
        ticker=ticker,
        buffer=buffer_size,
        handler=None
    )
    live_orders = oms.orders_peek(exc=exc)

    async def l2_handler(lob):
        #this is called on orderbook tick
        #lob is a quantpylib.hft.lob.LOB object
        #submit actions on book depth stream
        return

    l2_feed = await feed.add_l2_book_feed(
        exc=exc,
        ticker=ticker,
        handler=l2_handler,
        buffer=buffer_size
    )

    trade_feeds[exc][ticker] = feed.get_feed(trade_feed)
    l2_feeds[exc][ticker] = feed.get_feed(l2_feed)
    await play()

The l2_handler receives a lob object which is a quantpylib.hft.lob.LOB object - we can obtain either the live buffer from this object or statistics such as mid, vamp indicators and vol.

The oms does order tracking and maintains live_orders which is a quantpylib.standards.portfolio.Orders object. This is achieved via the gateway's underlying socket connections and requests. Note that the oms itself is a separate functionality provided by quantpylib, and is able to do a variety of useful things - such as registering coroutine handlers for order updates, position fills and so on - see examples here.

In this section, we will not register any order/position update handlers, and just rely on the live updation of our orders which is intitated by default on oms.init(). Say, inside the l2_handler we would like to submit/cancel orders using the following logic:

1. Determine the price we want to quote, say the third from top of book. Fix order value at 50.0.
2. If there is no pending (submitted and unacknowledged) bid or ask, or existing orders that are tighter than price at step 1 - submit an order at determined price.
3. If there are acknowledged orders that are tighter than the determined levels, cancel them. If there are acknowledged orders that are further from top of book than the determined levels, let them sit in the order book to maintain price-time priority.
4. If there are more than 5 resting orders on each side, cancel excess orders starting from lowest price priority.

The above logic tries to pick up taker orders that slam a thin orderbook through multiple levels. Obviously, the feasibility of this depends on the market, our risk management, and whether a mean-reversionary effect exists upon said price impact, and this effect relative to costs/adverse fills. The logic tries to maintain time priority for duplicate levels. We make no comment or assertions on the viability of said 'rules'.

For specifics on how to pass the parameters to oms methods, refer to documentation and examples. gateway documentation and examples should be helpful.

    async def l2_handler(lob):
        mid = lob.get_mid()
        inventory = float(oms.get_position(exc=exc,ticker=ticker)) * mid
        bid_price = lob.bids[2,0]
        ask_price = lob.asks[2,0]
        order_bids = live_orders.get_bid_orders(ticker=ticker) 
        order_asks = live_orders.get_ask_orders(ticker=ticker)
        any_pending_bid = any(order.ord_status == markets.ORDER_STATUS_PENDING for order in order_bids)
        any_pending_ask = any(order.ord_status == markets.ORDER_STATUS_PENDING for order in order_asks)

        any_tight_bid = any(order.price is not None and order.price >= Decimal(str(bid_price)) for order in order_bids)
        any_tight_ask = any(order.price is not None and order.price <= Decimal(str(ask_price)) for order in order_asks)

        orders = []

        if not any_tight_bid and not any_pending_bid:
            orders.append({
                "exc":exc,
                "ticker":ticker,
                "amount":order_value/lob.get_mid(),
                "price":bid_price,
                "round_to_specs":True,
            })
        if not any_tight_ask and not any_pending_ask:
            orders.append({
                "exc":exc,
                "ticker":ticker,
                "amount":order_value/lob.get_mid() * -1,
                "price":ask_price,
                "round_to_specs":True,
            })

        ack_bids = [order for order in order_bids if order.ord_status != markets.ORDER_STATUS_PENDING] 
        ack_asks = [order for order in order_asks if order.ord_status != markets.ORDER_STATUS_PENDING] 

        cancel_bids = [order for order in ack_bids if order.price > Decimal(str(bid_price))]
        cancel_asks = [order for order in ack_asks if order.price < Decimal(str(ask_price))]
        cancels = cancel_bids + cancel_asks
        cancels += ack_bids[5 + len(cancel_bids):] 
        cancels += ack_asks[5 + len(cancel_asks):]
        cancels = [{
            "exc":order.exc,
            "ticker":order.ticker,
            "oid":order.oid
        } for order in cancels]

        if orders:
            await asyncio.gather(*[
                oms.limit_order(**order) for order in orders
            ])
        if cancels:
            await asyncio.gather(*[
                oms.cancel_order(**cancel) for cancel in cancels
            ])

On the web-platforms of selected exchanges, we should see the quotes submitted - here is bybit:

alt text

We see our tightest quotes are third from mid, with deeper levels sitting in the order book. A price jump from taker order 0.1088 to 0.1085 hit our bid.

Backtesting

In this section, we show how to backtest using the quantpylib.hft.feed.Feed and quantpylib.hft.oms.OMS objects. As pre-requisite: please read the section on using the Feed and OMS modules, as well as using the section on using them in our market making demo.

Minimal code change is required. In fact, from the market-making demo, all we change is:

simulated = True
if simulated:
    from quantpylib.hft.mocks import Replayer, Latencies
    (l2_data,trade_data) = load_pickle('hft_data.pickle')
    replayer = Replayer(
        l2_data = l2_data,
        trade_data = trade_data,
        gateway=gateway
    )
    oms = replayer.get_oms()
    feed = replayer.get_feed()
    play = lambda : replayer.play()
    time = lambda : replayer.time()
else:
    oms = OMS(gateway=gateway)
    feed = Feed(gateway=gateway)
    play = lambda : asyncio.sleep(60 * 10 * 3)
    time = lambda : _time()

The quantpylib.hft.mocks.Replayer class provides mock classes for the OMS and Feed that behaves like the actual trading agents. This Replayer simulates agents involved in trading, such as public/private feed latencies, order submissions, matching and more. We don't need anything else, when the await play() is called, the backtest is run. Note that we can pass in a number of optional paramters, for example:

exchange_fees = {
    "bybit": {
        "maker":0.0002,
        "taker":0.0005
    },
    "hyperliquid": {
        "maker":0.0001,
        "taker":0.0003
    }
}
exchange_latencies = {
    "bybit": {
        Latencies.REQ_PUBLIC:100,
        Latencies.REQ_PRIVATE:50,
        Latencies.ACK_PUBLIC:100,
        Latencies.ACK_PRIVATE:50,
        Latencies.FEED_PUBLIC:100,
        Latencies.FEED_PRIVATE:50,
    },
    "hyperliquid": {
        Latencies.REQ_PUBLIC:200,
        Latencies.REQ_PRIVATE:150,
        Latencies.ACK_PUBLIC:200,
        Latencies.ACK_PRIVATE:150,
        Latencies.FEED_PUBLIC:200,
        Latencies.FEED_PRIVATE:150,
    }
}
replayer = Replayer(
    l2_data = l2_data,
    trade_data = trade_data,
    gateway=gateway,
    exchange_fees=exchange_fees,
    exchange_latencies=exchange_latencies
)

The data format looks like this (data pickled from the demo run above):

print(l2_data.keys())
print(l2_data['bybit'].keys())
print(l2_data['bybit']['CELOUSDT'][0])
dict_keys(['bybit', 'hyperliquid'])
dict_keys(['CELOUSDT', 'AERGOUSDT', 'DYDXUSDT'])
{'ts': 1728232861170, 
'b': array([[7.46200e-01, 5.80100e+02],
       [7.46100e-01, 7.03100e+02],
       ...
       [7.41300e-01, 1.22740e+03]]), 
'a': array([[7.4640e-01, 1.1600e+02],
       [7.4650e-01, 1.0100e+01],
       ...
       [7.4900e-01, 3.7459e+03]])}

and

print(trade_data.keys())
print(trade_data['bybit'].keys())
print(trade_data['bybit']['CELOUSDT'][0])
dict_keys(['bybit', 'hyperliquid'])
dict_keys(['CELOUSDT', 'AERGOUSDT', 'DYDXUSDT'])
[ 1.72823125e+12  7.48300000e-01  8.10000000e+00 -1.00000000e+00]

On top of providing the mock classes, the Replayer class has utility functions that plot useful statistics about portfolio behaviour during the backtest. For example, we can print/plot the equity of an exchange:

if simulated:
    print(replayer.df_exchange(exc='bybit',plot=True))

with prints:

                              equity    inventory        pnl     CELOUSDT  AERGOUSDT  DYDXUSDT
ts
2024-10-06 16:37:25.646  9999.958291  -100.062330   0.000000  -100.062330    0.00000    0.0000
2024-10-06 16:37:30.646  9999.765185   321.157780  -0.193106   321.157780    0.00000    0.0000
2024-10-06 16:37:35.646  9999.743700   321.136295  -0.214591   321.136295    0.00000    0.0000
2024-10-06 16:37:40.646  9999.510305   435.397155  -0.447985   435.397155    0.00000    0.0000
2024-10-06 16:37:45.646  9999.510305   435.397155  -0.447985   435.397155    0.00000    0.0000
...                              ...          ...        ...          ...        ...       ...
2024-10-06 16:44:00.646  9990.384971  1869.103225  -9.573319  2347.421175   61.41905 -539.7370
2024-10-06 16:44:05.646  9990.542696  1869.260950  -9.415594  2347.578900   61.41905 -539.7370
2024-10-06 16:44:10.646  9989.687043  2081.486310 -10.271248  2559.192660   61.41905 -539.1254
2024-10-06 16:44:15.646  9988.824693  2062.252035 -11.133598  2540.487285   61.41905 -539.6543
2024-10-06 16:44:20.646  9990.357818  2057.837550  -9.600473  2536.072800   61.41905 -539.6543

with plot:

alt text

the exchange breakdown:

    print(replayer.df_portfolio(plot=True))

with prints:

                             bybit  hyperliquid  sum(exchange)
ts
2024-10-06 16:37:25.646   0.000000     0.000000       0.000000
2024-10-06 16:37:30.646  -0.193106     0.000000      -0.193106
2024-10-06 16:37:35.646  -0.214591     0.000000      -0.214591
2024-10-06 16:37:40.646  -0.447985     0.000000      -0.447985
2024-10-06 16:37:45.646  -0.447985     0.000000      -0.447985
...                            ...          ...            ...
2024-10-06 16:44:00.646  -9.573319    -0.156068      -9.729387
2024-10-06 16:44:05.646  -9.415594    -0.153772      -9.569366
2024-10-06 16:44:10.646 -10.271248    -0.313967     -10.585215
2024-10-06 16:44:15.646 -11.133598    -0.319717     -11.453315
2024-10-06 16:44:20.646  -9.600473    -0.349887      -9.950360

with plot:

alt text

the markouts by ticker, exchange or aggregated:

    print(replayer.df_markouts(ticker=None,exc=None,plot=True))
alt text

prices and fills:

    print(replayer.df_prices(ticker='CELOUSDT',exc='bybit',plot=True,with_fill=True))
alt text

Cheerios~ happy trading!

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 QUANTA GLOBAL PTE. LTD. 202328387H.
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More