Clock + Event-based market making + testing
A few days ago we added improvements to the market maker to fix bugs and allow for clock-based triggering of order actions.
With this addition, we are able to both trade AND test with the same code base strategies that involve tick data. Together with the simulator (Alpha, GeneticAlpha) suite and our new hft (feed, oms, mock) modules, we are able to trade on exchanges working with both OHLCV bars and tick data.
This post will be a demonstration of the tick data features and graphics that we have added. The full code is provided.
quantpylib is our community github repo for annual readers:
You can get the repo pass here, alternatively:
https://hangukquant.thinkific.com/courses/quantpylib
Moving on, we will go right into the demo. We will demonstrate the multi-exchange compatibility and portability of our logic. There is no alpha in the quoter - it is for illustrative purposes only.
Typically, a market-maker needs to be able to track their portfolio states: order states (pending/open/cancelled/rejected), open positions, account equity/balance and so on. In general, a market maker action triggers include but are not limited to internal clock cycles, trade arrival, orderbook delta updates pushed and a number of variable proprietary logic. We may choose to act on these data immediately upon arrival (a onTick behavior) or store it in some shared state that is later used to compute optimal market quotes. We will explore all of these options.
Let us make some imports:
import os
import asyncio
from pprint import pprint
from decimal import Decimal
from dotenv import load_dotenv
load_dotenv()
import quantpylib.standards.markets as markets
from quantpylib.hft.oms import OMS
from quantpylib.hft.feed import Feed
from quantpylib.gateway.master import Gateway
from quantpylib.utilities.general import _time
from quantpylib.utilities.general import save_pickle, load_pickle
exchanges = ['bybit','hyperliquid']
configs = {
"binance" : {
"tickers":["CELOUSDT"],
},
"bybit" : {
"tickers":["CELOUSDT","AERGOUSDT","DYDXUSDT"],
},
"hyperliquid" : {
"tickers":["GALA","RDNT","DYDX"],
},
}
config_keys = {
"binance" : {
"key":os.getenv('BIN_KEY'),
"secret":os.getenv('BIN_SECRET'),
},
"bybit" : {
"key":os.getenv('BYBIT_KEY'),
"secret":os.getenv('BYBIT_SECRET'),
},
"hyperliquid" : {
"key":os.getenv('HPL_KEY'),
"secret":os.getenv('HPL_SECRET'),
},
}
gateway = Gateway(config_keys=config_keys)
buffer_size = 10000
l2_feeds = {exc : {} for exc in exchanges}
trade_feeds = {exc : {} for exc in exchanges}
order_value = 50Define the keys for the exchanges you would like to integrate - in our demo, we would use bybit and hyperliquid. The keys are passed into the gateway. For demonstration, we will use a fixed order size of fifty dollars.
The gateway is the connector to the relevant exchanges - which is passed into the oms and feed objects. The oms handles order/position tracking, recovery, execution and auxiliary tasks. The feed does tick data subscription, storing, and retrieval.
We can instantiate the feed and oms and set up how long we want to run our quotes:
oms = OMS(gateway=gateway)
feed = Feed(gateway=gateway)
play = lambda : asyncio.sleep(60 * 10 * 3)
time = lambda : _time()For clock based trading agents, we can register a callback to the oms - this callback coroutine is called every specified interval. This could be as simple as 'submit/cancel quotes every 500ms' and so on. We will demonstrate an event-based quoter for now, but for demonstration - we will just print the exchange balances every 5 seconds - no action is taken. The actual quoter is done in a make coroutine that runs asynchronously for each ticker in each exchange:
async def clock_event():
pprint(await oms.get_all_balances())
async def main():
await gateway.init_clients()
await oms.init()
await oms.add_clock_callback(
callback=clock_event,
interval_ms=5000
)
quote_coros = []
for exchange in exchanges:
for ticker in configs[exchange]["tickers"]:
quote_coros.append(make(exc=exchange,ticker=ticker))
await asyncio.gather(*quote_coros)
#save data after finish quoting (can ignore), cleanup
l2_data = {
exc:{ticker:lob.buffer_as_list() for ticker,lob in l2_feed.items()}
for exc,l2_feed in l2_feeds.items()
}
trade_data = {
exc:{ticker:trades.get_buffer() for ticker,trades in trade_feed.items()}
for exc,trade_feed in trade_feeds.items()
}
save_pickle('hft_data.pickle',(l2_data,trade_data))
await oms.cleanup()
async def make(exc,ticker):
#implement maker logic
return
if __name__ == "__main__":
asyncio.run(main())Since we called the make function, we have to implement the maker logic. First, we will get a trade feed with no handler (this is just done for show, we won't use the trade data here). We will, however, register a handler for the orderbook ticks:
async def make(exc,ticker):
trade_feed = await feed.add_trades_feed(
exc=exc,
ticker=ticker,
buffer=buffer_size,
handler=None
)
live_orders = oms.orders_peek(exc=exc)
async def l2_handler(lob):
#this is called on orderbook tick
#lob is a quantpylib.hft.lob.LOB object
#submit actions on book depth stream
return
l2_feed = await feed.add_l2_book_feed(
exc=exc,
ticker=ticker,
handler=l2_handler,
buffer=buffer_size
)
trade_feeds[exc][ticker] = feed.get_feed(trade_feed)
l2_feeds[exc][ticker] = feed.get_feed(l2_feed)
await play()The l2_handler receives a lob object which is a quantpylib.hft.lob.LOB object - we can obtain either the live buffer from this object or statistics such as mid, vamp indicators and vol.
The oms does order tracking and maintains live_orders which is a quantpylib.standards.portfolio.Orders object. This is achieved via the gateway's underlying socket connections and requests. Note that the oms itself is a separate functionality provided by quantpylib, and is able to do a variety of useful things - such as registering coroutine handlers for order updates, position fills and so on - see examples here.
In this section, we will not register any order/position update handlers, and just rely on the live updation of our orders which is intitated by default on oms.init(). Say, inside the l2_handler we would like to submit/cancel orders using the following logic:
1. Determine the price we want to quote, say the third from top of book. Fix order value at 50.0.
2. If there is no pending (submitted and unacknowledged) bid or ask, or existing orders that are tighter than price at step 1 - submit an order at determined price.
3. If there are acknowledged orders that are tighter than the determined levels, cancel them. If there are acknowledged orders that are further from top of book than the determined levels, let them sit in the order book to maintain price-time priority.
4. If there are more than 5 resting orders on each side, cancel excess orders starting from lowest price priority.The above logic tries to pick up taker orders that slam a thin orderbook through multiple levels. Obviously, the feasibility of this depends on the market, our risk management, and whether a mean-reversionary effect exists upon said price impact, and this effect relative to costs/adverse fills. The logic tries to maintain time priority for duplicate levels. We make no comment or assertions on the viability of said 'rules'.
For specifics on how to pass the parameters to oms methods, refer to documentation and examples. gateway documentation and examples should be helpful.
async def l2_handler(lob):
mid = lob.get_mid()
inventory = float(oms.get_position(exc=exc,ticker=ticker)) * mid
bid_price = lob.bids[2,0]
ask_price = lob.asks[2,0]
order_bids = live_orders.get_bid_orders(ticker=ticker)
order_asks = live_orders.get_ask_orders(ticker=ticker)
any_pending_bid = any(order.ord_status == markets.ORDER_STATUS_PENDING for order in order_bids)
any_pending_ask = any(order.ord_status == markets.ORDER_STATUS_PENDING for order in order_asks)
any_tight_bid = any(order.price is not None and order.price >= Decimal(str(bid_price)) for order in order_bids)
any_tight_ask = any(order.price is not None and order.price <= Decimal(str(ask_price)) for order in order_asks)
orders = []
if not any_tight_bid and not any_pending_bid:
orders.append({
"exc":exc,
"ticker":ticker,
"amount":order_value/lob.get_mid(),
"price":bid_price,
"round_to_specs":True,
})
if not any_tight_ask and not any_pending_ask:
orders.append({
"exc":exc,
"ticker":ticker,
"amount":order_value/lob.get_mid() * -1,
"price":ask_price,
"round_to_specs":True,
})
ack_bids = [order for order in order_bids if order.ord_status != markets.ORDER_STATUS_PENDING]
ack_asks = [order for order in order_asks if order.ord_status != markets.ORDER_STATUS_PENDING]
cancel_bids = [order for order in ack_bids if order.price > Decimal(str(bid_price))]
cancel_asks = [order for order in ack_asks if order.price < Decimal(str(ask_price))]
cancels = cancel_bids + cancel_asks
cancels += ack_bids[5 + len(cancel_bids):]
cancels += ack_asks[5 + len(cancel_asks):]
cancels = [{
"exc":order.exc,
"ticker":order.ticker,
"oid":order.oid
} for order in cancels]
if orders:
await asyncio.gather(*[
oms.limit_order(**order) for order in orders
])
if cancels:
await asyncio.gather(*[
oms.cancel_order(**cancel) for cancel in cancels
])On the web-platforms of selected exchanges, we should see the quotes submitted - here is bybit:
We see our tightest quotes are third from mid, with deeper levels sitting in the order book. A price jump from taker order 0.1088 to 0.1085 hit our bid.
Backtesting
In this section, we show how to backtest using the quantpylib.hft.feed.Feed and quantpylib.hft.oms.OMS objects. As pre-requisite: please read the section on using the Feed and OMS modules, as well as using the section on using them in our market making demo.
Minimal code change is required. In fact, from the market-making demo, all we change is:
simulated = True
if simulated:
from quantpylib.hft.mocks import Replayer, Latencies
(l2_data,trade_data) = load_pickle('hft_data.pickle')
replayer = Replayer(
l2_data = l2_data,
trade_data = trade_data,
gateway=gateway
)
oms = replayer.get_oms()
feed = replayer.get_feed()
play = lambda : replayer.play()
time = lambda : replayer.time()
else:
oms = OMS(gateway=gateway)
feed = Feed(gateway=gateway)
play = lambda : asyncio.sleep(60 * 10 * 3)
time = lambda : _time()The quantpylib.hft.mocks.Replayer class provides mock classes for the OMS and Feed that behaves like the actual trading agents. This Replayer simulates agents involved in trading, such as public/private feed latencies, order submissions, matching and more. We don't need anything else, when the await play() is called, the backtest is run. Note that we can pass in a number of optional paramters, for example:
exchange_fees = {
"bybit": {
"maker":0.0002,
"taker":0.0005
},
"hyperliquid": {
"maker":0.0001,
"taker":0.0003
}
}
exchange_latencies = {
"bybit": {
Latencies.REQ_PUBLIC:100,
Latencies.REQ_PRIVATE:50,
Latencies.ACK_PUBLIC:100,
Latencies.ACK_PRIVATE:50,
Latencies.FEED_PUBLIC:100,
Latencies.FEED_PRIVATE:50,
},
"hyperliquid": {
Latencies.REQ_PUBLIC:200,
Latencies.REQ_PRIVATE:150,
Latencies.ACK_PUBLIC:200,
Latencies.ACK_PRIVATE:150,
Latencies.FEED_PUBLIC:200,
Latencies.FEED_PRIVATE:150,
}
}
replayer = Replayer(
l2_data = l2_data,
trade_data = trade_data,
gateway=gateway,
exchange_fees=exchange_fees,
exchange_latencies=exchange_latencies
)The data format looks like this (data pickled from the demo run above):
print(l2_data.keys())
print(l2_data['bybit'].keys())
print(l2_data['bybit']['CELOUSDT'][0])dict_keys(['bybit', 'hyperliquid'])
dict_keys(['CELOUSDT', 'AERGOUSDT', 'DYDXUSDT'])
{'ts': 1728232861170,
'b': array([[7.46200e-01, 5.80100e+02],
[7.46100e-01, 7.03100e+02],
...
[7.41300e-01, 1.22740e+03]]),
'a': array([[7.4640e-01, 1.1600e+02],
[7.4650e-01, 1.0100e+01],
...
[7.4900e-01, 3.7459e+03]])}and
print(trade_data.keys())
print(trade_data['bybit'].keys())
print(trade_data['bybit']['CELOUSDT'][0])dict_keys(['bybit', 'hyperliquid'])
dict_keys(['CELOUSDT', 'AERGOUSDT', 'DYDXUSDT'])
[ 1.72823125e+12 7.48300000e-01 8.10000000e+00 -1.00000000e+00]On top of providing the mock classes, the Replayer class has utility functions that plot useful statistics about portfolio behaviour during the backtest. For example, we can print/plot the equity of an exchange:
if simulated:
print(replayer.df_exchange(exc='bybit',plot=True))with prints:
equity inventory pnl CELOUSDT AERGOUSDT DYDXUSDT
ts
2024-10-06 16:37:25.646 9999.958291 -100.062330 0.000000 -100.062330 0.00000 0.0000
2024-10-06 16:37:30.646 9999.765185 321.157780 -0.193106 321.157780 0.00000 0.0000
2024-10-06 16:37:35.646 9999.743700 321.136295 -0.214591 321.136295 0.00000 0.0000
2024-10-06 16:37:40.646 9999.510305 435.397155 -0.447985 435.397155 0.00000 0.0000
2024-10-06 16:37:45.646 9999.510305 435.397155 -0.447985 435.397155 0.00000 0.0000
... ... ... ... ... ... ...
2024-10-06 16:44:00.646 9990.384971 1869.103225 -9.573319 2347.421175 61.41905 -539.7370
2024-10-06 16:44:05.646 9990.542696 1869.260950 -9.415594 2347.578900 61.41905 -539.7370
2024-10-06 16:44:10.646 9989.687043 2081.486310 -10.271248 2559.192660 61.41905 -539.1254
2024-10-06 16:44:15.646 9988.824693 2062.252035 -11.133598 2540.487285 61.41905 -539.6543
2024-10-06 16:44:20.646 9990.357818 2057.837550 -9.600473 2536.072800 61.41905 -539.6543with plot:
the exchange breakdown:
print(replayer.df_portfolio(plot=True))with prints:
bybit hyperliquid sum(exchange)
ts
2024-10-06 16:37:25.646 0.000000 0.000000 0.000000
2024-10-06 16:37:30.646 -0.193106 0.000000 -0.193106
2024-10-06 16:37:35.646 -0.214591 0.000000 -0.214591
2024-10-06 16:37:40.646 -0.447985 0.000000 -0.447985
2024-10-06 16:37:45.646 -0.447985 0.000000 -0.447985
... ... ... ...
2024-10-06 16:44:00.646 -9.573319 -0.156068 -9.729387
2024-10-06 16:44:05.646 -9.415594 -0.153772 -9.569366
2024-10-06 16:44:10.646 -10.271248 -0.313967 -10.585215
2024-10-06 16:44:15.646 -11.133598 -0.319717 -11.453315
2024-10-06 16:44:20.646 -9.600473 -0.349887 -9.950360with plot:
the markouts by ticker, exchange or aggregated:
print(replayer.df_markouts(ticker=None,exc=None,plot=True))
prices and fills:
print(replayer.df_prices(ticker='CELOUSDT',exc='bybit',plot=True,with_fill=True))
Cheerios~ happy trading!