Read more

Note that terminal access pass cost increases in 3 days.

Queue Modelling in HFT Simulations

In HFT market making, running high-fidelity tick-data simulations is particularly challenging. It is easy to invent false artefacts around market microstructure which generate positive PnL but do not hold in practice. This is because we make assumptions around important considerations like latency, queue priority, market impact and so on.

At a high level, an execution simulation has to be concerned with:

1. When our order lands on the order book.

2. When our cancellation lands on the order book.

3. Our estimated queue position relative to exogenous market actions.

4. The impact of other traders reacting to our orders.

5. The endogenous impact of our orders on the simulated order book.

The first two are latency problems. Order-latency distributions may be backed out from our own order traces, with an appropriate sampling method. Similarly, an order remains live while its cancellation is in flight. Market events between these timestamps have to be applied against the correct exchange-side state.

The third is the usual queue-position problem: how much liquidity is ahead of us, and how does that quantity change as trades, submissions, modifications and cancellations occur?

The fourth is a much harder problem. Historical replay tells us how other participants behaved in the observed market. It does not tell us how they would have behaved had our simulated order been larger, smaller or absent. Modelling this properly requires an endogenous model of external order flow - we will leave that outside the scope of this post.

Our focus in this post is the theoretical basis of points 3 and 5. In particular, we will separate two ideas which are often mixed together under “queue modelling”:

• Impact modelling: how should our order change the displayed and matchable liquidity inside the replay?

• Position modelling: where should our order sit, and how should exogenous events change the quantity ahead of it?

Given different choices to queue models, we also want to know how we may tune or calibrate them from a statistical framework.

MBO vs MBP

When we have market-by-order data—MBO (L3)—queue reconstruction is relatively straightforward. The feed identifies individual order submissions, modifications and cancellations. Assuming price-time priority, we can reconstruct the orders at a level and insert our own order according to its exchange-arrival timestamp.

When an external order is cancelled, we know which order left the queue. We therefore know whether the cancellation occurred ahead of or behind us.

In crypto, we typically have market-by-price data—MBP (L2). The feed gives us aggregate quantity at each price but does not identify the orders making up that quantity.

Consider a level displaying ten units. The level later falls to six. We know that four units have left, but the L2 update does not tell us where those units were in the queue.

If they were ahead of our order, our position has improved by four. If they were behind us, our position has not improved at all. This is the crux of estimating queue priority and building queue models.

An optimistic model may remove it from ahead of us; a pessimistic model may remove it from behind us; a probabilistic model sits somewhere between the two.

Before we get to this position problem, there is a lesser asked but equally important question.

Suppose the historical L2 book displays ten units at a price and we submit a two-unit order. Should the simulated book now display twelve units, or should our two units be treated as part of the ten already observed?

There is no one-size-fits-all answer. It depends on our order size, the liquidity of the market and, importantly, whether we were already quoting on the book used to record the historical data. This is an orthogonal question to point 4—should our orders should be represented inside the public book?

The good news is that we can answer such difficult questions by tuning our queue models and parameters and comparing them against empirical results, thereby reducing the dimensions of our problem.

In QuantPylib, the L2 and L3 order books are mapped through a common data structure. QuantPylib’s replay logic implements the discussed policies.

We will discuss how synthetic L2 MBP data can be projected onto an L3 MBO book by separating liquidity into anonymous and private sizes, over which we may configure different queueing and impact policies.

Projecting L2 into an L3 State

Consider a single price level displaying ten units of liquidity. Our two-unit order lands at the same price.

There are two ways to project the new state.

Additive Impact

The additive model is simple. Historical public liquidity and our private liquidity are disjoint.

If the observed L2 quantity is L and our total private quantity is O, the matchable size is:

Our order is appended to the price level when it arrives. Subsequent L2 updates change the public quantity independently of our order. Cancelling our order removes its entire remaining size.

Borrowed Impact

The borrowed model is more challenging because public and private liquidity are no longer disjoint accounting buckets.

For one side and price, define:

• L: public size reported by the historical L2 feed;

• O: total size of our active private orders;

• B: private size borrowed from—and therefore already counted in—the historical public size;

• E: anonymous external liquidity represented in the synthetic L3 book;

• T: total physical quantity available to the matcher.

The following invariants must hold:

For a fully borrowed-impact model, we target:

and therefore:

Probably confusing, so let’s walk through some interesting cases.

Private size exceeds the observed level

Suppose the historical level displays three units and we submit an order for five:

The private order can borrow only the three units available in the public observation:

All three public units are represented by our order. The remaining two units are necessarily additive.

The cancellation behaviour is slightly less obvious. If we cancel the five-unit order, only its two additive units should disappear from the physical book. The three borrowed units still belong to the historical public observation and must return as anonymous liquidity:

before cancellation:
[ours: 5]

L = 3, O = 5, B = 3, E = 0, T = 5

after cancellation:
[anonymous: 3]

L = 3, O = 0, B = 0, E = 3, T = 3

Cancelling the private claim does not cancel the historical market.

Public size moves through private size

Now suppose our private order has size ten while the historical L2 level displays five:

L = 5, O = 10, B = 5, E = 0, T = 10

The public level then rises from five to eight:

L = 8, O = 10, B = 8, E = 0, T = 10

The physical matchable size does not change. Three more units of our private order simply become borrowed from the updated public observation.

If the L2 level subsequently rises to twelve:

L = 12, O = 10, B = 10, E = 2, T = 12

Our full ten-unit order is now borrowed, while the remaining two public units must be represented as anonymous liquidity that queues behind our owned orders.

The reverse case - suppose the public level falls from twelve to seven:

before:
L = 12, O = 10, B = 10, E = 2, T = 12

after:
L = 7, O = 10, B = 7, E = 0, T = 10

The private order remains physically matchable for ten units. The two anonymous units disappear, while three units of the private order lose their borrowed attribution and become additive.

The different impact models fundamentally discuss the philosophy of whether our simulated orders should be treated as additional liquidity, or as liquidity already represented in the observed market data.

Queue Position Policies

Once the impact policy has produced a synthetic L3 level, the queue policy determines how exogenous reductions change our position.

New public liquidity joins the tail. Historical trades consume liquidity from the head. The ambiguous event is an L2 size reduction: did the disappearing liquidity sit ahead of us or behind us?

Optimistic and pessimistic bounds

Consider a borrowed-impact book with ten public units. We submit an order for two:

[anonymous: 8] [ours: 2]

The public level then rises from ten to fifteen. The new liquidity joins the tail:

[anonymous: 8] [ours: 2] [anonymous: 5]

The level subsequently falls back to ten.

An optimistic, front-cancellation policy removes the five units from the head:

[anonymous: 3] [ours: 2] [anonymous: 5]

Our queue position has improved by five. A subsequent trade for five units consumes the three anonymous units ahead of us and fills our two-unit order.

A pessimistic, back-cancellation policy removes the five units from the tail:

[anonymous: 8] [ours: 2]

Our position has not improved. The same five-unit trade does not reach us.

The market data is identical in both cases. Only the queue policy changes.

Probabilistic queue models

The optimistic and pessimistic models are useful bounds. A probabilistic model allocates the reduction between liquidity ahead of and behind our order.

Let:

• A: anonymous quantity ahead of our order;

• H: anonymous quantity behind our order;

• C: observed L2 reduction.

Define the relative queue position:

A simple probability function is:

where:

The exponent γ > 0 controls the curvature of the cancellation-probability function.

All curves meet at x=0.5. When the anonymous quantity is split equally ahead of and behind our order, the model assigns equal cancellation probability to both sides regardless of γ.

For γ > 1, the curve becomes steeper around the midpoint. The model increasingly assigns the reduction to whichever side contains more liquidity.

For 0 <γ < 1, the curve flattens. Cancellation probabilities are pulled towards 0.5, making the model less sensitive to the relative amount of liquidity ahead and behind us.

In practice, the probability function depends on rather predictable factors such as distance from the top of book. We can generalize the probability function by fitting different curves across reasonable market variables.

These policy choices and parameters can then be calibrated against empirical order traces.

Calibrating Queue Policies and Parameters

We now have several defensible impact and queue models. The remaining question is which model best represents the market we are trying to simulate.

Calibrating directly against strategy PnL is problematic. Market-making strategies are state-dependent: one different fill changes inventory, which changes subsequent quotes, cancellations and order sizes.

The action paths quickly diverge, leaving us unable to determine whether a PnL difference came from the queue model or from the strategy.

Instead, we calibrate through a closed-action replay. We hold the live order actions fixed and allow only their simulated outcomes to change.

Trace and tick-data objects

Event journalling architecture in quantpylib was discussed here:

Consider an order trace containing:

• order and cancellation intents;

• submission and cancellation timestamps;

• side, price, size, order type and time-in-force;

• acknowledgements and terminal order states;

• fills, including their quantity, price and exchange timestamp.

From this trace, we may extract the exogenous action tape:

submit order A at t1
submit order B at t2
cancel order A at t3
submit order C at t4
...

The second object is the public tick-data stream. It contains the market events required to reconstruct exchange-side state:

• L2 snapshots and updates;

• trades;

• exchange and receive timestamps;

• …
  

The market data capture infrastructure in QuantPylib supports these artefacts.

For a candidate model θ, we replay the same action tape of order submissions and cancellations against the same tick-data stream.

In QuantPylib, the wall clock is swapped against the replay clock and the same trace architecture generates replay trace files. We can then compare the live trace fill against the simulated trace file in a queue replay harness.

We can therefore compare outcomes order by order and determine quantitatively the “fitness function of a queue model”.

Fill classification

Let I be the set of submitted order intents. Define:

as the set of orders which received at least one live fill, and:

as the set filled under candidate model θ.

Fill precision is:

This answers: when the simulation produces a fill, how often did that order also fill live?

Fill recall is:

This answers: how many live-filled orders did the simulation reproduce?

A combined F1 score is:

A pessimistic model may have excellent precision while missing many live fills. An optimistic model may reproduce most live fills while inventing many others.

Lifecycle accuracy

Let s_i and ŝ_i,θ be the terminal live and simulated states of order i, such as filled, partially filled, cancelled or rejected.

Lifecycle accuracy is:

This catches differences which a binary filled/unfilled label misses, particularly partial fills and cancellation outcomes.

Fill timing

For orders filled in both traces, let t_i and t̂_i,θ be the timestamps of the first live and simulated fills. Define:

A smooth timing score is:

The scale τ determines the timing tolerance. The score ranges from zero to one. A score of one represents identical first-fill timestamps, while values near zero indicate timing errors which are large relative to τ. Semantically, it quantifies how closely the simulated trace reproduces first-fill timing for orders filled in both traces.

Fill-price similarity

Let p_i and p̂_i,θ be the live and simulated volume-weighted fill prices for an order filled in both traces.

The price error in basis points is:

We can convert this into a bounded score:

where κ controls the price-error tolerance and:

weights the comparison by overlapping filled notional.

For passive single-price orders, this metric may carry little information: both traces often fill at the submitted limit price. It is more useful for aggressive orders, multi-level executions and partial fills across prices.

Cancellation races

Cancellation behaviour deserves its own comparison. Let:

indicate that order i filled while its cancellation was in flight. Define r̂_i,θ similarly for the simulation. Cancellation-race accuracy is:

This exposes a common source of optimistic replay: allowing the order to disappear when the strategy decides to cancel rather than when the cancellation reaches the exchange.

Trace similarity as a fitness function

The component metrics can be combined into a trace-similarity score:

with:

The weights should reflect the use case.

For calibration sessions D_train, we select:

The purpose is not to find the queue model which produces the highest backtest PnL. It is to find the model whose order-level outcomes most closely resemble the empirical trace.

thanks for reading. cheers!

We also discussed how can you natively manage your own tick data lake with Quantpylib:

This is the architecture that supports exchange replay fidelity - from when the exchange processed data, to when it dispatched (TX) it, to when you received (RX) it, giving you critical observability into your own trading stack’s processing back-pressure and latency figures.

There is no substitute for high grade fidelity replay for latency-sensitive analytics involved in quantitative trading - public data simply does not make the cut.

Today, we release the ability to perform session-replay on your own tick data lake in the QuantTerminal.

Obtain your access pass here.

Terminal access pass cost increases in 1 week. More on the terminal here.
Next release will have

• private orders, positions, fills display on live exchanges

• private orders, positions, fills replay on tick data with your own trace files.

Next post on the blog discusses the architecture and technical details to designing a high fidelity tick data backtest - soon to be released on quantpylib!

The last of the Quantitative Trading Strategy series pertains to hft mm. Of course, for obvious reasons - I am unwilling to share in detail the specifics. However, I do want to share a piece of market microstructural analyses between binance and polymarket crypto markets I did.

This article is abit of a special format - instead of written form, this is simply a snippet from Lecture 7 of my (yet to be released) hft lectures I have on the topic. Then some food for thought and tricks.

The specific figures there are no longer applicable due to the changing dynamics of the markets involved, but the techniques therein are applicable no matter and to orthogonal markets…

Read more

Currency desks are commonly divided by geography and market structure: G10, EMFX, LatAm, and so on. In FX, the policy regime plays a significant role in price discovery, and central bank leaning is one of the primary sources of price discovery.

Python Code for replication is provided at the end of the article.

In this series, we will be looking at trading SGD crosses. The MAS manages the Singapore dollar against a basket of currencies through the S$NEER.

The regime is normally described as basket, band and crawl:

• Basket: SGD is managed against the currencies of Singapore’s major trading partners rather than against one bilateral rate.

• Band: the trade-weighted index is allowed to move within an undisclosed policy band.

• Crawl: the slope of that band is reviewed and adjusted as the inflation and growth outlook changes.

These parameters are not disclosed, except for rough “policy guidance”. MAS may intervene through spot or forward FX transactions. Insofar as possible, market forces are otherwise allowed to determine where the currency trades inside the band.

I will leave the economic motivation of these policies as self-reading.

As you may imagine - for a number of business functions (such as fx hedging, speculation) in banks, hedge funds and whatnot - currency desks are undeterred in modelling these parameters.

We will perform such an exercise with public data and construct a trading strategy on a basket of tradable assets. Since MAS explains policy decisions in qualitative language, we will come up with a simple, algorithmic state machine.

In order to assess the quality of our policy recalibration, we will cross-reference with United Overseas Bank (UOB)’s 2019 and 2021 paper.

We will take a look at these subtasks:

• data preparation for statistical modelling

• performing numerical analysis through constrained regression

• algorithmic reconstruction of policy mid, band and width

• portfolio construction from proxy index

• portfolio construction on tradable basket

data preparation for statistical modelling

We require two datasets for the initial statistical model: the official S$NEER history and bilateral SGD exchange rates. Both are available for free from MAS.

The official S$NEER history can be downloaded as an XLSX file from the MAS exchange-rate statistics page. For our purposes the workbook contains two relevant fields:

date          sneer
1999-01-08    100.00
1999-01-15     99.75
1999-01-22    100.10

Each value represents the weekly average S$NEER for the week ending on the stated date. This gives us the dependent variable. We next need a set of explanatory FX rates.

daily FX data

Daily end-of-period exchange rates are available through the MAS API catalogue. The request requires a (free to obtain) API key and returns JSON records. We retain the following currency fields:

USD, CNY, EUR, MYR, JPY, AUD, GBP, IDR,
KRW, THB, CHF, TWD, INR, PHP, VND and CAD

This is our candidate universe, not a claim about the actual basket. The statistical model will determine which currencies receive weight. The raw columns use the following quote convention:

usd_sgd         SGD per 1 USD
eur_sgd         SGD per 1 EUR
jpy_sgd_100     SGD per 100 JPY
idr_sgd_100     SGD per 100 IDR

We normalise this to sgd_xxx convention.

A complete cross-section is available beginning January 2008. The FX data is daily while the official S$NEER series is weekly. Since the target represents a weekly average, we take the arithmetic average of each FX series over the same week.

If W_t is the set of daily observations belonging to week t, the weekly normalized FX level is

Let S_t denote the official weekly-average S$NEER level for week t. The resulting DataFrame has one row for each common observation week t=0,…,T. We denote this weekly level panel by

The first n columns, F, contain the normalized weekly-average FX levels. The final column, S, contains the corresponding official weekly-average S$NEER levels. In pandas, regression_df is simply the tabular representation of 𝒟.

constructing the basket through restricted least squares

The statistical problem is to infer a dynamic set of currency weights from the weekly level panel 𝒟=[F | S]. We use a weighted geometric basket.

Let S_t denote the official S$NEER level in week t, and let F_t,i denote the normalized weekly-average value of SGD against currency i. We model the changes by differencing the sneer value:

The residual varepsilon_t absorbs anything else not captured by our candidate universe.

Define the weekly S$NEER return

and the vector of bilateral FX returns obtained from the first n columns of 𝒟,

Stacking T observations gives

and

The regression can then be written compactly as

An unconstrained least-squares estimator would solve

This may fit the historical S$NEER well while producing an economically implausible basket. The resulting weights may be negative or may not sum to one.

We know more about the object than ordinary least squares uses. Basket weights should be non-negative, and the complete basket should carry 100% weight. We therefore solve the restricted problem

The feasible set is the probability simplex

The restrictions give the coefficients an immediate economic interpretation as basket weights.

numerical solution (skip if not interested in the math)

The squared-error objective

is convex, with gradient

We solve the constrained problem using projected gradient descent. From a feasible vector w_k, first take an unconstrained gradient step

then project it back onto the simplex:

The projection is itself the optimization problem

Its solution has the form

where θ is chosen such that

Intuitively, we subtract a common threshold from the unconstrained vector, set negative coordinates to zero, and choose the threshold so that the remaining weights sum to one.

We use the corresponding step size

and stop when successive projected weight vectors are sufficiently close:

rolling estimation

We estimate a separate basket for every observation week using the most recent m=260 weekly returns, or approximately five years. This produces a sequence of models rather than one final model, which are the dynamic weights of the basket across time:

predicting the S$NEER

A model indexed by week t contains the official S$NEER observation for that week. We shift each fitted weight vector forward by one observation through the preceding observation, ŵ_t-1, and multiply them by the current weekly FX-return vector r_t. The weighted sum is the predicted S$NEER log return. Applying that return to the previous official level gives

How well do the previously estimated basket weights explain the next official weekly move?

The actual and predicted levels lie almost on top of one another with a 0.972 correlation between the predicted and official weekly log returns, and a 5.07bps in weekly return RMSE.

The fit is strong enough to treat the estimated basket as a useful S$NEER proxy. We have a number of remaining parameters to estimate.

reconstructing the policy band

MAS would make a brilliant X finfluencer, because they are experts in vague-posting. MAS summarises monetary policy statements in roughly this form:

date          slope                 width                   centre
2018-04-13    Increase slightly     -                       -
2020-03-30    Set at 0%             -                       Re-centre downwards
2022-04-14    Increase slightly     -                       Re-centre upwards

Our job is to translate qualitative policy language into a deterministic state machine.

Let g denote the annualized slope of the midpoint and let h denote the log half-width of the band. An unchanged field carries its previous value forward. Every other phrase updates one component of the policy state:

• increase very slightly: g ← g+0.5δ

• increase slightly: g ← g+δ

• increase: g ← g+2δ

• reduce slightly: g ← max(0,g-δ)

• reduce: g ← max(0,g-2δ)

• set at 0%: g ← 0

• widen slightly: h ← 1.25h

• widen: h ← 1.5h

• restore narrower band: h ← h₀

• re-centre: reset the midpoint to the prevailing (predicted) S̃_d

The rules are symmetric. The zero floor prevents a sequence of reductions from implying a negative slope due to MAS slope policy since the outset.

We set one slope tick to δ=0.5% per annum, use a baseline log half-width of h₀=0.02, or approximately 2% on either side of the midpoint, and anchor the April initial slope at 2% per annum.

This is aligned with the UOB’s paper. Once these values are fixed, the same rules are applied across the entire time series; there is no additional parameter fitting.

evolving the midpoint

Let M_d be the policy midpoint on day d. Between re-centring decisions, its log level accumulates the prevailing annualized slope:

where Δτ_d is the fraction of a year elapsed since the preceding observation. On a re-centring date, we reset M_d to the prevailing reconstructed S$NEER and continue accumulating drift from the new base.

The upper and lower boundaries are then

Using a geometric band keeps both boundaries positive and makes widening symmetric in log-return space.

We have now constructed a parsimonious model calibrated against a deterministic state machine, and verified that results are somewhat in line with a reputable bank. If we are economist, we would be happy, but as traders, we make no money from this…

But how may we construct a continuous portfolio from this?

portfolio construction

Read more

• when an order was created locally and sent into the wire

• when the order arrived at the exchange

• when the order-ack was received locally

• and all other state transitions.

These artefacts can be used downstream for a number of important business functions such as

• latency analysis;

• slippage analysis;

• markout and adverse-selection analysis;

• tick-to-trade shortfall;

• trading-cost attribution; and

• reconciliation and incident replay.

Event journalling is a core component of replay fidelity. A replay recovers the order in which the live process observed and accepted state changes. A journal lets us rebuild those transitions.

We are running 1-week of discounts of access to Quantpylib:

In this article, we give a rundown of quantpylib’s native binary event-journal architecture and how to use it, including the Quantpylib Event Trace (QET) compiler. We will also show how quantpylib users can produce QET trace files directly from an OMS workflow.

Artefacts

In quantpylib, the following portfolio artefacts are journaled:

• order state changes;

• fill insertions;

• position deltas;

• order snapshots; and

• position snapshots.

We will use an order lifecycle as the running example. Consider the following HYPE limit order:

from decimal import Decimal


order = {
    "exc": "hyperliquid",
    "ticker": "HYPE",
    "amount": Decimal("1"),
    "price": Decimal("20"),
    "tif": "Alo",
    "cloid": oms.rand_cloid(exc="hyperliquid"),
}

await oms.limit_order(**order)

The OMS first constructs the exchange-native order wire. Before submitting that wire, it inserts the order into its in-memory Orders ledger with a CREATE_PENDING status and a local ts_submit_ns. At this point the client order ID (cloid) is known, but the exchange order ID (oid) usually is not.

When an update arrives through the private order WebSocket, the ledger resolves the order by cloid or oid and applies its staleness and identity checks. A successful acknowledgement will typically complete the identity with an oid and transition the order to NEW. A rejected submission transitions to REJECTED. Later updates may move a live order through PARTIAL and FILLED, or through CANCEL_PENDING and CANCELLED when we attempt a cancellation.

The diagram is deliberately simplified. The exact path depends on the venue and execution outcome.

For the CREATE_PENDING order above, the in-memory object and its trace artefact look like this:

S identifies the trading session, while N is the session’s trace sequence number. Based on the state machine, the appropriate events. such as trace.orders.insert, trace.orders.state_transition is created.

These actions occur in the trading hot path, so an inefficient transport would make the journal unusable. The trace artefact is compactly encoded as raw bytes and handed to quantpylib's high performance logging subsystem.

The transport is therefore a native C++ implementation adapted into byte buffers over the LMAX Disruptor pattern, exposed in Python.

Read the design here: Designing State-of-the-Art Logging in Python.

Insofar as the trading system is concerned, the trace is complete. This makes the trace subsystem a lightweight attachment (a normal logging statement) and non-intrusively composed with client applications.

QET Compilation

Once the logging hand-off is complete, the log file receives JSONL records of the following shape.

{"level": "INFO", "time": "2026-07...", "message": "trace", "event": "trace.orders.insert", "trace": "qwAAAAMAAAAHAAAA...VzAw"}

event identifies the accepted mutation. trace is the original binary artefact, base64-encoded. Trace records and ordinary application logs may share the same file.

An event journal can be attached directly to a live trading system. quantpylib instead uses the log files as the substrate and compiles QET files in a separate, cold process.

The trading process is finished once it hands the record to the logger; parsing, ordering, deduplication, and journal I/O remain outside the live system.

Many firms already operate some form of the Grafana logging stack. Grafana Alloy acts as the collector. The usual destination is Grafana Loki, which stores compressed log chunks and indexes their labels for later queries.

quantpylib piggybacks on this battle-tested transport. We configure Alloy to push the same records to our compiler's Loki-compatible endpoint:

loki.write "qet_compiler" {
  endpoint {
    url = "http://127.0.0.1:33200/loki/api/v1/push"
  }
}

This can be the sole destination or added to an existing Alloy pipeline.

The compiler is a small local HTTP service that implements the Loki push API. It decompresses each Snappy-compressed protobuf batch, recovers the original JSON line, filters trace records, base64-decodes the trace, and validates the QET schema and record length.

The Loki entry timestamp selects the UTC output day; the trace's sid and seqno is used for ordering and idempotence. The compiler then appends the original binary records to a daily .qet file.

Example

Here we will give a walkthrough of how to wire them together.

Install Alloy using Grafana’s platform instructions. The Alloy CLI command reference is here.

1. start the compiler

python3.11 -m quantpylib.hft.qet_compiler serve \
  --root ./qet \
  --host 127.0.0.1 \
  --port 33200 \
  --buffer-size 48

Check its counters from another terminal:

curl http://127.0.0.1:33200/health

2. start Alloy

examples/configs.alloy (quantpylib) points Alloy at an absolute log glob and at the local compiler. Point Alloy at the file written by the example in step 3:

export QUANTPYLIB_LOG_PATH="$(pwd)/logs/journal.log"

alloy validate examples/configs.alloy
alloy run --storage.path=./.alloy-data examples/configs.alloy

3. run the example

import os
import asyncio
import logging
from decimal import Decimal

from dotenv import load_dotenv

from quantpylib.hft.oms import OMS
from quantpylib.logger import Logger
from quantpylib.gateway.master import Gateway


load_dotenv()

exchange = "bybit"
ticker = "HYPEUSDT"
amount = Decimal("1")
price = Decimal("10")

async def main():
    gateway = Gateway({
        exchange: {
            "key": os.environ["TEST_BYBIT_KEY"],
            "secret": os.environ["TEST_BYBIT_SECRET"],
        },
    })
    await gateway.init_clients()

    os.makedirs("logs", exist_ok=True)
    logger = Logger(
        name="perf",
        stdout_register=True,
        filename="journal.log",
        logs_dir="./logs",
        file_level=logging.INFO,
    )

    oms = OMS(gateway, logger=logger)

    try:
        await oms.init()
        cloid = oms.rand_cloid(exc=exchange)
        res = await oms.limit_order(
            exc=exchange,
            ticker=ticker,
            amount=amount,
            price=price,
            cloid=cloid,
        )
        print(res)
        await asyncio.sleep(5)

        res = await oms.cancel_order(exc=exchange, ticker=ticker, cloid=cloid)
        print(res)
    finally:
        await oms.cleanup()
        logger.shutdown()

if __name__ == "__main__":
    asyncio.run(main())

In journal.log we see one insert and three state transitions, corresponding to create, create-ack, cancel, and cancel-ack. We also see snapshots, which allow recovery when a new session begins.

{"ts":"2026-07-26 17:15:25.642","level":"INFO","filename":"portfolio.py","line":1157,"msg":"trace","event":"trace.orders.snapshot","trace":"b'...'"}
{"ts":"2026-07-26 17:15:25.766","level":"INFO","filename":"portfolio.py","line":408,"msg":"trace","event":"trace.positions.snapshot","trace":"b'...'"}
{"ts":"2026-07-26 17:15:25.965","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.insert","trace":"b'...'"}
{"ts":"2026-07-26 17:15:25.998","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.state_transition","trace":"b'...'"}
{"ts":"2026-07-26 17:15:31.000","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.state_transition","trace":"b'...'"}
{"ts":"2026-07-26 17:15:31.093","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.state_transition","trace":"b'...'"}

Inspect the generated qet file here:

from pathlib import Path

from quantpylib.hft.qet import QETReader, QET_SCHEMA_REGISTRY


path = sorted(Path("qet").glob("*.qet"))[-1]
reader = QETReader(path)

for event in reader.decoded_records():
    event_type = QET_SCHEMA_REGISTRY[event["schema_id"]].partition("(")[0]
    print(event["sid"], event["seqno"], event_type)

which gives me the following traces:

92375019 0 QETOrdersSnapshot
92375019 1 QETPositionsSnapshot
92375019 2 QETOrder
92375019 3 QETOrder
92375019 4 QETOrder
92375019 5 QETOrder

I myself have used quantpylib’s crypto wrappers on dexs, prediction markets et cetera in latency sensitive workflows to generate more than $1M in market making pnl, and more than $1B in volume driving my quantitative research and strategies.

The library allows developers to assimilate into their own trading pipeline, allowing sophisticated traders to use it’s well-abstracted features at the appropriate layer, whether it is for risk management, offline research, or live trading.

Today, we announce the QuantTerminal, which gives a face to Quantpylib’s core features - this will give quant teams an interface and access to end-end applications of the best and most powerful workflows possible in Quantpylib.

Target Core features

• Live, multi-exchange Tick Data Terminal

• Live, multi-exchange position and Order Tracking

• Tick Data Lake Archival and Session Replay

• Binary Order Event Trace for HFT Market Making

• Quantitative Analytics for Order Execution (markouts, slippage, latency etc)

• Live Telemetry

• and more!

This is an internal quant tooling system we’ve been sitting on, and we have decided to do a staged rollout of it. We understand that quant teams need to own their IPs - code and data are sacred. Therefore, the terminal application is transparent software that is intended to be run locally on your own workstation - with your own data. It’s powered by quantpylib, therefore access to the library is a prerequisite.

First stage of the rollout is the live terminal view. Here is a short 30-sec demo.

Note that it’s in a beta-stage rollout, and priced as such. As we continue to rollout, we will increase the price of the access pass accordingly.

Access, Quantpylib 1 Week Discount

The QuantTerminal access pass is available here:
https://quantpylib.hangukquant.com/pricing/pricing/

For one week, we are running discounted access to Quantpylib. Running QuantTerminal requires access to the Quantpylib library.

I have traded more than $100 million in volume across these platforms, generating more than $1 million in PnL. My operations have primarily revolved around three strategy families:

• cross-exchange, combinatorial and statistical arbitrage;

• event-driven arbitrage;

• high-frequency market making.

The first family is somewhat capacity-constrained. Liquidity remains shallow across many nascent venues, and seemingly identical contracts are not necessarily fungible. Two exchanges may list economically similar questions while using different resolution sources, deadlines, wording or discretionary procedures. What looks like a locked arbitrage can therefore contain rather more basis risk. In fact, as I write this post, here is an obvious one between poly and hpl.

The second family is considerably more scalable, but requires serious thought around engineering processes. I ran reduced-form versions of these strategies in the past before eventually pausing them. Too many parts of the system were “non-automatable”: particularly around sourcing new contracts, registering new data sources et-cetera.

Recent improvements in agentic systems—particularly following GPT5.5 and Opus4.8 make the problem interesting again in my opinion.

The third strategy family, high-frequency market making, is the most scalable of the three. It also requires proper domain expertise, quantitative reasoning and a reasonably intimate understanding of exchange microstructure. This is what placed me near the top of the relevant leaderboards in terms of both volume and PnL.

We have, in fact, documented an entire lecture series detailing the design and implementation of my HFT ops, to be later posted here.

It will not be released until yours truly is no longer involved in said markets.

For now, we will focus on event-driven arbitrage. Our discussion here will take a more idealistic approach compared to what I ran, accounting for the scalability now possible under accelerated agentic capabilities. The objective is a pipeline capable of serving hundreds to thousands, of prediction-market contracts.

Read more

A factor model is motivated by the observation that market movements are primarily driven by a relatively small number of common sources of risk. Assets may have thousands of individual return streams, but much of their variation is shared: broad market movements, industries, economic characteristics, liquidity conditions, or latent statistical relationships. The purpose of a factor model is to separate these common movements from asset-specific noise.

There are three broad approaches: Fama-style (time-series) portfolio factors, statistical factor models such as PCA, and Barra-style (cross-sectional) exposure models. They differ not merely in interpretation, but in the direction of estimation, the information supplied to the model, and the quantity being estimated.

Getting timely exposure to desirable factors — and hedging undesirable ones — is a primary concern of smart-beta funds and are a TRILLION dollar industry.

Factor analysis in crypto is still nascent. This article focuses on Barra-style models and how I actively trade them to enhance my pnl.

The full implementation is included at the bottom.

We will touch lightly on the first two, and then detail a Barra-style analysis with code.

Fama-style

A Fama-style factor is defined as the return of a portfolio. For example, a simplified value factor might be

More generally, if (p_k,t) contains the asset weights of factor portfolio (k), then

Once these factor-return histories have been constructed, the exposure of an individual asset is estimated through time:

The factor-return matrix (F) is known. The regression estimates (β_i):

For a single, demeaned factor,

Fama-style models are motivated by economic hypotheses about expected returns. Is a manager’s apparent alpha simply compensation for exposure to known factor strategies? Is our portfolio (statistically) significantly just some beta against a factor portfolio?

For a portfolio with estimated factor betas (β_p),

Statistical-style

A statistical factor model begins with the complete return matrix:

where rows are dates and columns are assets. It seeks a lower-dimensional approximation:

where F ∈ ℝ^T×K contains the statistical factor returns and L ∈ ℝ^N×K contains the asset loadings. In principal components analysis, first estimate the asset covariance matrix:

of demeaned returns. Decompose it as

The first (K) eigenvectors define the principal directions:

and the corresponding factor scores are

The resulting rank-(K) reconstruction is

Unlike the Fama regression, however, neither the factors nor the loadings are specified economically.

These are motivated mainly by dimensionality reduction. If hundreds of assets share a few dominant covariance patterns, the researcher may want to discover those patterns without imposing characteristic taxonomy.

Here is a seminal paper on enhanced portfolio optimization using principal components by AQR’s Pedersen et al.

Barra-style

The focus of our post, is the Barra-style model. We will first discover it’s mathematical basis, and then use them to identify sector performance.

We begin by specifying each asset’s exposures. On date (t),

contains exposures to K factors, such as categories, industries, countries, or continuous styles such as size and momentum.

The model then examines the cross-section of asset returns on that date:

The exposures (B_t) are known. The factor returns (f_t) are estimated using the (N) assets observed on that date.

This regression is repeated for every date, therefore produces a time series of factor returns from a sequence of cross-sectional regressions.

Barra-style models are motivated by the need to describe the current economic composition of a portfolio. Rather than waiting for a long return history to reveal a portfolio’s betas, the manager declares or measures the underlying asset exposures directly.

Repeating the cross-sectional regression each day produces a synthetic return series for every factor. By extension, if sector inclusion is defined as a factor, we obtain the synthetic return of the sector performance.

They are useful to:

• portfolio managers performing return attribution;

• risk managers estimating common and specific risk;

• traders neutralising unwanted industry or style bets;

• isolating sector performance for smart beta plays.

The factor return has a different meaning from a Fama-style factor. It is the return attributed to one unit of exposure after controlling for the other exposures in the cross-sectional regression.

Example'; crypto Barra model

We will demonstrate the categorical Barra analysis here using a reduced four-asset problem with two categories. The full regression code can be found at the bottom.

Let’s begin.

Suppose our universe contains four assets:

• BTC — L1; daily return 4%; liquidity score 4.

• ETH — L1; daily return 2%; liquidity score 2.

• DOGE — Meme; daily return 8%; liquidity score 1.

• PEPE — Meme; daily return 6%; liquidity score 1.

The objective is to estimate three returns:

The market factor represents the weighted return of the crypto universe. The category factors represent L1 and Meme performance relative to that market.

1. Construct the return vector

For each asset, the daily close-to-close return is

Collect the four asset returns into the vector

This is the dependent variable in the daily cross-sectional regression.

2. Construct the regression weights

For each day, determine asset’s regression weight. Reasonably, we may choose the square root of its preceding 30-day dollar volume:

The weights are then normalised:

Our simplified liquidity scores are

Since they sum to eight, the normalised weights are

The WLS weighting matrix is simply diag(w).

BTC contributes half of the regression weight, ETH contributes one quarter, and DOGE and PEPE each contribute one eighth.

3. Construct the exposure matrix

Every asset has market exposure equal to one. It also has exposure equal to one to its assigned category.

The exposure matrix is

for columns [market, L1, Meme] and rows [BTC, ETH, DOGE, PEPE].

Each day, the regression equation is given by (solve for f)

For instance,

The model assigns the same fitted common return to assets in the same category. Differences between the individual asset returns in a fitted category return remain in the residuals.

4. Remove collinearity problem

The market column is equal to the sum of the two category columns:

Consequently, the exposure matrix does not have full column rank.

For a particular solution

and any constant a,

produces exactly the same fitted asset returns.

We may solve this by requiring the liquidity-weighted average category return to equal zero.

5. Reduction of Degree of Freedom

Let the aggregate weight of category c be

In our example,

We impose

Substituting the category weights,

The constraint vector is therefore

and the constraint is

The leading zero means that the market coefficient is not directly constrained.

6. Solving the constrained WLS

We minimise the weighted squared residuals:

subject to

Introduce a Lagrange multiplier λ:

The first-order condition with respect to f is

Rearranging,

The first-order condition with respect to λ is

Combining the two equations produces the KKT system:

For our example,

The weighted return vector is

The first element is the weighted return contribution of the whole market:

The second is the weighted contribution from L1 assets:

The third is the weighted contribution from Meme assets:

The complete system is

Solving gives

The constraint is satisfied:

7. Interpretations

The weighted return of the L1 category is

The weighted return of the Meme category is

The category coefficients are the differences between these category returns and the weighted market return:

The fitted common return for an L1 asset is

The regression therefore tells us:

• the weighted market returned 4.25%;

• L1 underperformed the market by 0.9167%;

And the same exercise applies to the Memes category.

The residuals describe the returns not explained by category membership.

For BTC,

8. Repeat the analysis through time

The same cross-sectional regression is repeated for every date t. Each regression produces one market return and one relative return for every represented category:

Stacking the daily estimates produces a synthetic return history for every category/sector.

For example,

is the estimated history of the crypto meme-sector relative to the weighted crypto market. If we take these as ‘sector returns’ and compound them from 1, we essentially have a synthetic sector index. This is (theoretically) tradable with a factor-mimicking portfolio.

9. Recover the factor-mimicking weights

The estimated coefficients are linear combinations of the asset returns. We may therefore recover asset weights that reproduce them exactly.

For the market factor, the mimicking weights are the normalised regression weights:

For category c, the mimicking weights are

For L1, this gives

These weights sum to zero:

Their realised return is

For Meme,

Its return is

This exactly reproduces the L1 factor return.

The category factor can therefore be interpreted as a long position in the weighted category basket and a short position in the weighted market basket.

The factor-mimicking portfolio exists exactly in the mathematics. Essentially, what we have done is to transform quantitative analysis from price space to factor space, using statistical categorical analysis. In the process, we might

• better appreciate crypto asset returns as a mixture of factors/sectors

• construct a more informed portfolio by reducing noise in our samples

For instance, using “theoretical” factor-mimicking portfolios, we may look to construct a long-short delta neutral portfolio for “bullish sectors” against “bearish sectors”. Or we may look to “enhance” our trend construction from the previous post to trading in the factor space, subject to same considerations of portfolio construction that were discussed.

Code:

Let’s take a look an example construction of the above test to a universe of hundred tickers from Binance and Python code.

Read more

IN my first quant internship, I was tasked with testing commodity futures skew effects.

pm: "how did you account for contract roll"

me: "treated them as continuous, sometimes it's contango sometimes backwardated, so. it's small approximation error"

pm: 😑

In trading, a lot of effects and strategies are known. Index rebalancing, stat arb, yield curve arbitrage, market making - are all known strategies. The devil is in the details, and like my old self, doing something and doing something right are completely different things. If you are targeting these strategies, fantastic! You are in the right arena. For those testing some inverse shoulder rsi combo schtick, it’s time to come around. No shame in where the journey starts.

This article is dedicated a little bit to that - when asked about how one should “start their quant journey”, I always default to this one - a fantastically simple strategy that opens up a ton of critical questions that expand to more complex strategies. For example

• Execution fees and optimal rebalancing

• Signal quantisation

• Volatility and square root laws

• Diversification

• Backtesting sanitation

• Signal transformation

• Risk targeting/management

• Order automation

It is also one of the strategies that is “slow enough” that can be handled manually, and where different components mentioned above can be automated “one-at-a-time”, which is the good old principle of abstraction and core principle of building complex systems.

Our next post will continue on factor trading and how I leverage ‘smart-beta’ strategies to enhance portfolio returns.

Shall we begin?

(Code is attached at the bottom.)

Signal is the easy part.

The high-level rule is simple: buy markets that are going up and sell markets that are going down. It gives us a strategy whose intuition can be stated in one sentence, and then there is everything else.

Until we specify how strongly to trade the signal, how much risk to allocate to each asset, how contracts are sized, when trades are executed and how costs are charged, it’s not a quantifiable test.

Trend following is therefore a useful first quantitative strategy for two reasons. It is simple enough that we can intuit the entire research process, while being rich enough to expose common lapses in framework and backtesting.

Moskowitz, Ooi and Pedersen document this effect across equity-index, currency, commodity and bond futures. A diversified portfolio of these signals produced returns with little exposure to conventional asset-pricing factors and tended to perform well during large market moves.

Let’s just say, there is a ton (two centuries) of evidence supporting trend following: A Century of Evidence on Trend-Following Investing and Two Centuries of Trend Following.

As they say, if it’s robust, simple does the trick. The simplest discrete signal is the sign of a trailing return:

The forecast is +1 when the asset has appreciated over the previous h periods and -1 when it has depreciated.

A moving-average crossover produces a similar binary rule:

Now imagine if you were thirsty, and you could only drink or not drink a litre at a time. That would be miserable. A continuous forecast tries to retain information about trend strength.

A continuous forecast distinguishes between a weak and strong trend. It can also move gradually through zero rather than jumping directly from fully short to fully long. Continuous rules based on the statistical strength of a trend can materially reduce turnover without a statistically significant performance penalty.

One possible construction is a volatility-normalised trailing return:

In more nuance, weak trends appear to persist, but the relationship eventually saturates. We know this from the Potters and Bouchaud paper. The appropriate forecast may therefore look less like an infinitely increasing linear function and more like a clipped or saturating one. A sensible forecast transformation is often saturating:

Can I Have One HubbaHubba of Egg Contracts? On Risk.

Said no-one ever.

Suppose Bitcoin and interest-rate futures both produce a forecast of +1. What does this even mean? Giving them the same dollar exposure makes no sense from an economic standpoint. A standard normalisation feature to encode a dimensionless forecast is the forecast annualised return volatility of asset i:

where b_i is an instrument or asset-class risk budget.

The inverse-volatility term gives smaller positions to volatile assets and larger positions to quiet assets. With binary forecasts and equal risk budgets, each instrument contributes approximately the same standalone volatility before correlations are considered.

Extending this framework, asset normalisation does not guarantee portfolio risk stability. 10 contracts long A and 10 contracts short B is a different portfolio from 10 contracts long in each. Instruments are correlated, correlations are dynamic, and the positions are too.

Let u_t be the vector of preliminary exposures and let Sigma be the forecast covariance matrix of asset returns. The forecast volatility of the preliminary portfolio is:

Given a portfolio volatility target, the final exposure vector is

scaling the whole portfolio so that its forecast volatility equals the target.

There are therefore two distinct pieces of risk management:

1. Asset-level normalisation prevents volatile instruments from dominating the portfolio performance.

2. Portfolio-level scaling adjusts total leverage based on dynamic correlations and the realised volatility of the combined strategy in relation to target.

Volatility targeting has a wider literature of its own. Reducing exposure during high-volatility periods improves the historical Sharpe ratios of several risk factors.

We Live in the Real World

For a linear contract, a simplified conversion from notional weight to units is

where A is the portfolio capital, P is the contract price, M is contract multiplier and X is the currency conversion. The position must then respect integer contract quantities, minimum order sizes, tick sizes, leverage limits and venue-specific margin rules. This may surprise some crypto traders, but you can’t always buy 0.0001 of $MONKEY coin in liquid futures markets where standardised contracts trade in (large) fixed sizes. Accordingly, this tracking error can play a part in ‘which universe of assets’ are sensibly tradable for hedge funds - even when AUM is minimally tens of millions! When generating backtests, we need to be careful about directionally correct and economically meaningless results.

Next on the chopping board of reality are fees. The cost function includes commissions, bid-ask spreads, slippage, market impact and carry/funding costs.

Thankfully, when doing longer-term trend following, fees are more…secondary class citizens. However, as you evolve in your career as a trader, or look at higher frequency trading, you, too, will come to obsess about fees. But for now, in this post, we can be abit more cavalier about it.

To be frank, cost management and proper backtesting principles probably deserve an entire series on its own, but that borders on the HOW rather than the WHAT, and I promised not to bore you with the how in this series. I do have an entire post on statistical inferencing for backtest validation, if you are interested.

So, for now - COST IS IMPORTANT, and non-negligible, and ideas from optimal control theory suggest that we should…have continuous signals but not continuous positions. This prevents us from “flip-flopping” our holdings and paying the broker or the market maker excessively. Fortunately, we can get 95% of the way there with 5% of the brain: by applying “positional inertia” - the inertia to trade into target position when the current position is sufficiently close to the target.

The “rough-ideal” for inertia levels can be set by setting average trading costs and varying inertia levels, and then simply doing a walk forward backtest to find the optimal tradeoff. If robustness is anywhere found, a reasonable band will appear. Generally, the more noise (or low Sharpe your cost-free strategy is), the larger should be your inertia threshold. If your signal is all-noise, then you should just sit on your hands and stop trading all together.

Code

Comparing with and without costs:

From a code perspective, it is critical from a research infrastructural standpoint to encompass all (or most) of the above concerns in a parsimonious framework. What separates novice traders from more sophisticated quantitative analysts is the research harness.

Read more

We developed a number of critical tips and tricks there that the quantitative trader should be aware of when profiling and deploying trading strategies. It was very well received, and a reader even changed their username to thenimblemm 😅

I’ve been writing this blog for 5 years now, and over the years - we focused mainly on developing quantitative research frameworks - on the how rather than the what. Here and there, we discussed some quantitative strategies, which talk about the what.

After half a decade, I thought it would be a nice series to give some more detail into WHAT goes on behind the scenes in my ‘dirty operations’.

From trend following, to factor trading, FX policy arbitrage, event driven trading, to cross-exchange statistical arbitrage and hft market making - these 5 years was a busy one!

7 Days Discount

For 1 week, all subscription plans are 50% off:

https://www.research.hangukquant.com/d7b4564d

You can also support me with a Lifetime Subscription here:

https://buy.stripe.com/dRm3cw0mC05N6Am3XKe7m0f

Shout out to the ~300 lifetime members who have supported me, love ya back.

Hope you look forward to the series!

This is available in quantpylib, our quant repo for annual, paid subscribers:

We have redesigned critical core architecture and subsystems that will enable powerful workflows for traders using quantpylib in their trading infrastructure. Examples include performant and institutional-grade data replay, event journaling, telemetry systems and quantitative analytics.

We are proud to announce that managing your data lake has been now made a ~100~ lines of code powered by quantpylib, into raw binary flat files.

We have also shipped and documented a bunch of improvements to our backend, including cpp implementations for native orderbooks states. Once we launch versioning, we will deprecate old APIs.

See our data archival example here:

https://quantpylib.hangukquant.com/hft/hft/#data-archival

See our feed model and QBN encoding notes:

https://quantpylib.hangukquant.com/hft/feed/

https://quantpylib.hangukquant.com/hft/l0/

Our new, performant native orderbook implementation is documented here:
https://quantpylib.hangukquant.com/hft/orderbook/

We continue our aim of providing institutional grade trading infrastructure available to serious traders.

Note:
We intend to increase quantpylib’s costs over the coming weeks with a change in access model. Existing users with access will see no change. Non-subscribers may obtain an access pass here.

One of the revamped subsystems was the improvement of data capture architecture towards more institutional grade archival policies. This article will be dedicated to describing what may constitute such a change.

Modern quantitative research is becoming less constrained by storage cost and more constrained by the quality of the artefacts we choose to preserve. Cheap cloud object stores, high-throughput local disks, and increasingly capable agents have changed the economics of retention. It is now cheaper to store, easier to dig, making data storage policies and approach correctness a step-change more important as an interface for humans and intelligent solutions to sift through valuable insights hidden in the data.

For a trading desk, the highest-value artefacts are the operational record: market data, order traces, fills, gateway logs, replay journals, risk decisions, and the metadata that ties these streams together. Agents are good at sifting, ranking, joining, and summarising. But they are only as useful as the substrate underneath them.

Read more

And as of late, we have completely revamped the archival subsystem (to a custom L0 binary capture), and am in the process of rewriting the native orderbooks, as well as data replay systems. I am also working on a sidecar terminal application so that traders/market makers with strategies powered by quantpylib have institutional grade trading infrastructure backing them.

Anyhow, since I was on the topic, today, we are going to do a light(er) article on implementing an orderbook. Previously, we already spoke on this topic:

While that was a pretty simple and intuitive discussion, much more detail can be assigned to this task. It is also a common question at quant dev interviews, and I have recently had quant friends drilled on these.

Extending from our previous discussion, if we would like to extend an orderbook implementation into something matching-engine compatible and full-depth, the book needs to support queue position. That means the interface is no longer only MBP (market by price), but rather an MBO surface (market by order).

When it comes to L3 implementation, there are roughly two schools of thought. The standard text of reference is David Gross’s CppCon talk, When Nanoseconds Matter: Ultrafast Trading Systems in C++.

Some of the design pressures in this post are motivated by that discussion. Here we will provide some clarity and implementation on Gross’s solution and also expand on the crypto’s common L2 domain, and how we can support both interface efficiently without template overhead.

Read more

As a market maker with more than 100m+ of volume and leading PnL on the platform, the optimizations I’ve made to the SDK is now available to all quantpylib users. Websocket integration is also complete.

The optimizations include:

Read more

This post, we will discuss the introduction of a state of the art performance Python logging subsystem in quantpylib, and discuss some of the key design principles that allow us to achieve this. To my knowledge, among all Python logging frameworks, it is the lowest latency implementation out there.

As an aside, I am focused on making quantpylib into a more mature platform for serious quants. Beyond my expectations, the repo has found its way into many retail traders’ workflow, and even some small institutional trading stacks.

The project, although originally intended to be ‘my personal infra’ - is now bigger than myself, and accordingly, we will look to adopt higher standards appropriate to the gravity of its use cases. Although it is a best-effort guarantee service, the best-est of the effort will be brought forth.

Let’s get started!

A taster: (on log scale), quantpylib’s logger is 30-70x faster than standard logging and microsoft’s picologging library!

Logging is an important part of system observability. It gives us a way to reconstruct the system states, what decisions it made, and where things failed in live systems.

However, logging is not free. A log invocation may require caller discovery, record construction, string formatting, structured field handling, handler traversal, locks, stream writes, flush behavior, and sometimes exception or stack formatting. In latency-sensitive tasks such as trading, this cost can sit directly in the hot path.

In Python, reducing this cost is difficult because the usual logging path is intentionally dynamic. Python objects are allocated, frames may be inspected, dictionaries are populated, handlers own locks, and formatting often happens before the caller can move on. This is not the shape we want for threads parsing market data and executing time-sensitive actions.

To alleviate these issues, a number of existing libraries try to improve different parts of the logging problem. A couple examples -

• picologging is designed as a faster, mostly drop-in replacement for the standard library logging module. This is useful if the main constraint is compatibility with the existing logging API, while moving parts of the implementation into faster native (C) code.

• loguru attacks a slightly different problem. It makes logging much easier to configure and use: sinks, serialization, contextual binding, lazy formatting, and nicer ergonomics. Its lazy=True path can defer evaluation of expensive arguments until the message is actually emitted. Emitted log still travels through Loguru's event construction, formatting, and sink pipeline.

Although these are good libraries, most of them still preserve the broad log-record shape: create an event-like record, attach metadata, decide when the message is constructed, and push the result through a sink.

They differ in how much work is deferred and whether message construction happens directly in the hot path, but the core abstraction is still a dynamic log record carrying metadata through a logging pipeline. The end result is that log invocations cost up to microseconds. Where we are targeting t2t in the microsecond teens and below, observability is a hefty price to pay.

Our objective, therefore, is to be able to generate structured, useful logging in Python applications - that can be aggregated and analysed downstream in logging pipeline(s) such as as for ingestion into a Loki database.

In order to target this objective, we would like to ask ourselves the following questions:

• what components of log invocation add to the hot path?

• how do we reduce the number of Python object allocations in that path?

• which part of the logging workflow is actually costly: call-site lookup, record construction, argument formatting, handler traversal, serialization, or I/O?

• do we walk the Python frame stack to recover metadata such as filename, pathname, function name, or line number, and if so, how often?

• what concurrency mechanism makes logging thread-safe?

• if the logger acquires a lock, where is that lock acquired and how expensive is it under contention?

• does the caller thread format strings, or can formatting be deferred?

• does the caller thread call into the kernel for file or stream writes, or is that pushed into a writer thread?

• is the path synchronous with the sink, or does the caller only publish a compact record and move on?

Standard Path

A normal call like:

logger.info(”fill %d @ %f for %s”, qty, px, symbol)

can involve a surprisingly long sequence of work:

1. Enabled-level check: the logger has to decide whether the event is filtered out by the logger level, handler level, or effective inherited level.

2. Caller discovery / frame inspection: the logger may inspect Python frames to recover metadata such as filename, pathname, function name, and line number.

3. LogRecord construction: the logger creates a Python object representing the event.

4. Attribute population: the record is filled with message arguments, timestamps, process/thread metadata, exception information, stack data, and user-provided extra fields.

5. Handler traversal: Python logging is hierarchical, so the event may walk through logger and parent-handler control flow depending on propagation settings.

6. Filters: each logger or handler may run user-defined filters before the record is emitted.

7. Locks: handlers need synchronization because multiple threads may write into the same logging sink.

8. Formatting: the message string, structured fields, timestamp, exception text, or JSON representation are constructed.

9. Stream writes and flush behavior: the final bytes are written to stdout, a file, socket, or another sink, possibly crossing into the kernel or blocking on I/O.

The logging libraries including the ones we mentioned improve parts of this path, but many still operate in the same conceptual shape: create an event object, attach dynamic fields, render it, and pass it through a logging pipeline.

To break this latency floor, we have to remove pieces of the constraint rather than merely make each piece a little faster. The useful observation is that most log invocation sites are only partially dynamic.

For example, in printf-style logging invocations, the static information usually includes the format string, file name, line number, severity, number of arguments, and rough parameter types. The dynamic information is much smaller: the values passed into that invocation, occasional extra fields, and the timestamp.

These points carry forward into the implementation’s optimization techniques.

Techniques

For the specific implementation, refer to the quantpylib codebase. The high-level design is easier to discuss in terms of the techniques applied.

static information registry

The first technique is to treat a log invocation site as something that can be registered once and then referred to cheaply.

The Python-facing fast path resolves the caller into a key based on the CPython code object and line number. Implementation-wise, this goes through CPython frame objects: get the current frame, recover the PyCodeObject, read the line number, and combine the code-object pointer and line number through a small custom hash. The result is a per-thread cache key for that call site.

On a cache hit, the logger gets back a stable integer id. On a cache miss, the format string and frame metadata are registered into a static registry: pathname, filename, line number, severity, parameter count, rough parameter types, and an unload/formatting function. This id uniquely identifies the log invocation site.

Instead of constructing a fresh record that contains all metadata, each emitted record can contain a pointer to the static data plus only the dynamic payload:

struct StaticData {
  const char* pathname;
  const char* filename;
  const char* format;
  uint32_t line_number;
  LogLevel severity;
  int num_params;
  const ParamType* param_types;
  UnloadFn unloadfn;
};

struct LogEntry {
  const StaticData* static_data_ptr;
  uint64_t timestamp;
  uint16_t entry_size;
  uint8_t reserved[6];
  char args[];
};

There is a deliberate Python-specific constraint here that we enforce, which is that format strings should be stable literals.

For example, the intended shape:

log.info("fill %d @ %f for %s", qty, price, symbol)

is remarkably distinct from

log.info(f"fill {qty} @ {price} for {symbol}")

In the first case, the format string is a literal object associated with the call site. In CPython, the compiled code object holds constants in its constant table, and repeated execution of that call site loads the same Python string object from that table. Object identity is represented by the object’s address for its lifetime, i.e. what id(obj) exposes at the Python level. The logger uses this idea at the native layer by recording the PyObject* pointer for the format string. For repeated invocations of the same literal-style call site, that pointer remains stable.

In the second case, Python constructs a new string value dynamically, so the format string’s address is no longer a stable part of the call-site key. If the same source line starts producing different string objects, the logger cannot safely say that the cached id still refers to the same static logging schema, so it rejects the call.

This is a limitation we enforce so that the call-site cache is meaningful.

transportation layer

After the call-site cache resolves the static metadata id, the next problem is transportation: how do we move the dynamic log payload across the producer/writer thread boundary without turning the log invocation into a normal synchronized I/O operation?

The backend borrows from the LMAX Disruptor design. The important optimizations in that family of designs are as follows:

1. Preallocation: allocate the ring buffer upfront so memory can be reused instead of dynamically allocated on every log invocation.

2. Lock-free producer hot path: producers writing to the ring buffer avoid a lock/mutex on the latency-sensitive path, which removes the handler-lock shape of conventional logging.

3. Sequence caching: the producer memoizes the observed consumer sequence, amortizing the core-to-core cache-coherence traffic that would otherwise come from producer/consumer progress tracking.

Additionally, beyond the algorithmic improvements, the transportation layer makes optimizations designed to run well on modern hardware architectures:

4. Cache-line aware sequence counters: sequence counters are aligned and padded to cache-line boundaries to reduce false sharing.

5. Power-of-two indexing: ring positions can be computed with a bit mask instead of modulo arithmetic.

We generalize the LMAX Disruptor idea from a ring of fixed event objects to a ring over raw byte buffers.

The producer and the writer/consumer agree on a raw byte encoding for log entries. Instead of communicating Python objects or fixed event slots, the producer writes log entries as contiguous bytes using this encoding scheme, and the writer later decodes the bytes into formatted output. This matters because log records are not naturally fixed-size. A no-argument message, a three-argument printf-style message, and a message with dynamic extra fields have different byte footprints. Instead of forcing them into a fixed slot size, the logger reserves the exact number of bytes required for:

• the LogEntry header;

• the pointer to static metadata;

• timestamp and entry size;

• packed printf-style arguments;

• other fixed and variable length fields.

Each producing thread owns a thread-local log buffer. Conceptually, this turns the logging transport into multiple SPSC-style queues: each producer publishes into its own buffer, and the writer side consumes from those buffers. This reduces producer-side contention because producers are not fighting over the same write sequence.

The writer thread (in a non critical path) is then responsible for merging log entries from the thread-local buffers. In the implementation, this is done with an appropriate ordering structure such as a priority heap over timestamps. The writer side also decides where the log output should go, when writes should be flushed, and what policy should govern the generation of the actual log lines.

Further, the writer side can be pinned to a housekeeping core and can perform the heavier work of turning byte records into the desired output format, such as JSON or K=V logs. This not only defers formatting and heavy I/O away from the hot path, but also helps keep the producer working set smaller and friendlier to L1/L2 cache behavior, which matters on modern architectures.

Benchmark

Based on the techniques, here are the benchmarks, and the benchmarking code: (note that exceptions call Python runtime’s stack trace formatting/utilities before calling into the logger framework)

File Sink

=========== sink=file ===========

--- file / no_args ---
variant                   p50        p90        p99          max
stdlib                  6417n      7834n     15000n      129125n
loguru                 14792n     16796n     25750n     1213250n
picologging             5375n      6875n     13293n      110333n
structlog               3917n      4458n      8500n     1334334n
quantpylib-perf          167n       208n       375n       56708n

--- file / 1xint ---
variant                   p50        p90        p99          max
stdlib                  6667n      8500n     15418n      127875n
loguru                 15125n     18042n     29792n      706875n
picologging             5458n      7500n     13667n      131833n
structlog               4542n      4958n     10918n      108833n
quantpylib-perf          167n       208n       292n       10417n

--- file / 3xmixed ---
variant                   p50        p90        p99          max
stdlib                  6625n      7838n     14376n      114458n
loguru                 14875n     15542n     21250n     1903917n
picologging             5500n      5833n     11042n      124791n
structlog               4584n      4792n      8458n       28208n
quantpylib-perf          167n       209n       375n        1208n

--- file / extra ---
variant                   p50        p90        p99          max
stdlib                  7417n      8087n     14501n       29000n
loguru                 15791n     16417n     22000n     1438458n
picologging             5625n      6834n     12583n      112959n
structlog               4542n      4709n      6875n       26708n
quantpylib-perf          292n       334n       750n        1916n

--- file / exception ---
variant                   p50        p90        p99          max
stdlib                 37375n     44834n     68668n     1062792n
loguru                 49083n     60042n     97542n     3616500n
picologging            29458n     32458n     50125n      233792n
structlog              30416n     32167n     40626n     1301708n
quantpylib-perf        22834n     23459n     32667n       78333n

--- file / filtered ---
variant                   p50        p90        p99          max
stdlib                   125n       125n       375n        9083n
loguru                   459n       583n      1500n       24000n
picologging               42n        84n       334n       19958n
structlog               1042n      1084n      1458n       14458n
quantpylib-perf           42n        83n       167n       23667n

Stdout Sink

--- stdout / no_args ---
variant                   p50        p90        p99          max
stdlib                 11000n     13167n     37167n      553916n
loguru                 23833n     28083n    499468n     4076250n
picologging             9167n     10583n     20500n    56333375n
structlog               6083n      8625n     15459n      356625n
quantpylib-perf          167n       500n      1250n       34167n

--- stdout / 1xint ---
variant                   p50        p90        p99          max
stdlib                 11209n     13042n     23877n      197000n
loguru                 23833n     27541n    498848n     5344250n
picologging             8750n     10213n     14792n      212500n
structlog               6709n      8667n     13375n      109334n
quantpylib-perf          167n       542n      1041n       14333n

--- stdout / 3xmixed ---
variant                   p50        p90        p99          max
stdlib                 10959n     12750n     19750n      250958n
loguru                 24250n     29379n    530761n     3686250n
picologging             8958n     10458n     17375n      180542n
structlog               7250n      9042n     18668n      159709n
quantpylib-perf          208n       416n       709n       16166n

--- stdout / extra ---
variant                   p50        p90        p99          max
stdlib                 12958n     15084n     42048n      573750n
loguru                 25583n     29708n    527216n     3164916n
picologging             9042n     10542n     17590n      190875n
structlog               7917n      9625n     17209n       88417n
quantpylib-perf          292n       708n      1208n       40584n

--- stdout / exception ---
variant                   p50        p90        p99          max
stdlib                 68917n     89379n    553959n     3580625n
loguru                 83875n    109721n    816045n     6636292n
picologging            46125n     51625n     66709n     1413083n
structlog              63375n     82712n    591252n     3555583n
quantpylib-perf        24083n     39379n     63922n     1239959n

--- stdout / filtered ---
variant                   p50        p90        p99          max
stdlib                   125n       167n       541n       13500n
loguru                   500n       542n      1125n      119708n
picologging               42n        84n       125n        1042n
structlog               1167n      1625n      9542n      671791n
quantpylib-perf           42n       167n       334n        5292n

Bench Code:

import os
import sys
import time
import logging
from pathlib import Path

import numpy as np
import matplotlib.pyplot as plt
import quantpylib.logger as qlogger

LOG_FILE = "bench.log"
ITERS = 10_000
PAUSE_MICROS = 10

WORKLOADS = [
    "no_args",
    "1xint",
    "3xmixed",
    "extra",
    "exception",
    "filtered",
]
SINKS = ["file", "stdout"]

def invocations(fn, iters, pause_micros):
    fn() #warmup
    latencies = []
    for _ in range(iters):
        t0 = time.perf_counter_ns()
        fn()
        latencies.append(time.perf_counter_ns() - t0)
        if pause_micros:
            time.sleep(pause_micros / 1_000_000)
    return latencies

void_log = lambda logger: lambda : logger.info("hello")
int_log = lambda logger: lambda : logger.info("x=%d", 42)
mixed_log = lambda logger: lambda : logger.info("fill %d @ %f for %s", 42, 3.14, "btc")
extra_log = lambda logger: lambda : logger.info(
    "fill %d @ %f for %s",
    42,
    3.14,
    "btc",
    extra={"venue": "binance", "tag": "maker", "oid": 1234},
)
filtered_log = lambda logger: lambda : logger.debug("x=%d", 42)

def exception_log(logger):
    def log():
        try:
            1 / 0
        except Exception:
            logger.error(
                "error",
                extra={"exc": qlogger.get_exception(), "stack": qlogger.get_stack()},
            )
    return log

def bench_workloads(logger, iters=ITERS, pause_micros=PAUSE_MICROS):
    return {
        "no_args": invocations(void_log(logger), iters, pause_micros),
        "1xint": invocations(int_log(logger), iters, pause_micros),
        "3xmixed": invocations(mixed_log(logger), iters, pause_micros),
        "extra": invocations(extra_log(logger), iters, pause_micros),
        "exception": invocations(exception_log(logger), iters, pause_micros),
        "filtered": invocations(filtered_log(logger), iters, pause_micros),
    }

def _attach_handler(log, sink, handler_factory):
    for h in list(log.handlers):
        log.removeHandler(h)
    handler = handler_factory(sink)
    log.addHandler(handler)
    log.propagate = False

class LoguruAdapter:
    def __init__(self, logger):
        self._logger = logger

    def debug(self, msg, *args, extra=None):
        self._log("DEBUG", msg, args, extra)

    def info(self, msg, *args, extra=None):
        self._log("INFO", msg, args, extra)

    def error(self, msg, *args, extra=None):
        self._log("ERROR", msg, args, extra)

    def _log(self, level, msg, args, extra):
        logger = self._logger.bind(**extra) if extra else self._logger
        logger.opt(lazy=True).log(level, "{}", lambda: msg % args if args else msg)

class StructlogAdapter:
    def __init__(self, logger):
        self._logger = logger

    def debug(self, msg, *args, extra=None):
        self._log("debug", msg, args, extra)

    def info(self, msg, *args, extra=None):
        self._log("info", msg, args, extra)

    def error(self, msg, *args, extra=None):
        self._log("error", msg, args, extra)

    def _log(self, method, msg, args, extra):
        event = msg % args if args else msg
        kwargs = extra if extra else {}
        getattr(self._logger, method)(event, **kwargs)

def bench_stdlib(sink, log_file, **workload_kwargs):
    log = logging.getLogger("stdlib")
    log.setLevel(logging.INFO)
    def factory(sk):
        h = logging.FileHandler(log_file, mode='a') if sk == "file" else logging.StreamHandler()
        h.setLevel(logging.INFO)
        h.setFormatter(qlogger.JSONFormatter())
        return h
    _attach_handler(log, sink, factory)
    return bench_workloads(log, **workload_kwargs)

def bench_loguru(sink, log_file, **workload_kwargs):
    from loguru import logger
    logger.remove()
    output = log_file if sink == "file" else sys.stdout
    logger.add(output, level="INFO", serialize=True, enqueue=False)
    return bench_workloads(LoguruAdapter(logger), **workload_kwargs)

def bench_picologging(sink, log_file, **workload_kwargs):
    import picologging
    log = picologging.getLogger("picologging")
    log.setLevel(picologging.INFO)
    def factory(sk):
        h = picologging.FileHandler(log_file, mode='a') if sk == "file" else picologging.StreamHandler()
        h.setLevel(picologging.INFO)
        h.setFormatter(qlogger.JSONFormatter())
        return h
    _attach_handler(log, sink, factory)
    return bench_workloads(log, **workload_kwargs)

def bench_structlog(sink, log_file, **workload_kwargs):
    import structlog
    output = open(log_file, "a") if sink == "file" else sys.stdout
    structlog.configure(
        processors=[
            structlog.processors.add_log_level,
            structlog.processors.JSONRenderer(),
        ],
        wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
        logger_factory=structlog.WriteLoggerFactory(file=output),
        cache_logger_on_first_use=False,
    )
    try:
        return bench_workloads(StructlogAdapter(structlog.get_logger("structlog")), **workload_kwargs)
    finally:
        if sink == "file":
            output.close()

def bench_pylogger(sink, log_file, **workload_kwargs):
    from quantpylib.logger.logger import Logger
    log_path = Path(log_file)
    log = Logger(
        name=sink,
        register_handlers=[],
        stdout_register=(sink == "stdout"),
        stdout_level=logging.INFO,
        file_register=(sink == "file"),
        filename=log_path.name,
        logs_dir=str(log_path.parent or Path(".")),
        file_level=logging.INFO,
    )
    return bench_workloads(log, **workload_kwargs)

def bench_cplogger(sink, log_file, **workload_kwargs):
    from quantpylib.logger import Logger
    log_path = Path(log_file)
    log = Logger(
        # name="perf", defaults to perf
        stdout_register=(sink == "stdout"),
        stdout_level=logging.INFO,
        stdout_formatter_cls=qlogger.JSONFormatter,
        file_register=(sink == "file"),
        filename=log_path.name,
        logs_dir=str(log_path.parent or Path(".")),
        file_level=logging.INFO,
        file_formatter_cls=qlogger.JSONFormatter,
    )
    results = bench_workloads(log, **workload_kwargs)
    return results

def print_table(workload, results, sink):
    print(f"\n--- {sink} / {workload} ---")
    print(f"{'variant':<18} {'p50':>10} {'p90':>10} {'p99':>10} {'max':>12}")
    for variant, data in results.items():
        lats = np.asarray(data[workload])
        print(f"{variant:<18} {np.percentile(lats, 50):>9.0f}n {np.percentile(lats, 90):>9.0f}n {np.percentile(lats, 99):>9.0f}n {np.max(lats):>11.0f}n")

def plot_sink(sink, results):
    cols = 2
    rows = int(np.ceil(len(WORKLOADS) / cols))
    fig, axes = plt.subplots(rows, cols, figsize=(12, 4 * rows))
    for ax, w in zip(axes.flat, WORKLOADS):
        for variant, data in results.items():
            ax.plot(data[w], label=variant, alpha=0.6)
        ax.set_title(w)
        ax.set_xlabel('Iteration')
        ax.set_ylabel('Time (ns)')
        ax.set_yscale('log')
        ax.legend()
    for ax in axes.flat[len(WORKLOADS):]:
        ax.set_visible(False)
    fig.suptitle(f'logger latency: sink={sink}')
    fig.tight_layout()

def main():
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('-sink', '--sink', type=str, help='Specify logging sink = [file, stdout]', choices=SINKS, default='file')
    parser.add_argument("--log-file", type=str, default=LOG_FILE)
    parser.add_argument("--iters", type=int, default=ITERS)
    parser.add_argument("--plot", action=argparse.BooleanOptionalAction, default=True)
    args = parser.parse_args()
    sink = args.sink
    
    if os.path.exists(args.log_file):
        os.remove(args.log_file)

    all_results = {}
    print(f"\n=========== sink={sink} ===========")
    variants = {
        "stdlib": bench_stdlib(sink, args.log_file, iters=args.iters),
        "loguru": bench_loguru(sink, args.log_file, iters=args.iters),
        # "quantpylib-root": bench_pylogger(sink, args.log_file, iters=args.iters),
        "picologging": bench_picologging(sink, args.log_file, iters=args.iters),
        "structlog": bench_structlog(sink, args.log_file, iters=args.iters),
        "quantpylib-perf": bench_cplogger(sink, args.log_file, iters=args.iters),
    }
    all_results[sink] = variants
    
    for w in WORKLOADS:
        print_table(w, all_results[sink], sink)

    if args.plot:
        plot_sink(sink, all_results[sink])
        plt.show()

if __name__ == "__main__":
    main()

The Lighter SDK is designed to be both performance aware and intuitive, presenting a significantly improved interface over the official package. See full docs here:

https://quantpylib.hangukquant.com/wrappers/lighter/

and our market making tutorial here:

https://quantpylib.hangukquant.com/scripts/market_making/

We will update our polymarket SDK to follow the v2 integration soon.

Cheers!

As a continuation of the hip-3 perp dex integrations,

quantpylib now features Databento’s live dataset integration with the same init-subscribe-handler pipeline:

See docs and example script here:

We have also improved the order tracking features of our OMS utility library. To learn how to experiment with market making in just 300 lines of code with quantpylib, see our tutorial here.

That is practically the network latency of getting a market data payload from Binance through a cdn, which you might be doing if you are a mm anyway.

We can do much better.

In instances where orders are not preemptively signed, order signing lives directly on the hot path. Hyperliquid’s signing path (from signing.py) is a pipeline of canonicalization and hashing

EIP-712 encoding

and finally, secp256k1 signing.

Optimising this pipeline is a simple matter of appreciating the schema of an EIP712 payload, Ethereum’s structured-data signing format. The signer is given a typed message with 4 pieces:

• domain: identifies the application / signing domain

• types: declares the schema of the message

• primaryType: the top-level struct being signed

• message: the actual field values

The eth-account takes encode_typed_data of these components and produces a SignableMessage. On each order sign, it hashes the type definition of the EIP712 domain, performs ABI validation, obtains the domain separator, computes the Agent type hash - all of which are static.

The dynamic fields go into a tiny component, the connection-id and agent hash, which is part of the final digest:

keccak(”\x19\x01” || domain_separator || struct_hash)

a 32-byte digest signed by secp256k1. Between parsing the schema, walking the dictionary fields, ABI encoding and object creation/allocation - most of which are static per order - our hot path is not so hot anymore. If we can surgically stick to hashing only the dynamic components, our working set is a lot smaller.

Last but not least, significant performance degradations arise when we rely on native eth-account backend to perform signing. Elliptic curve signing relies on mathematically nontrivial operations, requiring large-integer modular arithmetic and scalar multiplication.

Following the ‘everything is object’ design, simple arithmetic such as ‘+’ requires namespace lookup, operator dispatch, pointer chasing, heap allocation and refcount churn in Python - cryptographic operations implemented at the Python layer is terrible news for low latency code.

Wow, Everything is Computer - President Donald Trump

Wow, Everything is Object - Python Engineer

Again, appreciating this truth is 90% of the job done - we can find solutions to push the cryptographic work from Python runtime into a secp256k1 backend optimised in C, such as the bitcoin-core library. Writing a tiny Python binding around it is the remaining 10%…oh wait, it already exists.

Bringing it all together, we can now replace our sign_l1_action with sign_l1_action_fast.

from coincurve import PrivateKey

_EIP712DOMAIN_TYPEHASH = keccak(b"EIP712Domain(string name,string version,uint256 chainId,address verifyingContract)")
_DOMAIN_SEPARATOR = keccak(
    _EIP712DOMAIN_TYPEHASH
    + keccak(b"Exchange")
    + keccak(b"1")
    + (1337).to_bytes(32, "big")
    + b'\x00' * 32
)
_AGENT_TYPEHASH = keccak(b"Agent(string source,bytes32 connectionId)")
_SOURCE_A = keccak(b"a")
_SOURCE_B = keccak(b"b")

def sign_l1_action_fast(ccwallet, action, vault_prefix, nonce, source_prefix, expires_after):
    data = msgpack.packb(action)
    data += nonce.to_bytes(8, "big")
    data += vault_prefix
    if expires_after is not None:
        data += b"\x00" + expires_after.to_bytes(8, "big")
    connection_id = keccak(data)
    agent_hash = keccak(_AGENT_TYPEHASH + source_prefix + connection_id)
    digest = keccak(b"\x19\x01" + _DOMAIN_SEPARATOR + agent_hash)
    sig = ccwallet.sign_recoverable(digest, hasher=None)
    return {"r": to_hex(sig[:32]), "s": to_hex(sig[32:64]), "v": sig[64] + 27}

class Hyperliquid:
    def __init__(self...):
        ...
        self.ccwallet = PrivateKey(self.wallet.key)
        self._vault_prefix = b"\x00" if self.vault_address is None else (b"\x01" + address_to_bytes(self.vault_address))
        self._src_prefix = _SOURCE_A if self.is_mainnet else _SOURCE_B

and there we go! Using a faster implementation of cryptographic signatures is most of the speedup, giving us 0.29-0.32ms in the same experiment. Using the pre-computed cache values gives a further 4-fold speedup, @ 0.08-0.09ms.

And that wasn’t even that hard…a simple trick in the Python Houdini

This is a RT of an article I wrote on X.

Exactly 2 years ago, on this date - I took a step into the treacherous crypto order books. My first foray into crypto trading was a quasi systematic funding arbitrage strategy, which I executed by hand using some spaghetti code dashboard to filter for funding pairs on binance-hyperliquid.

1 to 10 Sharpe

In those days, funding arbitrage was already a well known strategy, but execution, variation and reach played a big factor in realised APR. New exchanges seemed to be budding left and right, with unique funding mechanisms.

This pushed me to quickly build out more exchange connectors and a gateway functionality that would allow me to automate funding arbitrage strategies across multiple exchanges in a unified interface, which would later turn out to be the quantpylib project, a code repo started by my followers and then later lead by me to be a tool for both quantitative research and trading.

At this point, other projects like ccxt and hummingbot existed, but the quantpylib project focused more on connectivity and composability of features without relying on an architectural framework, allowing traders with variable strategies to implement high to low frequency strategies according to their needs. It integrated binance, bybit, woox, paradex and lighter, exchanges I executed arbs across. 2.5 years later and 1000+ commits later, quantpylib is still a work in progress.

I had decent reach and okay execution, enough to land me in the 40% APR range, while earning points - which ironically turned out to be even more lucrative than the pnl itself, all thanks to The El Jeffe crew. These days, with tools like @LorisTools, the scale and reach of such arbitrage strategies are extensive.

Despite the ~10 Sharpe, it was a strategy with extremely low vol, and scaling the pnl meant I had to put more of my already small net worth on chain, a risk I was not willing to take. It also took many man hours of attention, taking time away from investing in my own technical capabilities. In young, 'skilled' workers, knowledge compounds over the remaining career lifespan.

I shut it down.

At the same time, I was in a desperate struggle with my own mental faculties. I was, at heart, a builder. I built things in public, in an industry where large doses of skepticism were warranted. I had the imposter syndrome bug. Was I just a peddler of snake oil? Some people (including those I admired) and the anti-substack mob seemed to think so.

Hanguk’s income is from his newsletter. All he can do is implement a momentum trading strategy. I find it funny that Hanguk teaches systematic funding arbitrage but has to record pnl manually.

You write on a blog, I do this for a living, I would know.

I saw it in X replies, Discord channels, Reddit communities. Maybe I am just a fucking fraud, and I will never be that guy. After all, trading is an incredibly competitive industry, and greater man than me have failed. I am still young, and I can still pivot into being a quant trader where all my friends are. I could get referrals, I would be ‘in the industry’ and I won’t have to deal with all these bullshit.

These dark thoughts would later crescendo into anxiety and panic attacks. By all counts, my ‘online blog’ was bringing me 200k+ in ARR, and my trading was comparably rewarding. I was living my life. I once had no income and no pnl, and had an unshakeable confidence that I would ‘figure it out’. Having somewhat figured it out, it turned out that there were many cracks in this facade. Crypto strats were going to be squeezed by TradFi, and AI was going to wipe me out, who needs ‘research’ when everyone had a PhD-expert on their fingertips?

Wakeup, Doomer

The truth is, there were many people supporting me. For every nasty comment, there were ten others who cheered my work on.

Claude Mythos, OpenGod or whatever - I was well equipped and adaptable enough to thrive in a post-intelligent economy. I spent some time talking to my Dad, who had the wisdom of life to slap the doomer out of me.

It’s all in your head.

I got over it by biasing towards action. Day after day, I would wake up in sweats and palpitations, and before the darkness paralysed me, I would have done a handful of pushups, hopped in the shower, and was headed out to write some code. If I kept moving, it couldn’t catch me. It took a few months, and slowly but surely, I was okay again.

Fear is good marketing. Do not be the product.

10 to 50 Sharpe

In the months to follow, I would have the most productive schedule of my life.
I had always wanted to go from mid-frequency to high-frequency strategies. I wanted to be knowledgeable in niche domains and be a respected engineer. There was a lot more to learn, and my mind was liberated.

I worked on hft research, tooling, infrastructure and trading. I built new features on quantpylib, picked up C++, sharpened my mathematical knowledge and was allowed to be dumb again.

I was new, and publicly so. It had been awhile, but I was in love again. I read, I experimented, I asked and made mistakes. I basked in the wealth of knowledge. I didn’t have a PhD at my fingertips to replace me. I had one to teach me.

The best way to learn is by doing. In my frenzy of learning, researching and experimenting, I arrived at my next gold mine. Months of research, screaming at the monitor - I got my next reward, my first (successful, I had many unsuccessful ones) HFT strategy:

I consistently topped the leaderboards, fin-fluencers were writing nonsense threads about me, and I felt ... vindicated.

Sunset

When the sun rises, it must set.

There’s no such thing as the goose that lays the golden egg forever. - Jim Simons, Renaissance Technologies

Nowadays, I battle with the immortality of the goose. I hesitated if I should write this article. I’m hanging in there, but I am no longer a leaderboard resident.

As the sun sets, I am dumb again. But there is more iron in me.