we looked at how to do kernel tuning - where most of the time is spent on host t2t (typically in crypto infrastructures). Quoting from that article

Other than the network topology…high value work comes at the level of the kernel.

This qualification is important - because network latency is order magnitudes more important to get right than software t2t - here is father El Jefe on that note:

That is, optimising software t2t before working on network latencies is putting the cart before the horse. Missing the forest for the trees. Reading “how to pleasure your woman” without a first date.

In a previous post -

we talked about box placement strategies and how different azs, placement groups, instance families can play a part in network optimisation. We also talked about endpoint selection and ALBs.

Here we will talk about single box network edge selection. It’s essentially the opposite side of the same coin in proximity search. Code examples are in the quantcplib repo. Link at the bottom.

What Happens

Let’s consider what happens, when we try to connect to say,

The client presumably extracts the hostname fstream.binance.com, on port 443. This hostname must be translated into an IP address before we can open the TCP connection. The translation occurs via the libc function

This invokes the system’s configured name-resolution machinery. Depending on the host lookup configuration, the answer may come from a local hosts file, a resolver service or DNS. getaddrinfo gives us a linked list of socket addresses. The userspace network client then selects one of the addresses to connect to. Presumably, it walks the linked list and the first successful connection attempt is retained. The endpoint we use therefore depends on the addresses returned, their ordering and other less deliberate factors.

We can be more intentional about this edge resolution. For one, a wider set of candidates can be found via manual DNS querying than the heuristic set by the system resolver. Additionally, we are really interested in production-time market data latency, not arbitrary network latencies. We may question whether TCP handshake ranks translate to lower application layer ranks. We may also question whether such latency optimisations are persistent across time. And the answer will perhaps surprise.

Experiment

Read more

Our focus remains on providing institutional grade analysis and tools for quantitative traders -

https://quantpylib.hangukquant.com/terminal/

We continue to work on features in quantpylib, and will continue to serve by providing low latency access to the same ergonomic features through quantcplib in the future.

In the last post, we talked about kernel tuning -

we will continue exploring network optimisations and “cloud based” engineering in the upcoming posts.

Cheers

This applies also critically to software optimisation. One of the most important things to do in a lean engineering/trading team is to operate on the pareto frontier between performance optimisations and strategy discovery. As an extension, one can spend all the wrong time focusing on micro-optimisations when there are significantly more problematic bottlenecks.

Other than the network topology, algorithmic design and architecture - one of the most high value work comes at the level of the kernel. More precisely, it is about getting the kernel and userspace programs to cooperate.

I previously wrote a broader (and simpler) Linux kernel tuning guide. Here, we shall do something more concrete with the actual optimisations and code examples - do refer to the quantcplib repo for the actual scripts.

We will profile the ingress latency of a low-latency websocket client in quantcplib against a CPython implementation, measuring the latency between the AWS ENA RX hardware timestamp to a parsed WebSocket message. We will play around with different “settings” and discuss how they affect performance, with a soft, service level objective (SLO) of attaining ~10microseconds in ingress latency.

A Short Userspace Description

At a high level, in userspace, the receive path follows:

1. A nonblocking TCP socket calls recvmsg(), receiving ciphertext and the ENA hardware-timestamp control message together.

2. A custom OpenSSL BIO reads directly into the destination supplied by OpenSSL.

3. SSL_read_ex() writes plaintext into fixed application storage.

4. An incremental RFC 6455 frame parser consumes views over that storage.

5. An incremental message parser handles fragmentation and UTF-8 validation.

6. The message callback runs synchronously on the same network-owner core.

ENA DMA-writes into driver-managed receive buffers and recvmsg() still copies bytes from the kernel into userspace. The custom BIO removes an additional userspace ciphertext staging copy: we do not receive into our own temporary buffer and then write the same ciphertext into an OpenSSL MemoryBIO. This is sometimes described as zero-copy. The timestamp is attributed to the ciphertext receive which enabled the completing plaintext chunk. This is the fastest path for a data packet where the kernel is still responsible for ip/tcp processing and tls is performed in userspace -

Before Userspace

Before the network runtime can parse a WebSocket frame, an Ethernet frame must travel from the ENA device into the TCP socket.

The first decision is receive steering. ENA calculates an RSS hash from the packet headers and uses its indirection table to select an RX queue. One TCP connection will normally remain on one queue: additional queues distribute different flows.

At this point, the driver has already populated that queue’s RX submission ring with descriptors pointing to host receive buffers. When the packet arrives, ENA DMA-writes the packet into one of those buffers and posts a completion descriptor to the corresponding completion queue. The payload is now in host memory, but TCP has not processed it.

On the ordinary path, the next step is an interrupt. Interrupt moderation determines how quickly ENA raises the queue pair’s MSI-X interrupt after completions arrive. Deferring the interrupt allows several packets to be handled together, amortising interrupt and driver work; it also makes the first packet wait for the batch. This is known as interrupt moderation. A fixed rx-usecs value sets this delay directly, while adaptive moderation changes it according to recent traffic.

When the MSI-X interrupt is delivered, ENA automatically masks the queue interrupt and the hard-IRQ handler schedules the queue’s NAPI instance. The handler does very little packet work itself. The NAPI poll is normally serviced through the NET_RX softirq and consumes RX completions until the queue is empty or its packet budget is exhausted.

For every completion, the driver identifies the posted receive buffer and constructs an sk_buff, the kernel’s packet representation. The driver then submits the skb through napi_gro_receive(). The ENA driver documentation gives the concrete descriptor and skb path; the Linux NAPI documentation describes how the poll is scheduled and bounded.

GRO is the first important batching control inside that poll. When enabled, it can merge compatible packets from the same flow into a larger skb before the upper layers process them. This reduces repeated IP/TCP work and improves throughput, at the cost of batching and the bytes later observed by the application. With GRO disabled, packets proceed separately and the kernel pays more per-packet work.

The IP layer then processes the network packet, and TCP validates sequence state, handles reordering and makes contiguous stream bytes available on the socket receive queue. Once the socket becomes readable, the kernel wakes a task blocked in epoll_wait(). Epoll returns a readiness indication—not the bytes themselves—to the network runtime, which attempts a nonblocking recvmsg() and copies the available ciphertext, together with its ancillary timestamp data, into userspace.

Let’s highlight the available controls in this data path:

• RSS and IRQ placement determine which queue receives the flow and which CPU handles its MSI-X interrupt and initial NAPI work.

• Interrupt moderation determines how long a completion may wait before that interrupt is delivered.

• The NAPI budget limits how many packets one poll cycle processes before yielding.

• GRO determines whether compatible receive packets are combined before IP/TCP processing.

Two adjacent controls affect latency without changing this packet logic. Holding /dev/cpu_dma_latency open with a zero request constrains CPU idle-state exit latency through PM QoS; despite the name, it does not accelerate DMA. mlockall() and prefaulted buffers reduce disruptions from page faults and reclaim after the process is awake.

An alternative path: NAPI busy polling

An interrupt does not have to be the event which starts NAPI. With NAPI busy polling, an eligible socket read or readiness wait spends a bounded interval asking the kernel to poll the socket’s associated NAPI context before falling back to the MSI-X interrupt. If the completion arrives during that interval, packet processing can begin from the application’s syscall context without first paying the interrupt-delivery and task-wakeup path.

Everything after that entry point remains the ordinary kernel network stack. NAPI still consumes ENA completion descriptors, constructs skbs, passes them through GRO and IP/TCP, and places bytes on the socket receive queue. It changes how packet processing is initiated and which CPU performs it.

SO_BUSY_POLL sets the polling window for a socket. net.core.busy_read supplies the system default for socket reads, while net.core.busy_poll controls the global polling window for poll, select and eligible epoll waits. A busy-poll budget limits packets processed in one polling episode.

Topology Matters

We need to decide which RX queue receives flow, which CPU drains that queue, which CPU owns the socket and parser, and where the parsed event goes next.

For a small number of latency-sensitive WebSocket connections, each hot flow should land on a known RX queue and each queue should have one clear userspace owner. Two layouts are worth considering.

locality-first: interrupt-driven receive

For an epoll-driven client, one default is to place the RX queue’s MSI-X interrupt, its NAPI processing and the network-owner thread on the same logical CPU. The physical core should be reserved for the path: unrelated work should not run on that CPU or its SMT sibling.

RSS -> RX queue Q -> MSI-X / NAPI / TCP on CPU A
                                      -> epoll / recvmsg / TLS / parser on CPU A

This layout follows the semantics of readiness-driven receive. While the owner sleeps in epoll_wait(), the interrupt is useful work: it drains the queue, advances TCP and makes the socket readable. The same CPU then wakes into recvmsg(), TLS and WebSocket parsing. We pay for an interrupt, but not for an additional inter-processor hand-off between the kernel receive path and the application.

The cost is interference. A new RX interrupt can pre-empt the parser while it is handling an earlier message, and sustained NAPI work competes with userspace for the same execution resources. Colocation is therefore the latency-first default only while that core has headroom.

If the combined kernel and userspace work approaches saturation, a deliberate split can buy pipeline capacity:

RSS -> RX queue Q -> MSI-X / NAPI / TCP on CPU B
                                      -> wake / recvmsg / TLS / parser on CPU A

CPU A and CPU B should be different physical cores, preferably in the same last-level cache and NUMA node. This allows the kernel and parser to run concurrently, but it makes the socket hand-off and its cache traffic part of every packet’s path. Use the split when it improves the tail at the intended packet rate.

The controls need to express the same plan. RSS or hardware flow steering selects the queue; IRQ affinity selects where its interrupt and ordinary NAPI work run. irqbalance should be stopped.

We can verify this by generating traffic and verifing that the expected RX queue advances, that its MSI-X vector advances on the intended CPU in /proc/interrupts, and that network softirq activity appears where expected in /proc/softirqs.

isolation-first: NAPI busy polling

If the connection is continuously active and we are prepared to dedicate a core, NAPI busy polling changes the ownership boundary. The network owner on CPU A can run the queue’s NAPI poll from its receive or epoll path and consume the resulting socket data on the same CPU.

                         +-> busy-poll hit: NAPI / TCP / recvmsg / parser on CPU A
RSS -> RX queue Q -------+
                         +-> busy-poll miss: fallback MSI-X / NAPI on CPU A or CPU B

Busy polling does not permanently eliminate interrupts, so fallback placement is a real tradeoff. Keeping the IRQ on CPU A preserves locality when polling misses, which is usually better for sparse or bursty traffic, but an interrupt may pre-empt the parser. Moving the IRQ to CPU B protects CPU A from that interruption, but a miss now executes the kernel receive path on B and wakes A across cores.

There is no placement which simultaneously provides same-core fallback locality and freedom from interrupt interference.

Busy polling is associated with a NAPI instance, not exclusively with one TCP connection. A hot polling socket should therefore not share its RX queue with arbitrary noisy traffic. If one epoll instance relies on busy polling for several sockets, those sockets should share a NAPI ID.

place the consumer deliberately

The same decision appears after WebSocket parsing. If the handler’s work is bounded, keep parsing and the immediate trading decision on CPU A. That removes another hand-off from tick-to-trade. If downstream work is large or variable, make one explicit SPSC hand-off to a second physical core, preferably in the same cache and NUMA domain.

Where the machine or guest exposes meaningful NUMA locality, keep the RX queue’s interrupt CPU, network owner and hot receive state on the NIC-local node. Reserve separate housekeeping capacity for unrelated IRQs, timers, kernel workqueues, logging and control-plane work.

Comparing Implementations

The quantcplib client is the userspace implementation described above. Its nonblocking, epoll-driven socket receives ciphertext and the ENA hardware timestamp through recvmsg(). A custom OpenSSL BIO reads directly into OpenSSL’s destination, SSL_read_ex() writes plaintext into fixed application storage, and the incremental WebSocket parser consumes views over that storage. Parse completion is the synchronous message-callback entry, after UTF-8 validation and before any handler copy or JSON deserialisation.

The CPython 3.9.25 implementation uses a raw socket in timeout mode and calls recvmsg() to retain the same hardware timestamp. It writes the ciphertext into an ssl.MemoryBIO, drains TLS plaintext into a receive buffer, waits for a complete frame payload, assembles continuation frames, validates and decodes UTF-8, and materialises the completed Python str or bytes. Its measurement ends before JSON deserialisation and does not include an asyncio event loop.

I ran the two implementations on the same c8a.xlarge under the vanilla host configuration. Kernel tuning is deliberately excluded here and measured separately in the next section.

Each implementation contains five runs of 3,000 measured messages after 1,000 warm-up messages, for 30,000 measured messages in total. ENA’s PHC disciplined the system clock; every run began and ended within 500ns of the PHC, with a largest observed correction or offset of 346ns. All 30,000 measurements had valid clock ordering. The table reports the median of the five per-run percentiles.

Both implementations inherit the vanilla host’s large receive-path tail. quantcplib reaches 11.326us at p50 and 260.045us at p99; CPython reaches 27.479us and 281.584us. At p99, the kernel-side tail has largely swamped the userspace difference.

The marginal p50 segments separate the implementation cost from that tail. ENA RX to post-recvmsg() takes 10.541us in quantcplib and 11.031us in CPython. From recvmsg() to TLS plaintext, the figures are 0.510us and 2.610us; from TLS plaintext to parsed message, they are 0.330us and 11.300us. The socket boundary is nearly identical. quantcplib then preserves it through its custom BIO, fixed storage and incremental view-based parser, while CPython spends another 13.91us in MemoryBIO, buffered frame completion, UTF-8 decoding and Python object materialisation.

With Kernel Tuning

We can now return to the earlier kernel experiment itself. How do we go about tuning the kernel based on the discussions above? The kernel matrix used the same Tokyo c8a.xlarge with four vCPUs presented as four separate AMD EPYC 9R45 cores, with SMT disabled, one NUMA node, Amazon Linux kernel 6.18.44-99.149, and ENA driver 2.17.2g. Five 3,000-message runs were retained for each configuration, with 1,000 warm-up messages per run: 165,000 measured observations. All eleven stages were run in five randomised complete blocks.

Read more

NEW QUANT LECTURES RELEASE

https://lectures.hangukquant.com/courses/digital-options-mm

My first ever hft-series on the quant lectures platform - and it discusses in detail how I designed, implemented and deployed one of the most successful hft mm ops on crypto markets in Polymarket.

The lectures are information dense, and intense - 50-part lecture series lasting for 12 hours.

It is intended for a more technical audience with experience in programming. The lectures are released over 4 weeks with a tiered pricing model. Each release has limited seats and enrolment is First Come First Serve. Each successive tier will price the lectures higher.

Some of the lectures have been marked as preview-able.

Special Offers

• Annual and lifetime subscribers of research.hangukquant.com receive 500USD discount (the cost of an annual price plan) on the listed lecture.
  
  Monthly subscribers of research.hangukquant.com receive 100USD discount on the listed lecture.
  
  These discount codes will be sent to you separately.
  

• Unsubscribed readers or monthly readers who want to upgrade may first obtain the lecture pass and then reach out to me for complimentary upgrade on research.hangukquant.com. Existing users should use the codes sent to them instead.
  

• Lifetime subscribers of lectures.hangukquant.com will immediately get assigned seats in the lectures promptly and no action is required.
  

Lifetime pass is priced at 2400 USD until end of September. We will increase cost to 4200 USD once promotional period ends.

https://lectures.hangukquant.com/courses/lifetime

I’ve made 3 new tutorials with fully functional Python code backing my own market making workflows, including

1. Quantitative tick data lake management.

   This is how I manage my own raw flat data archives of market data, with data provenance for seqnos, observed jitter, etc. This allows downstream backtesting to retain “playback fidelity”, and replicate performance concerns such as back-pressure and jitter.

   https://quantpylib.hangukquant.com/tutorials/tick_data_archival/

   

2. Quantitative tick data simulation and backtests using data lake.

   This is the infrastructure that uses tick data and event journals from step 1 to try and replicate live trading behavior and performance. The objective is bipartite - first, to get as “true” of simulation to the live journal and secondly, to use the “true” simulation to test price discovery hypothesis.

   https://quantpylib.hangukquant.com/tutorials/market_making/

   

3. Live Trading, Diagnosis and Calibration of the Backtest Models.

   This is an iterative live trading process backed by steps 1 and 2. The live market making code is “unified” with the backtesting code, and live trading generates an action journal.

   https://quantpylib.hangukquant.com/tutorials/market_making/

All together, a live market making system generates live pnl, as well as observed actions such as fills, orders and positions. The tick data lake keeps an archive of the system’s data states. Later, the same strategy code is backtested against this data lake to generate simulated actions.

The simulated actions are compared with observed actions to report on backtest fidelity and tune queue models and improve backtest accuracy.

Analytics then run against both market data and observed actions to diagnose shortfall. Shortfalls are targeted by dev-iterations. The iterated strategy is run against the calibrated backtest to validate correctness and improvements. The iteration is then taken live.

Read more

I did get some requests to do a demo for a fully functioning mm system. Since crypto vol is picking up, I thought we shall write some code for a stink bidding market maker that targets slippage-insensitive taker flow such as liquidation orders. Code is in Python.

Read more

Read more

Note that terminal access pass cost increases in 3 days.

Queue Modelling in HFT Simulations

In HFT market making, running high-fidelity tick-data simulations is particularly challenging. It is easy to invent false artefacts around market microstructure which generate positive PnL but do not hold in practice. This is because we make assumptions around important considerations like latency, queue priority, market impact and so on.

At a high level, an execution simulation has to be concerned with:

1. When our order lands on the order book.

2. When our cancellation lands on the order book.

3. Our estimated queue position relative to exogenous market actions.

4. The impact of other traders reacting to our orders.

5. The endogenous impact of our orders on the simulated order book.

The first two are latency problems. Order-latency distributions may be backed out from our own order traces, with an appropriate sampling method. Similarly, an order remains live while its cancellation is in flight. Market events between these timestamps have to be applied against the correct exchange-side state.

The third is the usual queue-position problem: how much liquidity is ahead of us, and how does that quantity change as trades, submissions, modifications and cancellations occur?

The fourth is a much harder problem. Historical replay tells us how other participants behaved in the observed market. It does not tell us how they would have behaved had our simulated order been larger, smaller or absent. Modelling this properly requires an endogenous model of external order flow - we will leave that outside the scope of this post.

Our focus in this post is the theoretical basis of points 3 and 5. In particular, we will separate two ideas which are often mixed together under “queue modelling”:

• Impact modelling: how should our order change the displayed and matchable liquidity inside the replay?

• Position modelling: where should our order sit, and how should exogenous events change the quantity ahead of it?

Given different choices to queue models, we also want to know how we may tune or calibrate them from a statistical framework.

MBO vs MBP

When we have market-by-order data—MBO (L3)—queue reconstruction is relatively straightforward. The feed identifies individual order submissions, modifications and cancellations. Assuming price-time priority, we can reconstruct the orders at a level and insert our own order according to its exchange-arrival timestamp.

When an external order is cancelled, we know which order left the queue. We therefore know whether the cancellation occurred ahead of or behind us.

In crypto, we typically have market-by-price data—MBP (L2). The feed gives us aggregate quantity at each price but does not identify the orders making up that quantity.

Consider a level displaying ten units. The level later falls to six. We know that four units have left, but the L2 update does not tell us where those units were in the queue.

If they were ahead of our order, our position has improved by four. If they were behind us, our position has not improved at all. This is the crux of estimating queue priority and building queue models.

An optimistic model may remove it from ahead of us; a pessimistic model may remove it from behind us; a probabilistic model sits somewhere between the two.

Before we get to this position problem, there is a lesser asked but equally important question.

Suppose the historical L2 book displays ten units at a price and we submit a two-unit order. Should the simulated book now display twelve units, or should our two units be treated as part of the ten already observed?

There is no one-size-fits-all answer. It depends on our order size, the liquidity of the market and, importantly, whether we were already quoting on the book used to record the historical data. This is an orthogonal question to point 4—should our orders should be represented inside the public book?

The good news is that we can answer such difficult questions by tuning our queue models and parameters and comparing them against empirical results, thereby reducing the dimensions of our problem.

In QuantPylib, the L2 and L3 order books are mapped through a common data structure. QuantPylib’s replay logic implements the discussed policies.

We will discuss how synthetic L2 MBP data can be projected onto an L3 MBO book by separating liquidity into anonymous and private sizes, over which we may configure different queueing and impact policies.

Projecting L2 into an L3 State

Consider a single price level displaying ten units of liquidity. Our two-unit order lands at the same price.

There are two ways to project the new state.

Additive Impact

The additive model is simple. Historical public liquidity and our private liquidity are disjoint.

If the observed L2 quantity is L and our total private quantity is O, the matchable size is:

Our order is appended to the price level when it arrives. Subsequent L2 updates change the public quantity independently of our order. Cancelling our order removes its entire remaining size.

Borrowed Impact

The borrowed model is more challenging because public and private liquidity are no longer disjoint accounting buckets.

For one side and price, define:

• L: public size reported by the historical L2 feed;

• O: total size of our active private orders;

• B: private size borrowed from—and therefore already counted in—the historical public size;

• E: anonymous external liquidity represented in the synthetic L3 book;

• T: total physical quantity available to the matcher.

The following invariants must hold:

For a fully borrowed-impact model, we target:

and therefore:

Probably confusing, so let’s walk through some interesting cases.

Private size exceeds the observed level

Suppose the historical level displays three units and we submit an order for five:

The private order can borrow only the three units available in the public observation:

All three public units are represented by our order. The remaining two units are necessarily additive.

The cancellation behaviour is slightly less obvious. If we cancel the five-unit order, only its two additive units should disappear from the physical book. The three borrowed units still belong to the historical public observation and must return as anonymous liquidity:

before cancellation:
[ours: 5]

L = 3, O = 5, B = 3, E = 0, T = 5

after cancellation:
[anonymous: 3]

L = 3, O = 0, B = 0, E = 3, T = 3

Cancelling the private claim does not cancel the historical market.

Public size moves through private size

Now suppose our private order has size ten while the historical L2 level displays five:

L = 5, O = 10, B = 5, E = 0, T = 10

The public level then rises from five to eight:

L = 8, O = 10, B = 8, E = 0, T = 10

The physical matchable size does not change. Three more units of our private order simply become borrowed from the updated public observation.

If the L2 level subsequently rises to twelve:

L = 12, O = 10, B = 10, E = 2, T = 12

Our full ten-unit order is now borrowed, while the remaining two public units must be represented as anonymous liquidity that queues behind our owned orders.

The reverse case - suppose the public level falls from twelve to seven:

before:
L = 12, O = 10, B = 10, E = 2, T = 12

after:
L = 7, O = 10, B = 7, E = 0, T = 10

The private order remains physically matchable for ten units. The two anonymous units disappear, while three units of the private order lose their borrowed attribution and become additive.

The different impact models fundamentally discuss the philosophy of whether our simulated orders should be treated as additional liquidity, or as liquidity already represented in the observed market data.

Queue Position Policies

Once the impact policy has produced a synthetic L3 level, the queue policy determines how exogenous reductions change our position.

New public liquidity joins the tail. Historical trades consume liquidity from the head. The ambiguous event is an L2 size reduction: did the disappearing liquidity sit ahead of us or behind us?

Optimistic and pessimistic bounds

Consider a borrowed-impact book with ten public units. We submit an order for two:

[anonymous: 8] [ours: 2]

The public level then rises from ten to fifteen. The new liquidity joins the tail:

[anonymous: 8] [ours: 2] [anonymous: 5]

The level subsequently falls back to ten.

An optimistic, front-cancellation policy removes the five units from the head:

[anonymous: 3] [ours: 2] [anonymous: 5]

Our queue position has improved by five. A subsequent trade for five units consumes the three anonymous units ahead of us and fills our two-unit order.

A pessimistic, back-cancellation policy removes the five units from the tail:

[anonymous: 8] [ours: 2]

Our position has not improved. The same five-unit trade does not reach us.

The market data is identical in both cases. Only the queue policy changes.

Probabilistic queue models

The optimistic and pessimistic models are useful bounds. A probabilistic model allocates the reduction between liquidity ahead of and behind our order.

Let:

• A: anonymous quantity ahead of our order;

• H: anonymous quantity behind our order;

• C: observed L2 reduction.

Define the relative queue position:

A simple probability function is:

where:

The exponent γ > 0 controls the curvature of the cancellation-probability function.

All curves meet at x=0.5. When the anonymous quantity is split equally ahead of and behind our order, the model assigns equal cancellation probability to both sides regardless of γ.

For γ > 1, the curve becomes steeper around the midpoint. The model increasingly assigns the reduction to whichever side contains more liquidity.

For 0 <γ < 1, the curve flattens. Cancellation probabilities are pulled towards 0.5, making the model less sensitive to the relative amount of liquidity ahead and behind us.

In practice, the probability function depends on rather predictable factors such as distance from the top of book. We can generalize the probability function by fitting different curves across reasonable market variables.

These policy choices and parameters can then be calibrated against empirical order traces.

Calibrating Queue Policies and Parameters

We now have several defensible impact and queue models. The remaining question is which model best represents the market we are trying to simulate.

Calibrating directly against strategy PnL is problematic. Market-making strategies are state-dependent: one different fill changes inventory, which changes subsequent quotes, cancellations and order sizes.

The action paths quickly diverge, leaving us unable to determine whether a PnL difference came from the queue model or from the strategy.

Instead, we calibrate through a closed-action replay. We hold the live order actions fixed and allow only their simulated outcomes to change.

Trace and tick-data objects

Event journalling architecture in quantpylib was discussed here:

Consider an order trace containing:

• order and cancellation intents;

• submission and cancellation timestamps;

• side, price, size, order type and time-in-force;

• acknowledgements and terminal order states;

• fills, including their quantity, price and exchange timestamp.

From this trace, we may extract the exogenous action tape:

submit order A at t1
submit order B at t2
cancel order A at t3
submit order C at t4
...

The second object is the public tick-data stream. It contains the market events required to reconstruct exchange-side state:

• L2 snapshots and updates;

• trades;

• exchange and receive timestamps;

• …
  

The market data capture infrastructure in QuantPylib supports these artefacts.

For a candidate model θ, we replay the same action tape of order submissions and cancellations against the same tick-data stream.

In QuantPylib, the wall clock is swapped against the replay clock and the same trace architecture generates replay trace files. We can then compare the live trace fill against the simulated trace file in a queue replay harness.

We can therefore compare outcomes order by order and determine quantitatively the “fitness function of a queue model”.

Fill classification

Let I be the set of submitted order intents. Define:

as the set of orders which received at least one live fill, and:

as the set filled under candidate model θ.

Fill precision is:

This answers: when the simulation produces a fill, how often did that order also fill live?

Fill recall is:

This answers: how many live-filled orders did the simulation reproduce?

A combined F1 score is:

A pessimistic model may have excellent precision while missing many live fills. An optimistic model may reproduce most live fills while inventing many others.

Lifecycle accuracy

Let s_i and ŝ_i,θ be the terminal live and simulated states of order i, such as filled, partially filled, cancelled or rejected.

Lifecycle accuracy is:

This catches differences which a binary filled/unfilled label misses, particularly partial fills and cancellation outcomes.

Fill timing

For orders filled in both traces, let t_i and t̂_i,θ be the timestamps of the first live and simulated fills. Define:

A smooth timing score is:

The scale τ determines the timing tolerance. The score ranges from zero to one. A score of one represents identical first-fill timestamps, while values near zero indicate timing errors which are large relative to τ. Semantically, it quantifies how closely the simulated trace reproduces first-fill timing for orders filled in both traces.

Fill-price similarity

Let p_i and p̂_i,θ be the live and simulated volume-weighted fill prices for an order filled in both traces.

The price error in basis points is:

We can convert this into a bounded score:

where κ controls the price-error tolerance and:

weights the comparison by overlapping filled notional.

For passive single-price orders, this metric may carry little information: both traces often fill at the submitted limit price. It is more useful for aggressive orders, multi-level executions and partial fills across prices.

Cancellation races

Cancellation behaviour deserves its own comparison. Let:

indicate that order i filled while its cancellation was in flight. Define r̂_i,θ similarly for the simulation. Cancellation-race accuracy is:

This exposes a common source of optimistic replay: allowing the order to disappear when the strategy decides to cancel rather than when the cancellation reaches the exchange.

Trace similarity as a fitness function

The component metrics can be combined into a trace-similarity score:

with:

The weights should reflect the use case.

For calibration sessions D_train, we select:

The purpose is not to find the queue model which produces the highest backtest PnL. It is to find the model whose order-level outcomes most closely resemble the empirical trace.

thanks for reading. cheers!

We also discussed how can you natively manage your own tick data lake with Quantpylib:

This is the architecture that supports exchange replay fidelity - from when the exchange processed data, to when it dispatched (TX) it, to when you received (RX) it, giving you critical observability into your own trading stack’s processing back-pressure and latency figures.

There is no substitute for high grade fidelity replay for latency-sensitive analytics involved in quantitative trading - public data simply does not make the cut.

Today, we release the ability to perform session-replay on your own tick data lake in the QuantTerminal.

Obtain your access pass here.

Terminal access pass cost increases in 1 week. More on the terminal here.
Next release will have

• private orders, positions, fills display on live exchanges

• private orders, positions, fills replay on tick data with your own trace files.

Next post on the blog discusses the architecture and technical details to designing a high fidelity tick data backtest - soon to be released on quantpylib!

The last of the Quantitative Trading Strategy series pertains to hft mm. Of course, for obvious reasons - I am unwilling to share in detail the specifics. However, I do want to share a piece of market microstructural analyses between binance and polymarket crypto markets I did.

This article is abit of a special format - instead of written form, this is simply a snippet from Lecture 7 of my (yet to be released) hft lectures I have on the topic. Then some food for thought and tricks.

The specific figures there are no longer applicable due to the changing dynamics of the markets involved, but the techniques therein are applicable no matter and to orthogonal markets…

Read more

Currency desks are commonly divided by geography and market structure: G10, EMFX, LatAm, and so on. In FX, the policy regime plays a significant role in price discovery, and central bank leaning is one of the primary sources of price discovery.

Python Code for replication is provided at the end of the article.

In this series, we will be looking at trading SGD crosses. The MAS manages the Singapore dollar against a basket of currencies through the S$NEER.

The regime is normally described as basket, band and crawl:

• Basket: SGD is managed against the currencies of Singapore’s major trading partners rather than against one bilateral rate.

• Band: the trade-weighted index is allowed to move within an undisclosed policy band.

• Crawl: the slope of that band is reviewed and adjusted as the inflation and growth outlook changes.

These parameters are not disclosed, except for rough “policy guidance”. MAS may intervene through spot or forward FX transactions. Insofar as possible, market forces are otherwise allowed to determine where the currency trades inside the band.

I will leave the economic motivation of these policies as self-reading.

As you may imagine - for a number of business functions (such as fx hedging, speculation) in banks, hedge funds and whatnot - currency desks are undeterred in modelling these parameters.

We will perform such an exercise with public data and construct a trading strategy on a basket of tradable assets. Since MAS explains policy decisions in qualitative language, we will come up with a simple, algorithmic state machine.

In order to assess the quality of our policy recalibration, we will cross-reference with United Overseas Bank (UOB)’s 2019 and 2021 paper.

We will take a look at these subtasks:

• data preparation for statistical modelling

• performing numerical analysis through constrained regression

• algorithmic reconstruction of policy mid, band and width

• portfolio construction from proxy index

• portfolio construction on tradable basket

data preparation for statistical modelling

We require two datasets for the initial statistical model: the official S$NEER history and bilateral SGD exchange rates. Both are available for free from MAS.

The official S$NEER history can be downloaded as an XLSX file from the MAS exchange-rate statistics page. For our purposes the workbook contains two relevant fields:

date          sneer
1999-01-08    100.00
1999-01-15     99.75
1999-01-22    100.10

Each value represents the weekly average S$NEER for the week ending on the stated date. This gives us the dependent variable. We next need a set of explanatory FX rates.

daily FX data

Daily end-of-period exchange rates are available through the MAS API catalogue. The request requires a (free to obtain) API key and returns JSON records. We retain the following currency fields:

USD, CNY, EUR, MYR, JPY, AUD, GBP, IDR,
KRW, THB, CHF, TWD, INR, PHP, VND and CAD

This is our candidate universe, not a claim about the actual basket. The statistical model will determine which currencies receive weight. The raw columns use the following quote convention:

usd_sgd         SGD per 1 USD
eur_sgd         SGD per 1 EUR
jpy_sgd_100     SGD per 100 JPY
idr_sgd_100     SGD per 100 IDR

We normalise this to sgd_xxx convention.

A complete cross-section is available beginning January 2008. The FX data is daily while the official S$NEER series is weekly. Since the target represents a weekly average, we take the arithmetic average of each FX series over the same week.

If W_t is the set of daily observations belonging to week t, the weekly normalized FX level is

Let S_t denote the official weekly-average S$NEER level for week t. The resulting DataFrame has one row for each common observation week t=0,…,T. We denote this weekly level panel by

The first n columns, F, contain the normalized weekly-average FX levels. The final column, S, contains the corresponding official weekly-average S$NEER levels. In pandas, regression_df is simply the tabular representation of 𝒟.

constructing the basket through restricted least squares

The statistical problem is to infer a dynamic set of currency weights from the weekly level panel 𝒟=[F | S]. We use a weighted geometric basket.

Let S_t denote the official S$NEER level in week t, and let F_t,i denote the normalized weekly-average value of SGD against currency i. We model the changes by differencing the sneer value:

The residual varepsilon_t absorbs anything else not captured by our candidate universe.

Define the weekly S$NEER return

and the vector of bilateral FX returns obtained from the first n columns of 𝒟,

Stacking T observations gives

and

The regression can then be written compactly as

An unconstrained least-squares estimator would solve

This may fit the historical S$NEER well while producing an economically implausible basket. The resulting weights may be negative or may not sum to one.

We know more about the object than ordinary least squares uses. Basket weights should be non-negative, and the complete basket should carry 100% weight. We therefore solve the restricted problem

The feasible set is the probability simplex

The restrictions give the coefficients an immediate economic interpretation as basket weights.

numerical solution (skip if not interested in the math)

The squared-error objective

is convex, with gradient

We solve the constrained problem using projected gradient descent. From a feasible vector w_k, first take an unconstrained gradient step

then project it back onto the simplex:

The projection is itself the optimization problem

Its solution has the form

where θ is chosen such that

Intuitively, we subtract a common threshold from the unconstrained vector, set negative coordinates to zero, and choose the threshold so that the remaining weights sum to one.

We use the corresponding step size

and stop when successive projected weight vectors are sufficiently close:

rolling estimation

We estimate a separate basket for every observation week using the most recent m=260 weekly returns, or approximately five years. This produces a sequence of models rather than one final model, which are the dynamic weights of the basket across time:

predicting the S$NEER

A model indexed by week t contains the official S$NEER observation for that week. We shift each fitted weight vector forward by one observation through the preceding observation, ŵ_t-1, and multiply them by the current weekly FX-return vector r_t. The weighted sum is the predicted S$NEER log return. Applying that return to the previous official level gives

How well do the previously estimated basket weights explain the next official weekly move?

The actual and predicted levels lie almost on top of one another with a 0.972 correlation between the predicted and official weekly log returns, and a 5.07bps in weekly return RMSE.

The fit is strong enough to treat the estimated basket as a useful S$NEER proxy. We have a number of remaining parameters to estimate.

reconstructing the policy band

MAS would make a brilliant X finfluencer, because they are experts in vague-posting. MAS summarises monetary policy statements in roughly this form:

date          slope                 width                   centre
2018-04-13    Increase slightly     -                       -
2020-03-30    Set at 0%             -                       Re-centre downwards
2022-04-14    Increase slightly     -                       Re-centre upwards

Our job is to translate qualitative policy language into a deterministic state machine.

Let g denote the annualized slope of the midpoint and let h denote the log half-width of the band. An unchanged field carries its previous value forward. Every other phrase updates one component of the policy state:

• increase very slightly: g ← g+0.5δ

• increase slightly: g ← g+δ

• increase: g ← g+2δ

• reduce slightly: g ← max(0,g-δ)

• reduce: g ← max(0,g-2δ)

• set at 0%: g ← 0

• widen slightly: h ← 1.25h

• widen: h ← 1.5h

• restore narrower band: h ← h₀

• re-centre: reset the midpoint to the prevailing (predicted) S̃_d

The rules are symmetric. The zero floor prevents a sequence of reductions from implying a negative slope due to MAS slope policy since the outset.

We set one slope tick to δ=0.5% per annum, use a baseline log half-width of h₀=0.02, or approximately 2% on either side of the midpoint, and anchor the April initial slope at 2% per annum.

This is aligned with the UOB’s paper. Once these values are fixed, the same rules are applied across the entire time series; there is no additional parameter fitting.

evolving the midpoint

Let M_d be the policy midpoint on day d. Between re-centring decisions, its log level accumulates the prevailing annualized slope:

where Δτ_d is the fraction of a year elapsed since the preceding observation. On a re-centring date, we reset M_d to the prevailing reconstructed S$NEER and continue accumulating drift from the new base.

The upper and lower boundaries are then

Using a geometric band keeps both boundaries positive and makes widening symmetric in log-return space.

We have now constructed a parsimonious model calibrated against a deterministic state machine, and verified that results are somewhat in line with a reputable bank. If we are economist, we would be happy, but as traders, we make no money from this…

But how may we construct a continuous portfolio from this?

portfolio construction

Read more

• when an order was created locally and sent into the wire

• when the order arrived at the exchange

• when the order-ack was received locally

• and all other state transitions.

These artefacts can be used downstream for a number of important business functions such as

• latency analysis;

• slippage analysis;

• markout and adverse-selection analysis;

• tick-to-trade shortfall;

• trading-cost attribution; and

• reconciliation and incident replay.

Event journalling is a core component of replay fidelity. A replay recovers the order in which the live process observed and accepted state changes. A journal lets us rebuild those transitions.

We are running 1-week of discounts of access to Quantpylib:

In this article, we give a rundown of quantpylib’s native binary event-journal architecture and how to use it, including the Quantpylib Event Trace (QET) compiler. We will also show how quantpylib users can produce QET trace files directly from an OMS workflow.

Artefacts

In quantpylib, the following portfolio artefacts are journaled:

• order state changes;

• fill insertions;

• position deltas;

• order snapshots; and

• position snapshots.

We will use an order lifecycle as the running example. Consider the following HYPE limit order:

from decimal import Decimal


order = {
    "exc": "hyperliquid",
    "ticker": "HYPE",
    "amount": Decimal("1"),
    "price": Decimal("20"),
    "tif": "Alo",
    "cloid": oms.rand_cloid(exc="hyperliquid"),
}

await oms.limit_order(**order)

The OMS first constructs the exchange-native order wire. Before submitting that wire, it inserts the order into its in-memory Orders ledger with a CREATE_PENDING status and a local ts_submit_ns. At this point the client order ID (cloid) is known, but the exchange order ID (oid) usually is not.

When an update arrives through the private order WebSocket, the ledger resolves the order by cloid or oid and applies its staleness and identity checks. A successful acknowledgement will typically complete the identity with an oid and transition the order to NEW. A rejected submission transitions to REJECTED. Later updates may move a live order through PARTIAL and FILLED, or through CANCEL_PENDING and CANCELLED when we attempt a cancellation.

The diagram is deliberately simplified. The exact path depends on the venue and execution outcome.

For the CREATE_PENDING order above, the in-memory object and its trace artefact look like this:

S identifies the trading session, while N is the session’s trace sequence number. Based on the state machine, the appropriate events. such as trace.orders.insert, trace.orders.state_transition is created.

These actions occur in the trading hot path, so an inefficient transport would make the journal unusable. The trace artefact is compactly encoded as raw bytes and handed to quantpylib's high performance logging subsystem.

The transport is therefore a native C++ implementation adapted into byte buffers over the LMAX Disruptor pattern, exposed in Python.

Read the design here: Designing State-of-the-Art Logging in Python.

Insofar as the trading system is concerned, the trace is complete. This makes the trace subsystem a lightweight attachment (a normal logging statement) and non-intrusively composed with client applications.

QET Compilation

Once the logging hand-off is complete, the log file receives JSONL records of the following shape.

{"level": "INFO", "time": "2026-07...", "message": "trace", "event": "trace.orders.insert", "trace": "qwAAAAMAAAAHAAAA...VzAw"}

event identifies the accepted mutation. trace is the original binary artefact, base64-encoded. Trace records and ordinary application logs may share the same file.

An event journal can be attached directly to a live trading system. quantpylib instead uses the log files as the substrate and compiles QET files in a separate, cold process.

The trading process is finished once it hands the record to the logger; parsing, ordering, deduplication, and journal I/O remain outside the live system.

Many firms already operate some form of the Grafana logging stack. Grafana Alloy acts as the collector. The usual destination is Grafana Loki, which stores compressed log chunks and indexes their labels for later queries.

quantpylib piggybacks on this battle-tested transport. We configure Alloy to push the same records to our compiler's Loki-compatible endpoint:

loki.write "qet_compiler" {
  endpoint {
    url = "http://127.0.0.1:33200/loki/api/v1/push"
  }
}

This can be the sole destination or added to an existing Alloy pipeline.

The compiler is a small local HTTP service that implements the Loki push API. It decompresses each Snappy-compressed protobuf batch, recovers the original JSON line, filters trace records, base64-decodes the trace, and validates the QET schema and record length.

The Loki entry timestamp selects the UTC output day; the trace's sid and seqno is used for ordering and idempotence. The compiler then appends the original binary records to a daily .qet file.

Example

Here we will give a walkthrough of how to wire them together.

Install Alloy using Grafana’s platform instructions. The Alloy CLI command reference is here.

1. start the compiler

python3.11 -m quantpylib.hft.qet_compiler serve \
  --root ./qet \
  --host 127.0.0.1 \
  --port 33200 \
  --buffer-size 48

Check its counters from another terminal:

curl http://127.0.0.1:33200/health

2. start Alloy

examples/configs.alloy (quantpylib) points Alloy at an absolute log glob and at the local compiler. Point Alloy at the file written by the example in step 3:

export QUANTPYLIB_LOG_PATH="$(pwd)/logs/journal.log"

alloy validate examples/configs.alloy
alloy run --storage.path=./.alloy-data examples/configs.alloy

3. run the example

import os
import asyncio
import logging
from decimal import Decimal

from dotenv import load_dotenv

from quantpylib.hft.oms import OMS
from quantpylib.logger import Logger
from quantpylib.gateway.master import Gateway


load_dotenv()

exchange = "bybit"
ticker = "HYPEUSDT"
amount = Decimal("1")
price = Decimal("10")

async def main():
    gateway = Gateway({
        exchange: {
            "key": os.environ["TEST_BYBIT_KEY"],
            "secret": os.environ["TEST_BYBIT_SECRET"],
        },
    })
    await gateway.init_clients()

    os.makedirs("logs", exist_ok=True)
    logger = Logger(
        name="perf",
        stdout_register=True,
        filename="journal.log",
        logs_dir="./logs",
        file_level=logging.INFO,
    )

    oms = OMS(gateway, logger=logger)

    try:
        await oms.init()
        cloid = oms.rand_cloid(exc=exchange)
        res = await oms.limit_order(
            exc=exchange,
            ticker=ticker,
            amount=amount,
            price=price,
            cloid=cloid,
        )
        print(res)
        await asyncio.sleep(5)

        res = await oms.cancel_order(exc=exchange, ticker=ticker, cloid=cloid)
        print(res)
    finally:
        await oms.cleanup()
        logger.shutdown()

if __name__ == "__main__":
    asyncio.run(main())

In journal.log we see one insert and three state transitions, corresponding to create, create-ack, cancel, and cancel-ack. We also see snapshots, which allow recovery when a new session begins.

{"ts":"2026-07-26 17:15:25.642","level":"INFO","filename":"portfolio.py","line":1157,"msg":"trace","event":"trace.orders.snapshot","trace":"b'...'"}
{"ts":"2026-07-26 17:15:25.766","level":"INFO","filename":"portfolio.py","line":408,"msg":"trace","event":"trace.positions.snapshot","trace":"b'...'"}
{"ts":"2026-07-26 17:15:25.965","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.insert","trace":"b'...'"}
{"ts":"2026-07-26 17:15:25.998","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.state_transition","trace":"b'...'"}
{"ts":"2026-07-26 17:15:31.000","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.state_transition","trace":"b'...'"}
{"ts":"2026-07-26 17:15:31.093","level":"INFO","filename":"portfolio.py","line":1354,"msg":"trace","event":"trace.orders.state_transition","trace":"b'...'"}

Inspect the generated qet file here:

from pathlib import Path

from quantpylib.hft.qet import QETReader, QET_SCHEMA_REGISTRY


path = sorted(Path("qet").glob("*.qet"))[-1]
reader = QETReader(path)

for event in reader.decoded_records():
    event_type = QET_SCHEMA_REGISTRY[event["schema_id"]].partition("(")[0]
    print(event["sid"], event["seqno"], event_type)

which gives me the following traces:

92375019 0 QETOrdersSnapshot
92375019 1 QETPositionsSnapshot
92375019 2 QETOrder
92375019 3 QETOrder
92375019 4 QETOrder
92375019 5 QETOrder

I myself have used quantpylib’s crypto wrappers on dexs, prediction markets et cetera in latency sensitive workflows to generate more than $1M in market making pnl, and more than $1B in volume driving my quantitative research and strategies.

The library allows developers to assimilate into their own trading pipeline, allowing sophisticated traders to use it’s well-abstracted features at the appropriate layer, whether it is for risk management, offline research, or live trading.

Today, we announce the QuantTerminal, which gives a face to Quantpylib’s core features - this will give quant teams an interface and access to end-end applications of the best and most powerful workflows possible in Quantpylib.

Target Core features

• Live, multi-exchange Tick Data Terminal

• Live, multi-exchange position and Order Tracking

• Tick Data Lake Archival and Session Replay

• Binary Order Event Trace for HFT Market Making

• Quantitative Analytics for Order Execution (markouts, slippage, latency etc)

• Live Telemetry

• and more!

This is an internal quant tooling system we’ve been sitting on, and we have decided to do a staged rollout of it. We understand that quant teams need to own their IPs - code and data are sacred. Therefore, the terminal application is transparent software that is intended to be run locally on your own workstation - with your own data. It’s powered by quantpylib, therefore access to the library is a prerequisite.

First stage of the rollout is the live terminal view. Here is a short 30-sec demo.

Note that it’s in a beta-stage rollout, and priced as such. As we continue to rollout, we will increase the price of the access pass accordingly.

Access, Quantpylib 1 Week Discount

The QuantTerminal access pass is available here:
https://quantpylib.hangukquant.com/pricing/pricing/

For one week, we are running discounted access to Quantpylib. Running QuantTerminal requires access to the Quantpylib library.

I have traded more than $100 million in volume across these platforms, generating more than $1 million in PnL. My operations have primarily revolved around three strategy families:

• cross-exchange, combinatorial and statistical arbitrage;

• event-driven arbitrage;

• high-frequency market making.

The first family is somewhat capacity-constrained. Liquidity remains shallow across many nascent venues, and seemingly identical contracts are not necessarily fungible. Two exchanges may list economically similar questions while using different resolution sources, deadlines, wording or discretionary procedures. What looks like a locked arbitrage can therefore contain rather more basis risk. In fact, as I write this post, here is an obvious one between poly and hpl.

The second family is considerably more scalable, but requires serious thought around engineering processes. I ran reduced-form versions of these strategies in the past before eventually pausing them. Too many parts of the system were “non-automatable”: particularly around sourcing new contracts, registering new data sources et-cetera.

Recent improvements in agentic systems—particularly following GPT5.5 and Opus4.8 make the problem interesting again in my opinion.

The third strategy family, high-frequency market making, is the most scalable of the three. It also requires proper domain expertise, quantitative reasoning and a reasonably intimate understanding of exchange microstructure. This is what placed me near the top of the relevant leaderboards in terms of both volume and PnL.

We have, in fact, documented an entire lecture series detailing the design and implementation of my HFT ops, to be later posted here.

It will not be released until yours truly is no longer involved in said markets.

For now, we will focus on event-driven arbitrage. Our discussion here will take a more idealistic approach compared to what I ran, accounting for the scalability now possible under accelerated agentic capabilities. The objective is a pipeline capable of serving hundreds to thousands, of prediction-market contracts.

Read more

A factor model is motivated by the observation that market movements are primarily driven by a relatively small number of common sources of risk. Assets may have thousands of individual return streams, but much of their variation is shared: broad market movements, industries, economic characteristics, liquidity conditions, or latent statistical relationships. The purpose of a factor model is to separate these common movements from asset-specific noise.

There are three broad approaches: Fama-style (time-series) portfolio factors, statistical factor models such as PCA, and Barra-style (cross-sectional) exposure models. They differ not merely in interpretation, but in the direction of estimation, the information supplied to the model, and the quantity being estimated.

Getting timely exposure to desirable factors — and hedging undesirable ones — is a primary concern of smart-beta funds and are a TRILLION dollar industry.

Factor analysis in crypto is still nascent. This article focuses on Barra-style models and how I actively trade them to enhance my pnl.

The full implementation is included at the bottom.

We will touch lightly on the first two, and then detail a Barra-style analysis with code.

Fama-style

A Fama-style factor is defined as the return of a portfolio. For example, a simplified value factor might be

More generally, if (p_k,t) contains the asset weights of factor portfolio (k), then

Once these factor-return histories have been constructed, the exposure of an individual asset is estimated through time:

The factor-return matrix (F) is known. The regression estimates (β_i):

For a single, demeaned factor,

Fama-style models are motivated by economic hypotheses about expected returns. Is a manager’s apparent alpha simply compensation for exposure to known factor strategies? Is our portfolio (statistically) significantly just some beta against a factor portfolio?

For a portfolio with estimated factor betas (β_p),

Statistical-style

A statistical factor model begins with the complete return matrix:

where rows are dates and columns are assets. It seeks a lower-dimensional approximation:

where F ∈ ℝ^T×K contains the statistical factor returns and L ∈ ℝ^N×K contains the asset loadings. In principal components analysis, first estimate the asset covariance matrix:

of demeaned returns. Decompose it as

The first (K) eigenvectors define the principal directions:

and the corresponding factor scores are

The resulting rank-(K) reconstruction is

Unlike the Fama regression, however, neither the factors nor the loadings are specified economically.

These are motivated mainly by dimensionality reduction. If hundreds of assets share a few dominant covariance patterns, the researcher may want to discover those patterns without imposing characteristic taxonomy.

Here is a seminal paper on enhanced portfolio optimization using principal components by AQR’s Pedersen et al.

Barra-style

The focus of our post, is the Barra-style model. We will first discover it’s mathematical basis, and then use them to identify sector performance.

We begin by specifying each asset’s exposures. On date (t),

contains exposures to K factors, such as categories, industries, countries, or continuous styles such as size and momentum.

The model then examines the cross-section of asset returns on that date:

The exposures (B_t) are known. The factor returns (f_t) are estimated using the (N) assets observed on that date.

This regression is repeated for every date, therefore produces a time series of factor returns from a sequence of cross-sectional regressions.

Barra-style models are motivated by the need to describe the current economic composition of a portfolio. Rather than waiting for a long return history to reveal a portfolio’s betas, the manager declares or measures the underlying asset exposures directly.

Repeating the cross-sectional regression each day produces a synthetic return series for every factor. By extension, if sector inclusion is defined as a factor, we obtain the synthetic return of the sector performance.

They are useful to:

• portfolio managers performing return attribution;

• risk managers estimating common and specific risk;

• traders neutralising unwanted industry or style bets;

• isolating sector performance for smart beta plays.

The factor return has a different meaning from a Fama-style factor. It is the return attributed to one unit of exposure after controlling for the other exposures in the cross-sectional regression.

Example'; crypto Barra model

We will demonstrate the categorical Barra analysis here using a reduced four-asset problem with two categories. The full regression code can be found at the bottom.

Let’s begin.

Suppose our universe contains four assets:

• BTC — L1; daily return 4%; liquidity score 4.

• ETH — L1; daily return 2%; liquidity score 2.

• DOGE — Meme; daily return 8%; liquidity score 1.

• PEPE — Meme; daily return 6%; liquidity score 1.

The objective is to estimate three returns:

The market factor represents the weighted return of the crypto universe. The category factors represent L1 and Meme performance relative to that market.

1. Construct the return vector

For each asset, the daily close-to-close return is

Collect the four asset returns into the vector

This is the dependent variable in the daily cross-sectional regression.

2. Construct the regression weights

For each day, determine asset’s regression weight. Reasonably, we may choose the square root of its preceding 30-day dollar volume:

The weights are then normalised:

Our simplified liquidity scores are

Since they sum to eight, the normalised weights are

The WLS weighting matrix is simply diag(w).

BTC contributes half of the regression weight, ETH contributes one quarter, and DOGE and PEPE each contribute one eighth.

3. Construct the exposure matrix

Every asset has market exposure equal to one. It also has exposure equal to one to its assigned category.

The exposure matrix is

for columns [market, L1, Meme] and rows [BTC, ETH, DOGE, PEPE].

Each day, the regression equation is given by (solve for f)

For instance,

The model assigns the same fitted common return to assets in the same category. Differences between the individual asset returns in a fitted category return remain in the residuals.

4. Remove collinearity problem

The market column is equal to the sum of the two category columns:

Consequently, the exposure matrix does not have full column rank.

For a particular solution

and any constant a,

produces exactly the same fitted asset returns.

We may solve this by requiring the liquidity-weighted average category return to equal zero.

5. Reduction of Degree of Freedom

Let the aggregate weight of category c be

In our example,

We impose

Substituting the category weights,

The constraint vector is therefore

and the constraint is

The leading zero means that the market coefficient is not directly constrained.

6. Solving the constrained WLS

We minimise the weighted squared residuals:

subject to

Introduce a Lagrange multiplier λ:

The first-order condition with respect to f is

Rearranging,

The first-order condition with respect to λ is

Combining the two equations produces the KKT system:

For our example,

The weighted return vector is

The first element is the weighted return contribution of the whole market:

The second is the weighted contribution from L1 assets:

The third is the weighted contribution from Meme assets:

The complete system is

Solving gives

The constraint is satisfied:

7. Interpretations

The weighted return of the L1 category is

The weighted return of the Meme category is

The category coefficients are the differences between these category returns and the weighted market return:

The fitted common return for an L1 asset is

The regression therefore tells us:

• the weighted market returned 4.25%;

• L1 underperformed the market by 0.9167%;

And the same exercise applies to the Memes category.

The residuals describe the returns not explained by category membership.

For BTC,

8. Repeat the analysis through time

The same cross-sectional regression is repeated for every date t. Each regression produces one market return and one relative return for every represented category:

Stacking the daily estimates produces a synthetic return history for every category/sector.

For example,

is the estimated history of the crypto meme-sector relative to the weighted crypto market. If we take these as ‘sector returns’ and compound them from 1, we essentially have a synthetic sector index. This is (theoretically) tradable with a factor-mimicking portfolio.

9. Recover the factor-mimicking weights

The estimated coefficients are linear combinations of the asset returns. We may therefore recover asset weights that reproduce them exactly.

For the market factor, the mimicking weights are the normalised regression weights:

For category c, the mimicking weights are

For L1, this gives

These weights sum to zero:

Their realised return is

For Meme,

Its return is

This exactly reproduces the L1 factor return.

The category factor can therefore be interpreted as a long position in the weighted category basket and a short position in the weighted market basket.

The factor-mimicking portfolio exists exactly in the mathematics. Essentially, what we have done is to transform quantitative analysis from price space to factor space, using statistical categorical analysis. In the process, we might

• better appreciate crypto asset returns as a mixture of factors/sectors

• construct a more informed portfolio by reducing noise in our samples

For instance, using “theoretical” factor-mimicking portfolios, we may look to construct a long-short delta neutral portfolio for “bullish sectors” against “bearish sectors”. Or we may look to “enhance” our trend construction from the previous post to trading in the factor space, subject to same considerations of portfolio construction that were discussed.

Code:

Let’s take a look an example construction of the above test to a universe of hundred tickers from Binance and Python code.

Read more

IN my first quant internship, I was tasked with testing commodity futures skew effects.

pm: "how did you account for contract roll"

me: "treated them as continuous, sometimes it's contango sometimes backwardated, so. it's small approximation error"

pm: 😑

In trading, a lot of effects and strategies are known. Index rebalancing, stat arb, yield curve arbitrage, market making - are all known strategies. The devil is in the details, and like my old self, doing something and doing something right are completely different things. If you are targeting these strategies, fantastic! You are in the right arena. For those testing some inverse shoulder rsi combo schtick, it’s time to come around. No shame in where the journey starts.

This article is dedicated a little bit to that - when asked about how one should “start their quant journey”, I always default to this one - a fantastically simple strategy that opens up a ton of critical questions that expand to more complex strategies. For example

• Execution fees and optimal rebalancing

• Signal quantisation

• Volatility and square root laws

• Diversification

• Backtesting sanitation

• Signal transformation

• Risk targeting/management

• Order automation

It is also one of the strategies that is “slow enough” that can be handled manually, and where different components mentioned above can be automated “one-at-a-time”, which is the good old principle of abstraction and core principle of building complex systems.

Our next post will continue on factor trading and how I leverage ‘smart-beta’ strategies to enhance portfolio returns.

Shall we begin?

(Code is attached at the bottom.)

Signal is the easy part.

The high-level rule is simple: buy markets that are going up and sell markets that are going down. It gives us a strategy whose intuition can be stated in one sentence, and then there is everything else.

Until we specify how strongly to trade the signal, how much risk to allocate to each asset, how contracts are sized, when trades are executed and how costs are charged, it’s not a quantifiable test.

Trend following is therefore a useful first quantitative strategy for two reasons. It is simple enough that we can intuit the entire research process, while being rich enough to expose common lapses in framework and backtesting.

Moskowitz, Ooi and Pedersen document this effect across equity-index, currency, commodity and bond futures. A diversified portfolio of these signals produced returns with little exposure to conventional asset-pricing factors and tended to perform well during large market moves.

Let’s just say, there is a ton (two centuries) of evidence supporting trend following: A Century of Evidence on Trend-Following Investing and Two Centuries of Trend Following.

As they say, if it’s robust, simple does the trick. The simplest discrete signal is the sign of a trailing return:

The forecast is +1 when the asset has appreciated over the previous h periods and -1 when it has depreciated.

A moving-average crossover produces a similar binary rule:

Now imagine if you were thirsty, and you could only drink or not drink a litre at a time. That would be miserable. A continuous forecast tries to retain information about trend strength.

A continuous forecast distinguishes between a weak and strong trend. It can also move gradually through zero rather than jumping directly from fully short to fully long. Continuous rules based on the statistical strength of a trend can materially reduce turnover without a statistically significant performance penalty.

One possible construction is a volatility-normalised trailing return:

In more nuance, weak trends appear to persist, but the relationship eventually saturates. We know this from the Potters and Bouchaud paper. The appropriate forecast may therefore look less like an infinitely increasing linear function and more like a clipped or saturating one. A sensible forecast transformation is often saturating:

Can I Have One HubbaHubba of Egg Contracts? On Risk.

Said no-one ever.

Suppose Bitcoin and interest-rate futures both produce a forecast of +1. What does this even mean? Giving them the same dollar exposure makes no sense from an economic standpoint. A standard normalisation feature to encode a dimensionless forecast is the forecast annualised return volatility of asset i:

where b_i is an instrument or asset-class risk budget.

The inverse-volatility term gives smaller positions to volatile assets and larger positions to quiet assets. With binary forecasts and equal risk budgets, each instrument contributes approximately the same standalone volatility before correlations are considered.

Extending this framework, asset normalisation does not guarantee portfolio risk stability. 10 contracts long A and 10 contracts short B is a different portfolio from 10 contracts long in each. Instruments are correlated, correlations are dynamic, and the positions are too.

Let u_t be the vector of preliminary exposures and let Sigma be the forecast covariance matrix of asset returns. The forecast volatility of the preliminary portfolio is:

Given a portfolio volatility target, the final exposure vector is

scaling the whole portfolio so that its forecast volatility equals the target.

There are therefore two distinct pieces of risk management:

1. Asset-level normalisation prevents volatile instruments from dominating the portfolio performance.

2. Portfolio-level scaling adjusts total leverage based on dynamic correlations and the realised volatility of the combined strategy in relation to target.

Volatility targeting has a wider literature of its own. Reducing exposure during high-volatility periods improves the historical Sharpe ratios of several risk factors.

We Live in the Real World

For a linear contract, a simplified conversion from notional weight to units is

where A is the portfolio capital, P is the contract price, M is contract multiplier and X is the currency conversion. The position must then respect integer contract quantities, minimum order sizes, tick sizes, leverage limits and venue-specific margin rules. This may surprise some crypto traders, but you can’t always buy 0.0001 of $MONKEY coin in liquid futures markets where standardised contracts trade in (large) fixed sizes. Accordingly, this tracking error can play a part in ‘which universe of assets’ are sensibly tradable for hedge funds - even when AUM is minimally tens of millions! When generating backtests, we need to be careful about directionally correct and economically meaningless results.

Next on the chopping board of reality are fees. The cost function includes commissions, bid-ask spreads, slippage, market impact and carry/funding costs.

Thankfully, when doing longer-term trend following, fees are more…secondary class citizens. However, as you evolve in your career as a trader, or look at higher frequency trading, you, too, will come to obsess about fees. But for now, in this post, we can be abit more cavalier about it.

To be frank, cost management and proper backtesting principles probably deserve an entire series on its own, but that borders on the HOW rather than the WHAT, and I promised not to bore you with the how in this series. I do have an entire post on statistical inferencing for backtest validation, if you are interested.

So, for now - COST IS IMPORTANT, and non-negligible, and ideas from optimal control theory suggest that we should…have continuous signals but not continuous positions. This prevents us from “flip-flopping” our holdings and paying the broker or the market maker excessively. Fortunately, we can get 95% of the way there with 5% of the brain: by applying “positional inertia” - the inertia to trade into target position when the current position is sufficiently close to the target.

The “rough-ideal” for inertia levels can be set by setting average trading costs and varying inertia levels, and then simply doing a walk forward backtest to find the optimal tradeoff. If robustness is anywhere found, a reasonable band will appear. Generally, the more noise (or low Sharpe your cost-free strategy is), the larger should be your inertia threshold. If your signal is all-noise, then you should just sit on your hands and stop trading all together.

Code

Comparing with and without costs:

From a code perspective, it is critical from a research infrastructural standpoint to encompass all (or most) of the above concerns in a parsimonious framework. What separates novice traders from more sophisticated quantitative analysts is the research harness.

Read more

We developed a number of critical tips and tricks there that the quantitative trader should be aware of when profiling and deploying trading strategies. It was very well received, and a reader even changed their username to thenimblemm 😅

I’ve been writing this blog for 5 years now, and over the years - we focused mainly on developing quantitative research frameworks - on the how rather than the what. Here and there, we discussed some quantitative strategies, which talk about the what.

After half a decade, I thought it would be a nice series to give some more detail into WHAT goes on behind the scenes in my ‘dirty operations’.

From trend following, to factor trading, FX policy arbitrage, event driven trading, to cross-exchange statistical arbitrage and hft market making - these 5 years was a busy one!

7 Days Discount

For 1 week, all subscription plans are 50% off:

https://www.research.hangukquant.com/d7b4564d

You can also support me with a Lifetime Subscription here:

https://buy.stripe.com/dRm3cw0mC05N6Am3XKe7m0f

Shout out to the ~300 lifetime members who have supported me, love ya back.

Hope you look forward to the series!

This is available in quantpylib, our quant repo for annual, paid subscribers:

We have redesigned critical core architecture and subsystems that will enable powerful workflows for traders using quantpylib in their trading infrastructure. Examples include performant and institutional-grade data replay, event journaling, telemetry systems and quantitative analytics.

We are proud to announce that managing your data lake has been now made a ~100~ lines of code powered by quantpylib, into raw binary flat files.

We have also shipped and documented a bunch of improvements to our backend, including cpp implementations for native orderbooks states. Once we launch versioning, we will deprecate old APIs.

See our data archival example here:

https://quantpylib.hangukquant.com/hft/hft/#data-archival

See our feed model and QBN encoding notes:

https://quantpylib.hangukquant.com/hft/feed/

https://quantpylib.hangukquant.com/hft/l0/

Our new, performant native orderbook implementation is documented here:
https://quantpylib.hangukquant.com/hft/orderbook/

We continue our aim of providing institutional grade trading infrastructure available to serious traders.

Note:
We intend to increase quantpylib’s costs over the coming weeks with a change in access model. Existing users with access will see no change. Non-subscribers may obtain an access pass here.

One of the revamped subsystems was the improvement of data capture architecture towards more institutional grade archival policies. This article will be dedicated to describing what may constitute such a change.

Modern quantitative research is becoming less constrained by storage cost and more constrained by the quality of the artefacts we choose to preserve. Cheap cloud object stores, high-throughput local disks, and increasingly capable agents have changed the economics of retention. It is now cheaper to store, easier to dig, making data storage policies and approach correctness a step-change more important as an interface for humans and intelligent solutions to sift through valuable insights hidden in the data.

For a trading desk, the highest-value artefacts are the operational record: market data, order traces, fills, gateway logs, replay journals, risk decisions, and the metadata that ties these streams together. Agents are good at sifting, ranking, joining, and summarising. But they are only as useful as the substrate underneath them.

Read more

And as of late, we have completely revamped the archival subsystem (to a custom L0 binary capture), and am in the process of rewriting the native orderbooks, as well as data replay systems. I am also working on a sidecar terminal application so that traders/market makers with strategies powered by quantpylib have institutional grade trading infrastructure backing them.

Anyhow, since I was on the topic, today, we are going to do a light(er) article on implementing an orderbook. Previously, we already spoke on this topic:

While that was a pretty simple and intuitive discussion, much more detail can be assigned to this task. It is also a common question at quant dev interviews, and I have recently had quant friends drilled on these.

Extending from our previous discussion, if we would like to extend an orderbook implementation into something matching-engine compatible and full-depth, the book needs to support queue position. That means the interface is no longer only MBP (market by price), but rather an MBO surface (market by order).

When it comes to L3 implementation, there are roughly two schools of thought. The standard text of reference is David Gross’s CppCon talk, When Nanoseconds Matter: Ultrafast Trading Systems in C++.

Some of the design pressures in this post are motivated by that discussion. Here we will provide some clarity and implementation on Gross’s solution and also expand on the crypto’s common L2 domain, and how we can support both interface efficiently without template overhead.

Read more