One of the revamped subsystems was the improvement of data capture architecture towards more institutional grade archival policies. This article will be dedicated to describing what may constitute such a change.

Modern quantitative research is becoming less constrained by storage cost and more constrained by the quality of the artefacts we choose to preserve. Cheap cloud object stores, high-throughput local disks, and increasingly capable agents have changed the economics of retention. It is now cheaper to store, easier to dig, making data storage policies and approach correctness a step-change more important as an interface for humans and intelligent solutions to sift through valuable insights hidden in the data.

For a trading desk, the highest-value artefacts are the operational record: market data, order traces, fills, gateway logs, replay journals, risk decisions, and the metadata that ties these streams together. Agents are good at sifting, ranking, joining, and summarising. But they are only as useful as the substrate underneath them.

Read more

And as of late, we have completely revamped the archival subsystem (to a custom L0 binary capture), and am in the process of rewriting the native orderbooks, as well as data replay systems. I am also working on a sidecar terminal application so that traders/market makers with strategies powered by quantpylib have institutional grade trading infrastructure backing them.

Anyhow, since I was on the topic, today, we are going to do a light(er) article on implementing an orderbook. Previously, we already spoke on this topic:

While that was a pretty simple and intuitive discussion, much more detail can be assigned to this task. It is also a common question at quant dev interviews, and I have recently had quant friends drilled on these.

Extending from our previous discussion, if we would like to extend an orderbook implementation into something matching-engine compatible and full-depth, the book needs to support queue position. That means the interface is no longer only MBP (market by price), but rather an MBO surface (market by order).

When it comes to L3 implementation, there are roughly two schools of thought. The standard text of reference is David Gross’s CppCon talk, When Nanoseconds Matter: Ultrafast Trading Systems in C++.

Some of the design pressures in this post are motivated by that discussion. Here we will provide some clarity and implementation on Gross’s solution and also expand on the crypto’s common L2 domain, and how we can support both interface efficiently without template overhead.

Read more

As a market maker with more than 100m+ of volume and leading PnL on the platform, the optimizations I’ve made to the SDK is now available to all quantpylib users. Websocket integration is also complete.

The optimizations include:

Read more

This post, we will discuss the introduction of a state of the art performance Python logging subsystem in quantpylib, and discuss some of the key design principles that allow us to achieve this. To my knowledge, among all Python logging frameworks, it is the lowest latency implementation out there.

As an aside, I am focused on making quantpylib into a more mature platform for serious quants. Beyond my expectations, the repo has found its way into many retail traders’ workflow, and even some small institutional trading stacks.

The project, although originally intended to be ‘my personal infra’ - is now bigger than myself, and accordingly, we will look to adopt higher standards appropriate to the gravity of its use cases. Although it is a best-effort guarantee service, the best-est of the effort will be brought forth.

Let’s get started!

A taster: (on log scale), quantpylib’s logger is 30-70x faster than standard logging and microsoft’s picologging library!

Logging is an important part of system observability. It gives us a way to reconstruct the system states, what decisions it made, and where things failed in live systems.

However, logging is not free. A log invocation may require caller discovery, record construction, string formatting, structured field handling, handler traversal, locks, stream writes, flush behavior, and sometimes exception or stack formatting. In latency-sensitive tasks such as trading, this cost can sit directly in the hot path.

In Python, reducing this cost is difficult because the usual logging path is intentionally dynamic. Python objects are allocated, frames may be inspected, dictionaries are populated, handlers own locks, and formatting often happens before the caller can move on. This is not the shape we want for threads parsing market data and executing time-sensitive actions.

To alleviate these issues, a number of existing libraries try to improve different parts of the logging problem. A couple examples -

• picologging is designed as a faster, mostly drop-in replacement for the standard library logging module. This is useful if the main constraint is compatibility with the existing logging API, while moving parts of the implementation into faster native (C) code.

• loguru attacks a slightly different problem. It makes logging much easier to configure and use: sinks, serialization, contextual binding, lazy formatting, and nicer ergonomics. Its lazy=True path can defer evaluation of expensive arguments until the message is actually emitted. Emitted log still travels through Loguru's event construction, formatting, and sink pipeline.

Although these are good libraries, most of them still preserve the broad log-record shape: create an event-like record, attach metadata, decide when the message is constructed, and push the result through a sink.

They differ in how much work is deferred and whether message construction happens directly in the hot path, but the core abstraction is still a dynamic log record carrying metadata through a logging pipeline. The end result is that log invocations cost up to microseconds. Where we are targeting t2t in the microsecond teens and below, observability is a hefty price to pay.

Our objective, therefore, is to be able to generate structured, useful logging in Python applications - that can be aggregated and analysed downstream in logging pipeline(s) such as as for ingestion into a Loki database.

In order to target this objective, we would like to ask ourselves the following questions:

• what components of log invocation add to the hot path?

• how do we reduce the number of Python object allocations in that path?

• which part of the logging workflow is actually costly: call-site lookup, record construction, argument formatting, handler traversal, serialization, or I/O?

• do we walk the Python frame stack to recover metadata such as filename, pathname, function name, or line number, and if so, how often?

• what concurrency mechanism makes logging thread-safe?

• if the logger acquires a lock, where is that lock acquired and how expensive is it under contention?

• does the caller thread format strings, or can formatting be deferred?

• does the caller thread call into the kernel for file or stream writes, or is that pushed into a writer thread?

• is the path synchronous with the sink, or does the caller only publish a compact record and move on?

Standard Path

A normal call like:

logger.info(”fill %d @ %f for %s”, qty, px, symbol)

can involve a surprisingly long sequence of work:

1. Enabled-level check: the logger has to decide whether the event is filtered out by the logger level, handler level, or effective inherited level.

2. Caller discovery / frame inspection: the logger may inspect Python frames to recover metadata such as filename, pathname, function name, and line number.

3. LogRecord construction: the logger creates a Python object representing the event.

4. Attribute population: the record is filled with message arguments, timestamps, process/thread metadata, exception information, stack data, and user-provided extra fields.

5. Handler traversal: Python logging is hierarchical, so the event may walk through logger and parent-handler control flow depending on propagation settings.

6. Filters: each logger or handler may run user-defined filters before the record is emitted.

7. Locks: handlers need synchronization because multiple threads may write into the same logging sink.

8. Formatting: the message string, structured fields, timestamp, exception text, or JSON representation are constructed.

9. Stream writes and flush behavior: the final bytes are written to stdout, a file, socket, or another sink, possibly crossing into the kernel or blocking on I/O.

The logging libraries including the ones we mentioned improve parts of this path, but many still operate in the same conceptual shape: create an event object, attach dynamic fields, render it, and pass it through a logging pipeline.

To break this latency floor, we have to remove pieces of the constraint rather than merely make each piece a little faster. The useful observation is that most log invocation sites are only partially dynamic.

For example, in printf-style logging invocations, the static information usually includes the format string, file name, line number, severity, number of arguments, and rough parameter types. The dynamic information is much smaller: the values passed into that invocation, occasional extra fields, and the timestamp.

These points carry forward into the implementation’s optimization techniques.

Techniques

For the specific implementation, refer to the quantpylib codebase. The high-level design is easier to discuss in terms of the techniques applied.

static information registry

The first technique is to treat a log invocation site as something that can be registered once and then referred to cheaply.

The Python-facing fast path resolves the caller into a key based on the CPython code object and line number. Implementation-wise, this goes through CPython frame objects: get the current frame, recover the PyCodeObject, read the line number, and combine the code-object pointer and line number through a small custom hash. The result is a per-thread cache key for that call site.

On a cache hit, the logger gets back a stable integer id. On a cache miss, the format string and frame metadata are registered into a static registry: pathname, filename, line number, severity, parameter count, rough parameter types, and an unload/formatting function. This id uniquely identifies the log invocation site.

Instead of constructing a fresh record that contains all metadata, each emitted record can contain a pointer to the static data plus only the dynamic payload:

struct StaticData {
  const char* pathname;
  const char* filename;
  const char* format;
  uint32_t line_number;
  LogLevel severity;
  int num_params;
  const ParamType* param_types;
  UnloadFn unloadfn;
};

struct LogEntry {
  const StaticData* static_data_ptr;
  uint64_t timestamp;
  uint16_t entry_size;
  uint8_t reserved[6];
  char args[];
};

There is a deliberate Python-specific constraint here that we enforce, which is that format strings should be stable literals.

For example, the intended shape:

log.info("fill %d @ %f for %s", qty, price, symbol)

is remarkably distinct from

log.info(f"fill {qty} @ {price} for {symbol}")

In the first case, the format string is a literal object associated with the call site. In CPython, the compiled code object holds constants in its constant table, and repeated execution of that call site loads the same Python string object from that table. Object identity is represented by the object’s address for its lifetime, i.e. what id(obj) exposes at the Python level. The logger uses this idea at the native layer by recording the PyObject* pointer for the format string. For repeated invocations of the same literal-style call site, that pointer remains stable.

In the second case, Python constructs a new string value dynamically, so the format string’s address is no longer a stable part of the call-site key. If the same source line starts producing different string objects, the logger cannot safely say that the cached id still refers to the same static logging schema, so it rejects the call.

This is a limitation we enforce so that the call-site cache is meaningful.

transportation layer

After the call-site cache resolves the static metadata id, the next problem is transportation: how do we move the dynamic log payload across the producer/writer thread boundary without turning the log invocation into a normal synchronized I/O operation?

The backend borrows from the LMAX Disruptor design. The important optimizations in that family of designs are as follows:

1. Preallocation: allocate the ring buffer upfront so memory can be reused instead of dynamically allocated on every log invocation.

2. Lock-free producer hot path: producers writing to the ring buffer avoid a lock/mutex on the latency-sensitive path, which removes the handler-lock shape of conventional logging.

3. Sequence caching: the producer memoizes the observed consumer sequence, amortizing the core-to-core cache-coherence traffic that would otherwise come from producer/consumer progress tracking.

Additionally, beyond the algorithmic improvements, the transportation layer makes optimizations designed to run well on modern hardware architectures:

4. Cache-line aware sequence counters: sequence counters are aligned and padded to cache-line boundaries to reduce false sharing.

5. Power-of-two indexing: ring positions can be computed with a bit mask instead of modulo arithmetic.

We generalize the LMAX Disruptor idea from a ring of fixed event objects to a ring over raw byte buffers.

The producer and the writer/consumer agree on a raw byte encoding for log entries. Instead of communicating Python objects or fixed event slots, the producer writes log entries as contiguous bytes using this encoding scheme, and the writer later decodes the bytes into formatted output. This matters because log records are not naturally fixed-size. A no-argument message, a three-argument printf-style message, and a message with dynamic extra fields have different byte footprints. Instead of forcing them into a fixed slot size, the logger reserves the exact number of bytes required for:

• the LogEntry header;

• the pointer to static metadata;

• timestamp and entry size;

• packed printf-style arguments;

• other fixed and variable length fields.

Each producing thread owns a thread-local log buffer. Conceptually, this turns the logging transport into multiple SPSC-style queues: each producer publishes into its own buffer, and the writer side consumes from those buffers. This reduces producer-side contention because producers are not fighting over the same write sequence.

The writer thread (in a non critical path) is then responsible for merging log entries from the thread-local buffers. In the implementation, this is done with an appropriate ordering structure such as a priority heap over timestamps. The writer side also decides where the log output should go, when writes should be flushed, and what policy should govern the generation of the actual log lines.

Further, the writer side can be pinned to a housekeeping core and can perform the heavier work of turning byte records into the desired output format, such as JSON or K=V logs. This not only defers formatting and heavy I/O away from the hot path, but also helps keep the producer working set smaller and friendlier to L1/L2 cache behavior, which matters on modern architectures.

Benchmark

Based on the techniques, here are the benchmarks, and the benchmarking code: (note that exceptions call Python runtime’s stack trace formatting/utilities before calling into the logger framework)

File Sink

=========== sink=file ===========

--- file / no_args ---
variant                   p50        p90        p99          max
stdlib                  6417n      7834n     15000n      129125n
loguru                 14792n     16796n     25750n     1213250n
picologging             5375n      6875n     13293n      110333n
structlog               3917n      4458n      8500n     1334334n
quantpylib-perf          167n       208n       375n       56708n

--- file / 1xint ---
variant                   p50        p90        p99          max
stdlib                  6667n      8500n     15418n      127875n
loguru                 15125n     18042n     29792n      706875n
picologging             5458n      7500n     13667n      131833n
structlog               4542n      4958n     10918n      108833n
quantpylib-perf          167n       208n       292n       10417n

--- file / 3xmixed ---
variant                   p50        p90        p99          max
stdlib                  6625n      7838n     14376n      114458n
loguru                 14875n     15542n     21250n     1903917n
picologging             5500n      5833n     11042n      124791n
structlog               4584n      4792n      8458n       28208n
quantpylib-perf          167n       209n       375n        1208n

--- file / extra ---
variant                   p50        p90        p99          max
stdlib                  7417n      8087n     14501n       29000n
loguru                 15791n     16417n     22000n     1438458n
picologging             5625n      6834n     12583n      112959n
structlog               4542n      4709n      6875n       26708n
quantpylib-perf          292n       334n       750n        1916n

--- file / exception ---
variant                   p50        p90        p99          max
stdlib                 37375n     44834n     68668n     1062792n
loguru                 49083n     60042n     97542n     3616500n
picologging            29458n     32458n     50125n      233792n
structlog              30416n     32167n     40626n     1301708n
quantpylib-perf        22834n     23459n     32667n       78333n

--- file / filtered ---
variant                   p50        p90        p99          max
stdlib                   125n       125n       375n        9083n
loguru                   459n       583n      1500n       24000n
picologging               42n        84n       334n       19958n
structlog               1042n      1084n      1458n       14458n
quantpylib-perf           42n        83n       167n       23667n

Stdout Sink

--- stdout / no_args ---
variant                   p50        p90        p99          max
stdlib                 11000n     13167n     37167n      553916n
loguru                 23833n     28083n    499468n     4076250n
picologging             9167n     10583n     20500n    56333375n
structlog               6083n      8625n     15459n      356625n
quantpylib-perf          167n       500n      1250n       34167n

--- stdout / 1xint ---
variant                   p50        p90        p99          max
stdlib                 11209n     13042n     23877n      197000n
loguru                 23833n     27541n    498848n     5344250n
picologging             8750n     10213n     14792n      212500n
structlog               6709n      8667n     13375n      109334n
quantpylib-perf          167n       542n      1041n       14333n

--- stdout / 3xmixed ---
variant                   p50        p90        p99          max
stdlib                 10959n     12750n     19750n      250958n
loguru                 24250n     29379n    530761n     3686250n
picologging             8958n     10458n     17375n      180542n
structlog               7250n      9042n     18668n      159709n
quantpylib-perf          208n       416n       709n       16166n

--- stdout / extra ---
variant                   p50        p90        p99          max
stdlib                 12958n     15084n     42048n      573750n
loguru                 25583n     29708n    527216n     3164916n
picologging             9042n     10542n     17590n      190875n
structlog               7917n      9625n     17209n       88417n
quantpylib-perf          292n       708n      1208n       40584n

--- stdout / exception ---
variant                   p50        p90        p99          max
stdlib                 68917n     89379n    553959n     3580625n
loguru                 83875n    109721n    816045n     6636292n
picologging            46125n     51625n     66709n     1413083n
structlog              63375n     82712n    591252n     3555583n
quantpylib-perf        24083n     39379n     63922n     1239959n

--- stdout / filtered ---
variant                   p50        p90        p99          max
stdlib                   125n       167n       541n       13500n
loguru                   500n       542n      1125n      119708n
picologging               42n        84n       125n        1042n
structlog               1167n      1625n      9542n      671791n
quantpylib-perf           42n       167n       334n        5292n

Bench Code:

import os
import sys
import time
import logging
from pathlib import Path

import numpy as np
import matplotlib.pyplot as plt
import quantpylib.logger as qlogger

LOG_FILE = "bench.log"
ITERS = 10_000
PAUSE_MICROS = 10

WORKLOADS = [
    "no_args",
    "1xint",
    "3xmixed",
    "extra",
    "exception",
    "filtered",
]
SINKS = ["file", "stdout"]

def invocations(fn, iters, pause_micros):
    fn() #warmup
    latencies = []
    for _ in range(iters):
        t0 = time.perf_counter_ns()
        fn()
        latencies.append(time.perf_counter_ns() - t0)
        if pause_micros:
            time.sleep(pause_micros / 1_000_000)
    return latencies

void_log = lambda logger: lambda : logger.info("hello")
int_log = lambda logger: lambda : logger.info("x=%d", 42)
mixed_log = lambda logger: lambda : logger.info("fill %d @ %f for %s", 42, 3.14, "btc")
extra_log = lambda logger: lambda : logger.info(
    "fill %d @ %f for %s",
    42,
    3.14,
    "btc",
    extra={"venue": "binance", "tag": "maker", "oid": 1234},
)
filtered_log = lambda logger: lambda : logger.debug("x=%d", 42)

def exception_log(logger):
    def log():
        try:
            1 / 0
        except Exception:
            logger.error(
                "error",
                extra={"exc": qlogger.get_exception(), "stack": qlogger.get_stack()},
            )
    return log

def bench_workloads(logger, iters=ITERS, pause_micros=PAUSE_MICROS):
    return {
        "no_args": invocations(void_log(logger), iters, pause_micros),
        "1xint": invocations(int_log(logger), iters, pause_micros),
        "3xmixed": invocations(mixed_log(logger), iters, pause_micros),
        "extra": invocations(extra_log(logger), iters, pause_micros),
        "exception": invocations(exception_log(logger), iters, pause_micros),
        "filtered": invocations(filtered_log(logger), iters, pause_micros),
    }

def _attach_handler(log, sink, handler_factory):
    for h in list(log.handlers):
        log.removeHandler(h)
    handler = handler_factory(sink)
    log.addHandler(handler)
    log.propagate = False

class LoguruAdapter:
    def __init__(self, logger):
        self._logger = logger

    def debug(self, msg, *args, extra=None):
        self._log("DEBUG", msg, args, extra)

    def info(self, msg, *args, extra=None):
        self._log("INFO", msg, args, extra)

    def error(self, msg, *args, extra=None):
        self._log("ERROR", msg, args, extra)

    def _log(self, level, msg, args, extra):
        logger = self._logger.bind(**extra) if extra else self._logger
        logger.opt(lazy=True).log(level, "{}", lambda: msg % args if args else msg)

class StructlogAdapter:
    def __init__(self, logger):
        self._logger = logger

    def debug(self, msg, *args, extra=None):
        self._log("debug", msg, args, extra)

    def info(self, msg, *args, extra=None):
        self._log("info", msg, args, extra)

    def error(self, msg, *args, extra=None):
        self._log("error", msg, args, extra)

    def _log(self, method, msg, args, extra):
        event = msg % args if args else msg
        kwargs = extra if extra else {}
        getattr(self._logger, method)(event, **kwargs)

def bench_stdlib(sink, log_file, **workload_kwargs):
    log = logging.getLogger("stdlib")
    log.setLevel(logging.INFO)
    def factory(sk):
        h = logging.FileHandler(log_file, mode='a') if sk == "file" else logging.StreamHandler()
        h.setLevel(logging.INFO)
        h.setFormatter(qlogger.JSONFormatter())
        return h
    _attach_handler(log, sink, factory)
    return bench_workloads(log, **workload_kwargs)

def bench_loguru(sink, log_file, **workload_kwargs):
    from loguru import logger
    logger.remove()
    output = log_file if sink == "file" else sys.stdout
    logger.add(output, level="INFO", serialize=True, enqueue=False)
    return bench_workloads(LoguruAdapter(logger), **workload_kwargs)

def bench_picologging(sink, log_file, **workload_kwargs):
    import picologging
    log = picologging.getLogger("picologging")
    log.setLevel(picologging.INFO)
    def factory(sk):
        h = picologging.FileHandler(log_file, mode='a') if sk == "file" else picologging.StreamHandler()
        h.setLevel(picologging.INFO)
        h.setFormatter(qlogger.JSONFormatter())
        return h
    _attach_handler(log, sink, factory)
    return bench_workloads(log, **workload_kwargs)

def bench_structlog(sink, log_file, **workload_kwargs):
    import structlog
    output = open(log_file, "a") if sink == "file" else sys.stdout
    structlog.configure(
        processors=[
            structlog.processors.add_log_level,
            structlog.processors.JSONRenderer(),
        ],
        wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
        logger_factory=structlog.WriteLoggerFactory(file=output),
        cache_logger_on_first_use=False,
    )
    try:
        return bench_workloads(StructlogAdapter(structlog.get_logger("structlog")), **workload_kwargs)
    finally:
        if sink == "file":
            output.close()

def bench_pylogger(sink, log_file, **workload_kwargs):
    from quantpylib.logger.logger import Logger
    log_path = Path(log_file)
    log = Logger(
        name=sink,
        register_handlers=[],
        stdout_register=(sink == "stdout"),
        stdout_level=logging.INFO,
        file_register=(sink == "file"),
        filename=log_path.name,
        logs_dir=str(log_path.parent or Path(".")),
        file_level=logging.INFO,
    )
    return bench_workloads(log, **workload_kwargs)

def bench_cplogger(sink, log_file, **workload_kwargs):
    from quantpylib.logger import Logger
    log_path = Path(log_file)
    log = Logger(
        # name="perf", defaults to perf
        stdout_register=(sink == "stdout"),
        stdout_level=logging.INFO,
        stdout_formatter_cls=qlogger.JSONFormatter,
        file_register=(sink == "file"),
        filename=log_path.name,
        logs_dir=str(log_path.parent or Path(".")),
        file_level=logging.INFO,
        file_formatter_cls=qlogger.JSONFormatter,
    )
    results = bench_workloads(log, **workload_kwargs)
    return results

def print_table(workload, results, sink):
    print(f"\n--- {sink} / {workload} ---")
    print(f"{'variant':<18} {'p50':>10} {'p90':>10} {'p99':>10} {'max':>12}")
    for variant, data in results.items():
        lats = np.asarray(data[workload])
        print(f"{variant:<18} {np.percentile(lats, 50):>9.0f}n {np.percentile(lats, 90):>9.0f}n {np.percentile(lats, 99):>9.0f}n {np.max(lats):>11.0f}n")

def plot_sink(sink, results):
    cols = 2
    rows = int(np.ceil(len(WORKLOADS) / cols))
    fig, axes = plt.subplots(rows, cols, figsize=(12, 4 * rows))
    for ax, w in zip(axes.flat, WORKLOADS):
        for variant, data in results.items():
            ax.plot(data[w], label=variant, alpha=0.6)
        ax.set_title(w)
        ax.set_xlabel('Iteration')
        ax.set_ylabel('Time (ns)')
        ax.set_yscale('log')
        ax.legend()
    for ax in axes.flat[len(WORKLOADS):]:
        ax.set_visible(False)
    fig.suptitle(f'logger latency: sink={sink}')
    fig.tight_layout()

def main():
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('-sink', '--sink', type=str, help='Specify logging sink = [file, stdout]', choices=SINKS, default='file')
    parser.add_argument("--log-file", type=str, default=LOG_FILE)
    parser.add_argument("--iters", type=int, default=ITERS)
    parser.add_argument("--plot", action=argparse.BooleanOptionalAction, default=True)
    args = parser.parse_args()
    sink = args.sink
    
    if os.path.exists(args.log_file):
        os.remove(args.log_file)

    all_results = {}
    print(f"\n=========== sink={sink} ===========")
    variants = {
        "stdlib": bench_stdlib(sink, args.log_file, iters=args.iters),
        "loguru": bench_loguru(sink, args.log_file, iters=args.iters),
        # "quantpylib-root": bench_pylogger(sink, args.log_file, iters=args.iters),
        "picologging": bench_picologging(sink, args.log_file, iters=args.iters),
        "structlog": bench_structlog(sink, args.log_file, iters=args.iters),
        "quantpylib-perf": bench_cplogger(sink, args.log_file, iters=args.iters),
    }
    all_results[sink] = variants
    
    for w in WORKLOADS:
        print_table(w, all_results[sink], sink)

    if args.plot:
        plot_sink(sink, all_results[sink])
        plt.show()

if __name__ == "__main__":
    main()

The Lighter SDK is designed to be both performance aware and intuitive, presenting a significantly improved interface over the official package. See full docs here:

https://quantpylib.hangukquant.com/wrappers/lighter/

and our market making tutorial here:

https://quantpylib.hangukquant.com/scripts/market_making/

We will update our polymarket SDK to follow the v2 integration soon.

Cheers!

As a continuation of the hip-3 perp dex integrations,

quantpylib now features Databento’s live dataset integration with the same init-subscribe-handler pipeline:

See docs and example script here:

We have also improved the order tracking features of our OMS utility library. To learn how to experiment with market making in just 300 lines of code with quantpylib, see our tutorial here.

I sincerely appreciate each and everyone of you.

Anyway, I received a handful of DM’s recently asking for ‘special arrangements’ for access to read the work. I am very flattered, but I thought I shall share the link publicly.

Let’s start with this very blog. I’ll create the other links to other resources such as lectures and repo when there are enough people asking for it.

So, 7 Days, 40% off the Lifetime Access.

https://buy.stripe.com/dRm3cw0mC05N6Am3XKe7m0f

That’s all for this post. As promised, Lighter gateway will be integrated over the coming days.

Look forward to having you…. for the next 5 years.

A Cat.

That is practically the network latency of getting a market data payload from Binance through a cdn, which you might be doing if you are a mm anyway.

We can do much better.

In instances where orders are not preemptively signed, order signing lives directly on the hot path. Hyperliquid’s signing path (from signing.py) is a pipeline of canonicalization and hashing

EIP-712 encoding

and finally, secp256k1 signing.

Optimising this pipeline is a simple matter of appreciating the schema of an EIP712 payload, Ethereum’s structured-data signing format. The signer is given a typed message with 4 pieces:

• domain: identifies the application / signing domain

• types: declares the schema of the message

• primaryType: the top-level struct being signed

• message: the actual field values

The eth-account takes encode_typed_data of these components and produces a SignableMessage. On each order sign, it hashes the type definition of the EIP712 domain, performs ABI validation, obtains the domain separator, computes the Agent type hash - all of which are static.

The dynamic fields go into a tiny component, the connection-id and agent hash, which is part of the final digest:

keccak(”\x19\x01” || domain_separator || struct_hash)

a 32-byte digest signed by secp256k1. Between parsing the schema, walking the dictionary fields, ABI encoding and object creation/allocation - most of which are static per order - our hot path is not so hot anymore. If we can surgically stick to hashing only the dynamic components, our working set is a lot smaller.

Last but not least, significant performance degradations arise when we rely on native eth-account backend to perform signing. Elliptic curve signing relies on mathematically nontrivial operations, requiring large-integer modular arithmetic and scalar multiplication.

Following the ‘everything is object’ design, simple arithmetic such as ‘+’ requires namespace lookup, operator dispatch, pointer chasing, heap allocation and refcount churn in Python - cryptographic operations implemented at the Python layer is terrible news for low latency code.

Wow, Everything is Computer - President Donald Trump

Wow, Everything is Object - Python Engineer

Again, appreciating this truth is 90% of the job done - we can find solutions to push the cryptographic work from Python runtime into a secp256k1 backend optimised in C, such as the bitcoin-core library. Writing a tiny Python binding around it is the remaining 10%…oh wait, it already exists.

Bringing it all together, we can now replace our sign_l1_action with sign_l1_action_fast.

from coincurve import PrivateKey

_EIP712DOMAIN_TYPEHASH = keccak(b"EIP712Domain(string name,string version,uint256 chainId,address verifyingContract)")
_DOMAIN_SEPARATOR = keccak(
    _EIP712DOMAIN_TYPEHASH
    + keccak(b"Exchange")
    + keccak(b"1")
    + (1337).to_bytes(32, "big")
    + b'\x00' * 32
)
_AGENT_TYPEHASH = keccak(b"Agent(string source,bytes32 connectionId)")
_SOURCE_A = keccak(b"a")
_SOURCE_B = keccak(b"b")

def sign_l1_action_fast(ccwallet, action, vault_prefix, nonce, source_prefix, expires_after):
    data = msgpack.packb(action)
    data += nonce.to_bytes(8, "big")
    data += vault_prefix
    if expires_after is not None:
        data += b"\x00" + expires_after.to_bytes(8, "big")
    connection_id = keccak(data)
    agent_hash = keccak(_AGENT_TYPEHASH + source_prefix + connection_id)
    digest = keccak(b"\x19\x01" + _DOMAIN_SEPARATOR + agent_hash)
    sig = ccwallet.sign_recoverable(digest, hasher=None)
    return {"r": to_hex(sig[:32]), "s": to_hex(sig[32:64]), "v": sig[64] + 27}

class Hyperliquid:
    def __init__(self...):
        ...
        self.ccwallet = PrivateKey(self.wallet.key)
        self._vault_prefix = b"\x00" if self.vault_address is None else (b"\x01" + address_to_bytes(self.vault_address))
        self._src_prefix = _SOURCE_A if self.is_mainnet else _SOURCE_B

and there we go! Using a faster implementation of cryptographic signatures is most of the speedup, giving us 0.29-0.32ms in the same experiment. Using the pre-computed cache values gives a further 4-fold speedup, @ 0.08-0.09ms.

And that wasn’t even that hard…a simple trick in the Python Houdini

This is a RT of an article I wrote on X.

Exactly 2 years ago, on this date - I took a step into the treacherous crypto order books. My first foray into crypto trading was a quasi systematic funding arbitrage strategy, which I executed by hand using some spaghetti code dashboard to filter for funding pairs on binance-hyperliquid.

1 to 10 Sharpe

In those days, funding arbitrage was already a well known strategy, but execution, variation and reach played a big factor in realised APR. New exchanges seemed to be budding left and right, with unique funding mechanisms.

This pushed me to quickly build out more exchange connectors and a gateway functionality that would allow me to automate funding arbitrage strategies across multiple exchanges in a unified interface, which would later turn out to be the quantpylib project, a code repo started by my followers and then later lead by me to be a tool for both quantitative research and trading.

At this point, other projects like ccxt and hummingbot existed, but the quantpylib project focused more on connectivity and composability of features without relying on an architectural framework, allowing traders with variable strategies to implement high to low frequency strategies according to their needs. It integrated binance, bybit, woox, paradex and lighter, exchanges I executed arbs across. 2.5 years later and 1000+ commits later, quantpylib is still a work in progress.

I had decent reach and okay execution, enough to land me in the 40% APR range, while earning points - which ironically turned out to be even more lucrative than the pnl itself, all thanks to The El Jeffe crew. These days, with tools like @LorisTools, the scale and reach of such arbitrage strategies are extensive.

Despite the ~10 Sharpe, it was a strategy with extremely low vol, and scaling the pnl meant I had to put more of my already small net worth on chain, a risk I was not willing to take. It also took many man hours of attention, taking time away from investing in my own technical capabilities. In young, 'skilled' workers, knowledge compounds over the remaining career lifespan.

I shut it down.

At the same time, I was in a desperate struggle with my own mental faculties. I was, at heart, a builder. I built things in public, in an industry where large doses of skepticism were warranted. I had the imposter syndrome bug. Was I just a peddler of snake oil? Some people (including those I admired) and the anti-substack mob seemed to think so.

Hanguk’s income is from his newsletter. All he can do is implement a momentum trading strategy. I find it funny that Hanguk teaches systematic funding arbitrage but has to record pnl manually.

You write on a blog, I do this for a living, I would know.

I saw it in X replies, Discord channels, Reddit communities. Maybe I am just a fucking fraud, and I will never be that guy. After all, trading is an incredibly competitive industry, and greater man than me have failed. I am still young, and I can still pivot into being a quant trader where all my friends are. I could get referrals, I would be ‘in the industry’ and I won’t have to deal with all these bullshit.

These dark thoughts would later crescendo into anxiety and panic attacks. By all counts, my ‘online blog’ was bringing me 200k+ in ARR, and my trading was comparably rewarding. I was living my life. I once had no income and no pnl, and had an unshakeable confidence that I would ‘figure it out’. Having somewhat figured it out, it turned out that there were many cracks in this facade. Crypto strats were going to be squeezed by TradFi, and AI was going to wipe me out, who needs ‘research’ when everyone had a PhD-expert on their fingertips?

Wakeup, Doomer

The truth is, there were many people supporting me. For every nasty comment, there were ten others who cheered my work on.

Claude Mythos, OpenGod or whatever - I was well equipped and adaptable enough to thrive in a post-intelligent economy. I spent some time talking to my Dad, who had the wisdom of life to slap the doomer out of me.

It’s all in your head.

I got over it by biasing towards action. Day after day, I would wake up in sweats and palpitations, and before the darkness paralysed me, I would have done a handful of pushups, hopped in the shower, and was headed out to write some code. If I kept moving, it couldn’t catch me. It took a few months, and slowly but surely, I was okay again.

Fear is good marketing. Do not be the product.

10 to 50 Sharpe

In the months to follow, I would have the most productive schedule of my life.
I had always wanted to go from mid-frequency to high-frequency strategies. I wanted to be knowledgeable in niche domains and be a respected engineer. There was a lot more to learn, and my mind was liberated.

I worked on hft research, tooling, infrastructure and trading. I built new features on quantpylib, picked up C++, sharpened my mathematical knowledge and was allowed to be dumb again.

I was new, and publicly so. It had been awhile, but I was in love again. I read, I experimented, I asked and made mistakes. I basked in the wealth of knowledge. I didn’t have a PhD at my fingertips to replace me. I had one to teach me.

The best way to learn is by doing. In my frenzy of learning, researching and experimenting, I arrived at my next gold mine. Months of research, screaming at the monitor - I got my next reward, my first (successful, I had many unsuccessful ones) HFT strategy:

I consistently topped the leaderboards, fin-fluencers were writing nonsense threads about me, and I felt ... vindicated.

Sunset

When the sun rises, it must set.

There’s no such thing as the goose that lays the golden egg forever. - Jim Simons, Renaissance Technologies

Nowadays, I battle with the immortality of the goose. I hesitated if I should write this article. I’m hanging in there, but I am no longer a leaderboard resident.

As the sun sets, I am dumb again. But there is more iron in me.

Following our last update -

The full list of new websocket channels, hip-3 dex integrations, example script and documentation has been pushed.

You may refer to them here:
https://quantpylib.hangukquant.com/wrappers/hyperliquid/

We are looking to complete integrations for Lighter and Aster DEX in the next updates.

Cheerios

Read more about quantpylib here:

and docs here:

https://quantpylib.hangukquant.com

As promised, we are continuing the building on quantpylib and integrating it with new features, platforms and improved performance. On this patch, we have updated the Hyperliquid gateway. The last major update on the gateway included seamless integrations with spot asset trading - since then the Hyperliquid eco has had tremendous amount of new features, including new margin systems, HIP-3 DEX integrations, no-op fast cancel paths, twaps, builder codes and so on.

All of them are now available in our hyperliquid gateway wrapper.

Docs and example scripts to follow.

Future integrations will also include other platforms and (tradfi) data sources that have RWA streams.

Happy Trading!

We then talked about how the insurgence of crypto and blockchain matching platforms provide small quant teams and even individuals to deploy high sharpe strategies in pseudo-competitive markets due lower barriers and fragmented liquidity.

The nature of these budding platform often have attendant issues in one form or another, including behavioural and performance ones. These issues also often take weeks to months/years to get ironed out, some despite already having been flagged.

Whether it is engineering inaptitude (monolithic stack), prioritisation (these platforms often have a multitude of pressing concerns and distribution objectives), awareness (the issue is not recognised by leadership) or negligence (aware but don’t care), often flaws exist.

In some instances, you can shout at them “THIS ISSUE IS COSTING YOU HUNDREDS OF THOUSANDS” and they will give flaky answers, leaving you to wonder if you should have told them in the first place or taken advantage…

This was a conversation in the past year from [REDACTED] crypto platform, where the reply to my ticket was how much money they had raised!

In most cases, the issue is more subtle, such as performance. Undocumented API endpoints, wrongly documented endpoints, undocumented parameters and so on. These can be rectified to some extent by looking at the developer tools (Chrome F-12) network payloads and observing the message traffic.

In other cases, everything appears as is, and no behavioural flaws are apparent. However, if you are looking to onboard a platform and play a significant role in liquidity provision, it is absolutely necessary for you to understand the in-and-outs of both the functional and behavioural kinks of interfaces presented by the exchange.

For example, a cancel order functionality may be available via these features:

Read more

One of the things I like to do when my friends hop firms is to ask what the focus of the interview was, the skills involved et cetera. I was rather surprised that Python still has a good seat in the software stack of crypto trading firms, and not in the “dashboard and risk” layer. In the actual trading logic.

Anyways, I did come across scenarios for my own (low latency) trading purposes, where I wanted to optimise feed parsing but the entire system was written in Python. It is a pretty neat trick, the gains are nice (~10x), use case is diverse and implementation is simple - fantastic to share.

Let’s go!

Read more

Article explores the Performance Monitoring Unit (PMU), its role in code profiling, and introduces a lightweight Python-based controller for direct PMU access. While demonstrated in Python for clarity, the underlying syscalls are straightforward to port to C or C++.

In the next post we will showcase using tools written here - to benchmark a custom bba parse against orjson.

Read more

On a tangent, other techniques/agreements such as micro benchmarking, hardware timestamping, mm contracts come to mind as important. On a rather precariously long list of items, I want to break them down by layer.

My philosophy around this research/blog is if it’s useful to me, it’s useful for someone else. I wanted a nice compilation as a personal handbook. Think it will be pretty useful for a lot of readers, hence I will compile them in the market notes down the line.

This post is dedicated to the (linux) kernel component.

Read more

These profiles give us a better latency characteristic of a trading system that has queue depth ~ O(1). However, we know that market data is bursty. We are interested in what affects the shape of the latency distribution under load.

Under load, latency also needs to take into consideration queuing time in the pipeline. In effect, this queuing is incurred at all the network hops, as well as the system’s NIC, kernel processing stages etc…here we only look at app-layer queues. Bursty feeds push arrival rate above instantaneous service capacity and force backlog; once backlog exists, latency becomes backlog-dependent.

Read more

Previously, we discussed the lock-free ringbuffer implementation, showcasing a canonical port of the Java Disruptor implementation from LMAX.

An implementation https://github.com/hangukquant/disruptor_cpp

You can also refer to our market notes on cache friendly design:

and see the quant notes on the implementation details:

This post is dedicated to running a simple benchmark of the Disruptor’s latency profile. We begin with a simple benchmarking code, and then improve it to have a better appreciation of the ringbuffer’s profile. I credit the post to @ Bytes Smith was kind enough to share his insights with exceedingly clear explanations and code. Whether you are programming in Python, Java, or C++ - this is worth reading with a piece of paper: it is a useful exercise for thinking about your own benchmarks and (very possibly) quantitative interviews.

A short description of the Disruptor

At a high level, the Disruptor is a preallocated, cache-friendly ring buffer paired with a sequencer that hands out monotonically increasing sequence numbers. Producers “claim” a slot (a sequence), write the payload into that slot, then “publish” it. Consumers advance their own sequences as they process events, and producers use gating sequences to avoid overwriting data that has not been consumed.

Key pieces:

• RingBuffer: fixed-size array, typically power-of-two sized for cheap indexing (seq & (N-1)).

• Sequencer: coordinates claims/publication and enforces backpressure via gating sequences.

• SequenceBarrier: lets consumers wait until a sequence is available (published).

• WaitStrategy: controls how consumers wait.

• EventProcessor / EventHandler: consumer loop + callback.

This diagram and a short code snippet below should be enough to explain its usage in low latency inter-thread synchronization patterns.

Code snippet for a simple Google Benchmark:

Spin up a consumer thread to read events published to the ringbuffer. Set timer.

For 1 million events, repeat:

• Producer claims the next sequence from the ringbuffer (next()).

• Producer writes the payload (value = i) into the claimed slot.

• Producer publishes the sequence to make it visible to the consumer.

• Producer waits for the consumer sequence to catch up (i.e., all events consumed).

• Stop timer and record the delta.

Issues

Depending on the workload at hand, this benchmark can be either useful or nearly meaningless in the context of trading systems. In trading systems, objects synchronized across “thread boundaries” might include market data payloads, logging messages, watchdog flags, risk checks, internal signals, etc. These have very different performance questions.

Depending on the task, different observable metrics should be prioritized:

• Throughput / average throughput: e.g., backpressure on a heavy logging system. We don’t want backpressure because it drops messages or stalls the producer, possibly in the hot path.

• Invocation latency: e.g., “how much does logging in the hot path add to t2t?” Here service time downstream is less important as log aggregation downstream is batch processed and not time sensitive.

• Message hand-off latency: the t2t hot path question—“how long until another core can act on this critical information?”

• In-flight distribution: jitter and tail latency. Queue time vs service time effects latency shape and burst behavior under load/volatility.

In the benchmark displayed, the code is primarily a steady-state throughput in a hot pipeline and not a direct measurement of handoff latency per message. It is closer to “max pipeline rate under ideal conditions”.

Consider these:

1. Batching effects hide per-message latency

   • Disruptor consumers often drain available sequences in a tight loop. Even if you “publish one-by-one”, the consumer may observe bursts due to scheduling, cache effects, and barrier semantics.

   • If your real handler does parse + compute + branch-y logic, the consumer won’t look like this at all. Memory access patterns are less predictable; cache hit rates would be significantly lower in practice.

2. Payload realism

   • A real market data payload might be on the order of hundreds of bytes, often touching multiple cache lines and performing parsing that creates unpredictable memory access patterns.

   • A single int64_t write is a best-case scenario for cache locality and store-forwarding behavior.

3. Market data “information entropy” is heavy-tailed

   • Consider BBA / incremental updates: many messages are qty-deltas or otherwise non-impactful to fair value and quoting decisions.

   • The consequential message triggers quote updates and often coincides with bursts: more inbound messages, more in-flight backlog, more cache churn, and longer effective queue time.

   • Average throughput under a benign stream is a poor proxy for “what happens when it matters”.

4. CPU affinity / topology dominates the result

   • With no core affinity, the OS may schedule producer/consumer:

     • on the same physical core (time-slicing),

     • on SMT siblings (shared execution resources),

     • on different cores sharing parts of cache hierarchy,

     • or (worse) across NUMA nodes.

   • Each case changes the coherence traffic pattern and latency regime. If you care about “handoff across cores”, we must control placement.

In any case, we shall question if the trading system in question requires synchronization across separate NUMA nodes. Lastly (assuming the two threads land on separate physical cores), there is true parallelism and real time (elapsed time) is a better metric than cpu time (which aggregates across threads).

In the worst case, for the benchmark in question, since the handler is a no-op - there is potential for dead code elimination, optimizing the store-op of the payload.

Improvements

We fix the benchmark in view of the issues presented. Further, to approximate single payload handoff latency (usually what we care about for a “one packet” hop), we record a latency sample by:

1. Producer writes into ring A and publishes.

2. Consumer reads it and immediately echoes it back via ring B.

3. Producer waits for the echo, computes RTT in cycles, and divides by two as an approximation of one-way handoff latency.

This “ping-pong” pattern deliberately measures something closer to the cache-to-cache transfer + sequencing overhead + barrier/wait overhead,
under a controlled topology.

The noop handler is replaced with meaningful but low-overhead logic. First, some meta-functions:

Here, measure_tsc_freq_hz reads the TSC (time-stamp counter), a monotonically increasing cycle counter on x86. _mm_lfence() is used as a serialization barrier to reduce out-of-order effects around RDTSC. Sleeping for 100ms and multiplying by 10 provides an approximate per second TSC frequency.

You can refer to this post for more notes on the TSC.

pretty self explanatory… and finally the benchmark code:

In the next post we will discuss a slightly different benchmark to analyse a trading system under load.

Cheers!

In the first post, we discussed where latencies come from in cloud-hosted infrastructures for crypto hft mm. We discussed how ALBs can add jitter

and how to find the best ‘front door’. Upon finding the right endpoint (if any) - we discussed AWS box placement strategies and search heuristics to increase probability of reducing the physical distance to server racks, such as using partition and spread placement groups.

We want to round this up with some more AWS-native notes, and talk a little bit on the importance of network topology and writing a custom gateway for multi-asset/regional arbitrage. I wanted to include some notes regarding on-prem server tuning, but I think it’s not really crypto-specific MM so I will put that in a separate post.

Read more

These week, we cover some more advanced topics in the area of latency optimisation. We are currently doing a week long thousand dollar discount for our research platforms:

Continuing the thread on optimization…

Read more