In my series on nimble market making - we talked about how important it was to measure and profile:
This applies also critically to software optimisation. One of the most important things to do in a lean engineering/trading team is to operate on the pareto frontier between performance optimisations and strategy discovery. As an extension, one can spend all the wrong time focusing on micro-optimisations when there are significantly more problematic bottlenecks.
Other than the network topology, algorithmic design and architecture - one of the most high value work comes at the level of the kernel. More precisely, it is about getting the kernel and userspace programs to cooperate.
I previously wrote a broader (and simpler) Linux kernel tuning guide. Here, we shall do something more concrete with the actual optimisations and code examples - do refer to the quantcplib repo for the actual scripts.
We will profile the ingress latency of a low-latency websocket client in quantcplib against a CPython implementation, measuring the latency between the AWS ENA RX hardware timestamp to a parsed WebSocket message. We will play around with different “settings” and discuss how they affect performance, with a soft, service level objective (SLO) of attaining ~10microseconds in ingress latency.
A Short Userspace Description
At a high level, in userspace, the receive path follows:
A nonblocking TCP socket calls
recvmsg(), receiving ciphertext and the ENA hardware-timestamp control message together.A custom OpenSSL BIO reads directly into the destination supplied by OpenSSL.
SSL_read_ex()writes plaintext into fixed application storage.An incremental RFC 6455 frame parser consumes views over that storage.
An incremental message parser handles fragmentation and UTF-8 validation.
The message callback runs synchronously on the same network-owner core.
ENA DMA-writes into driver-managed receive buffers and recvmsg() still copies bytes from the kernel into userspace. The custom BIO removes an additional userspace ciphertext staging copy: we do not receive into our own temporary buffer and then write the same ciphertext into an OpenSSL MemoryBIO. This is sometimes described as zero-copy. The timestamp is attributed to the ciphertext receive which enabled the completing plaintext chunk. This is the fastest path for a data packet where the kernel is still responsible for ip/tcp processing and tls is performed in userspace -
Before Userspace
Before the network runtime can parse a WebSocket frame, an Ethernet frame must travel from the ENA device into the TCP socket.
The first decision is receive steering. ENA calculates an RSS hash from the packet headers and uses its indirection table to select an RX queue. One TCP connection will normally remain on one queue: additional queues distribute different flows.
At this point, the driver has already populated that queue’s RX submission ring with descriptors pointing to host receive buffers. When the packet arrives, ENA DMA-writes the packet into one of those buffers and posts a completion descriptor to the corresponding completion queue. The payload is now in host memory, but TCP has not processed it.
On the ordinary path, the next step is an interrupt. Interrupt moderation determines how quickly ENA raises the queue pair’s MSI-X interrupt after completions arrive. Deferring the interrupt allows several packets to be handled together, amortising interrupt and driver work; it also makes the first packet wait for the batch. This is known as interrupt moderation. A fixed rx-usecs value sets this delay directly, while adaptive moderation changes it according to recent traffic.
When the MSI-X interrupt is delivered, ENA automatically masks the queue interrupt and the hard-IRQ handler schedules the queue’s NAPI instance. The handler does very little packet work itself. The NAPI poll is normally serviced through the NET_RX softirq and consumes RX completions until the queue is empty or its packet budget is exhausted.
For every completion, the driver identifies the posted receive buffer and constructs an sk_buff, the kernel’s packet representation. The driver then submits the skb through napi_gro_receive(). The ENA driver documentation gives the concrete descriptor and skb path; the Linux NAPI documentation describes how the poll is scheduled and bounded.
GRO is the first important batching control inside that poll. When enabled, it can merge compatible packets from the same flow into a larger skb before the upper layers process them. This reduces repeated IP/TCP work and improves throughput, at the cost of batching and the bytes later observed by the application. With GRO disabled, packets proceed separately and the kernel pays more per-packet work.
The IP layer then processes the network packet, and TCP validates sequence state, handles reordering and makes contiguous stream bytes available on the socket receive queue. Once the socket becomes readable, the kernel wakes a task blocked in epoll_wait(). Epoll returns a readiness indication—not the bytes themselves—to the network runtime, which attempts a nonblocking recvmsg() and copies the available ciphertext, together with its ancillary timestamp data, into userspace.
Let’s highlight the available controls in this data path:
RSS and IRQ placement determine which queue receives the flow and which CPU handles its MSI-X interrupt and initial NAPI work.
Interrupt moderation determines how long a completion may wait before that interrupt is delivered.
The NAPI budget limits how many packets one poll cycle processes before yielding.
GRO determines whether compatible receive packets are combined before IP/TCP processing.
Two adjacent controls affect latency without changing this packet logic. Holding /dev/cpu_dma_latency open with a zero request constrains CPU idle-state exit latency through PM QoS; despite the name, it does not accelerate DMA. mlockall() and prefaulted buffers reduce disruptions from page faults and reclaim after the process is awake.
An alternative path: NAPI busy polling
An interrupt does not have to be the event which starts NAPI. With NAPI busy polling, an eligible socket read or readiness wait spends a bounded interval asking the kernel to poll the socket’s associated NAPI context before falling back to the MSI-X interrupt. If the completion arrives during that interval, packet processing can begin from the application’s syscall context without first paying the interrupt-delivery and task-wakeup path.
Everything after that entry point remains the ordinary kernel network stack. NAPI still consumes ENA completion descriptors, constructs skbs, passes them through GRO and IP/TCP, and places bytes on the socket receive queue. It changes how packet processing is initiated and which CPU performs it.
SO_BUSY_POLL sets the polling window for a socket. net.core.busy_read supplies the system default for socket reads, while net.core.busy_poll controls the global polling window for poll, select and eligible epoll waits. A busy-poll budget limits packets processed in one polling episode.
Topology Matters
We need to decide which RX queue receives flow, which CPU drains that queue, which CPU owns the socket and parser, and where the parsed event goes next.
For a small number of latency-sensitive WebSocket connections, each hot flow should land on a known RX queue and each queue should have one clear userspace owner. Two layouts are worth considering.
locality-first: interrupt-driven receive
For an epoll-driven client, one default is to place the RX queue’s MSI-X interrupt, its NAPI processing and the network-owner thread on the same logical CPU. The physical core should be reserved for the path: unrelated work should not run on that CPU or its SMT sibling.
RSS -> RX queue Q -> MSI-X / NAPI / TCP on CPU A
-> epoll / recvmsg / TLS / parser on CPU AThis layout follows the semantics of readiness-driven receive. While the owner sleeps in epoll_wait(), the interrupt is useful work: it drains the queue, advances TCP and makes the socket readable. The same CPU then wakes into recvmsg(), TLS and WebSocket parsing. We pay for an interrupt, but not for an additional inter-processor hand-off between the kernel receive path and the application.
The cost is interference. A new RX interrupt can pre-empt the parser while it is handling an earlier message, and sustained NAPI work competes with userspace for the same execution resources. Colocation is therefore the latency-first default only while that core has headroom.
If the combined kernel and userspace work approaches saturation, a deliberate split can buy pipeline capacity:
RSS -> RX queue Q -> MSI-X / NAPI / TCP on CPU B
-> wake / recvmsg / TLS / parser on CPU ACPU A and CPU B should be different physical cores, preferably in the same last-level cache and NUMA node. This allows the kernel and parser to run concurrently, but it makes the socket hand-off and its cache traffic part of every packet’s path. Use the split when it improves the tail at the intended packet rate.
The controls need to express the same plan. RSS or hardware flow steering selects the queue; IRQ affinity selects where its interrupt and ordinary NAPI work run. irqbalance should be stopped.
We can verify this by generating traffic and verifing that the expected RX queue advances, that its MSI-X vector advances on the intended CPU in /proc/interrupts, and that network softirq activity appears where expected in /proc/softirqs.
isolation-first: NAPI busy polling
If the connection is continuously active and we are prepared to dedicate a core, NAPI busy polling changes the ownership boundary. The network owner on CPU A can run the queue’s NAPI poll from its receive or epoll path and consume the resulting socket data on the same CPU.
+-> busy-poll hit: NAPI / TCP / recvmsg / parser on CPU A
RSS -> RX queue Q -------+
+-> busy-poll miss: fallback MSI-X / NAPI on CPU A or CPU BBusy polling does not permanently eliminate interrupts, so fallback placement is a real tradeoff. Keeping the IRQ on CPU A preserves locality when polling misses, which is usually better for sparse or bursty traffic, but an interrupt may pre-empt the parser. Moving the IRQ to CPU B protects CPU A from that interruption, but a miss now executes the kernel receive path on B and wakes A across cores.
There is no placement which simultaneously provides same-core fallback locality and freedom from interrupt interference.
Busy polling is associated with a NAPI instance, not exclusively with one TCP connection. A hot polling socket should therefore not share its RX queue with arbitrary noisy traffic. If one epoll instance relies on busy polling for several sockets, those sockets should share a NAPI ID.
place the consumer deliberately
The same decision appears after WebSocket parsing. If the handler’s work is bounded, keep parsing and the immediate trading decision on CPU A. That removes another hand-off from tick-to-trade. If downstream work is large or variable, make one explicit SPSC hand-off to a second physical core, preferably in the same cache and NUMA domain.
Where the machine or guest exposes meaningful NUMA locality, keep the RX queue’s interrupt CPU, network owner and hot receive state on the NIC-local node. Reserve separate housekeeping capacity for unrelated IRQs, timers, kernel workqueues, logging and control-plane work.
Comparing Implementations
The quantcplib client is the userspace implementation described above. Its nonblocking, epoll-driven socket receives ciphertext and the ENA hardware timestamp through recvmsg(). A custom OpenSSL BIO reads directly into OpenSSL’s destination, SSL_read_ex() writes plaintext into fixed application storage, and the incremental WebSocket parser consumes views over that storage. Parse completion is the synchronous message-callback entry, after UTF-8 validation and before any handler copy or JSON deserialisation.
The CPython 3.9.25 implementation uses a raw socket in timeout mode and calls recvmsg() to retain the same hardware timestamp. It writes the ciphertext into an ssl.MemoryBIO, drains TLS plaintext into a receive buffer, waits for a complete frame payload, assembles continuation frames, validates and decodes UTF-8, and materialises the completed Python str or bytes. Its measurement ends before JSON deserialisation and does not include an asyncio event loop.
I ran the two implementations on the same c8a.xlarge under the vanilla host configuration. Kernel tuning is deliberately excluded here and measured separately in the next section.
Each implementation contains five runs of 3,000 measured messages after 1,000 warm-up messages, for 30,000 measured messages in total. ENA’s PHC disciplined the system clock; every run began and ended within 500ns of the PHC, with a largest observed correction or offset of 346ns. All 30,000 measurements had valid clock ordering. The table reports the median of the five per-run percentiles.
Both implementations inherit the vanilla host’s large receive-path tail. quantcplib reaches 11.326us at p50 and 260.045us at p99; CPython reaches 27.479us and 281.584us. At p99, the kernel-side tail has largely swamped the userspace difference.
The marginal p50 segments separate the implementation cost from that tail. ENA RX to post-recvmsg() takes 10.541us in quantcplib and 11.031us in CPython. From recvmsg() to TLS plaintext, the figures are 0.510us and 2.610us; from TLS plaintext to parsed message, they are 0.330us and 11.300us. The socket boundary is nearly identical. quantcplib then preserves it through its custom BIO, fixed storage and incremental view-based parser, while CPython spends another 13.91us in MemoryBIO, buffered frame completion, UTF-8 decoding and Python object materialisation.
With Kernel Tuning
We can now return to the earlier kernel experiment itself. How do we go about tuning the kernel based on the discussions above? The kernel matrix used the same Tokyo c8a.xlarge with four vCPUs presented as four separate AMD EPYC 9R45 cores, with SMT disabled, one NUMA node, Amazon Linux kernel 6.18.44-99.149, and ENA driver 2.17.2g. Five 3,000-message runs were retained for each configuration, with 1,000 warm-up messages per run: 165,000 measured observations. All eleven stages were run in five randomised complete blocks.