Latency Kills

It has been a long while since last post but I would like to post a few times on an area I have hearing a lot about for last few months. It is not a new issue by any means but there has some recent reemphasis on it for a host of reasons. The issue is latency - so view this as a first with possibly more to come.

Remember the Highway Warning - Speed Kills?
Well, in electronic markets it is reversed - Speed Wins - Latency Kills.

From very high frequency markets, as is common in the financial markets, to broader albeit less latency-sensitive ones, such as eBay and shopping cart dependent Amazon, latency can kill business, kill profitability. I'll be continuing to watch this trend over the coming months but let's start by looking at the problem of latency generally and some of the common patterns that emerge.

Latency is a fact of life in computerized systems, particularly distributed systems - in fact is a fact of physics since computers and networks are at their core limited by the speed of light, the speed of electrons pulsing through microprocessors, network cards, optic fibers etc. While the speed of light sounds fast (and it is, more than 186,000 miles/sec as any high school physics book will tell you), its impact starts to add up the more the microprocessor needs to do, the more times memory needs to be accessed, the more times circuit boards are traversed, and, critically, the more physical distance needs to be traversed. And in today's globally networked systems, these distances can be long, sometimes thousands of miles. But even a relatively short hop (from say from a data center in Boston where a stock market order might originate to a data center in New Jersey where the order is executed) can take several ms.

While each component does indeed become optimized - microprocessors go faster and faster (remember Moore's Law), RAM and storage go faster and more optical fiber gets used - the real problem surfaces when all discrete, optimized components and systems are inter-connected as they are in today's Internet and today's electronic markets. Essentially they become co dependent.

Nothing controversial or dramatically insightful here. The main point of this is that there are many disparate sources of latency that multiply as systems and people are connected. And this latency has a business impact.

In high frequency financial markets, latency is measured in millseconds (one thousandth of a second) and even microseconds (one millionth of a second) and delays often mean lost business - business lost because someone else "hit the quote" first. Thousands of orders are at risk - and just one missed order may cost hundreds of dollars or even several thousand dollars.

In eCommerce applications, latency is measured in seconds for user responsiveness and subseconds for the servers servicing these user requests but the problem is compounded due to scale - millions of users. As for its impact, just think what often happens when a shopping cart takes too long to respond ... the shopper cancels the order or just moves on to a competitor's site. How much do these lost orders add up to?

So while the dynamics and requirements of different markets vary widely, because they are all based on interconnected, co-dependent systems and people, latency is an almost everpresent concern and needs to be squeezed out where possible and managed where it is not squeezed out. Common patterns, techniques and technologies have emerged to help solve this problem from a range of angles. Some emerging top contenders include:

• High performance stream processing to reduce the time it takes to process high speed streams of data (such as financial or geospatial data)
  
• Non invasive network level latency monitoring to determine how much latency each component in a distributed system contributes in an end-to-end transaction and ideally to route around the bottleneck
  
• Rich Internet Applications (RIA) using Ajax, Flash and other emerging technologies to reduce the number of round trips a browser needs to make with a web server
  
• Real time Content Delivery Networks (CDN) to move real time content as close as possible to where it is needed and to route requests for data to the closest possible source.
  

It'll be interesting to see how these trends and technologies develop, particularly as user-generated content continues on its exponential path and even trends to the "real time".