Real Time Web Architecture

Kind of an eye chart, but here is an architecture for real time web apps, a possible reference architecture.

Note the inclusion of CEP (like Apama, Streambase or Esker) and Data streaming (like Cometd) along with traditional web and mashup servers. I think that is where the future resides - being able to handle both client pulls (dynamic and static) as well as real time data feeds. Real time data feeds traditionally have been feeds (like securities prices) that are limited to certain markets but increasingly they are broader, mass market feeds - such as Feedburner, Facebook, Twitter and phone/sms - integrating both system-generated and human-generated messages.

Programming languages will be all over the map (from compiled C#, Java, etc) to open web scripting (Python, Javascript etc.) to messaging and proprietary optimized event processing languages like StreamSQL and Apama's monitor script.

Depending on feedback, I hope to describe these components in more detail in future posts and also keep this updated to reflect technology and market developments.

Using Apama Trading Strategies Platform

I have recently had the occasion to pound on the algorithmics trading platform from Apama - both for use at a major Wall Street sell side shop and as a platform for an enterprise risk management system my company is exploring.

The Apama platform offers several components, including:

• Event Modeller - an IDE for developing applications. (See screenshot below)
• Event Manager - the event processing engine. They call it a correlator since it "correlates" events.
• Smart Blocks builder - for building reusable components
• Dashboard generator - to generate the UI's
  
• Some out of the box trading analytics(VWAP, EWMA, etc) , utilities (e.g. timer) and calculators (P&L, position, etc)
• Some sample trading strategies, such as Statistical arbitrage (StatArb)

Event Manager

The Event Manager is the guts of the system. It processes events guided by a proprietary event definition and scripting language (monitorscript). Events are defined using a simple text-based scheme. For example, here is the definition for a stock quote provided in one of the samples:

event StockTick {
string stockCode;
wildcard integer bidQuant;
wildcard float bidPrice;
wildcard integer askQuant;
wildcard float askPrice;
wildcard integer lastQuant;
wildcard float lastPrice;
}


One of the nice nice things about defining events or objects in this way is the ease in which they are defined. There is also no run time casting like a compiled language - which has both pros and cons. I tend like simplicity - must be simple minded.

Monitor script is more complex. It defines relationships between events. It is designed for speed, to enable processing of large volumes of events looking for potentially complex time-based relationships at high speed. All in flight events are kept in memory to speed access and processing. There are several key constructs but I'll call out 2 in particular:

• event handlers. This is fundamental to Apama and where it is optimized. The language sets filters and actions on events. The filters can be very complex.
• Routers and Emitters - are used to generate new derivative events
  

There is a Java interface to monitorscript that most Java developers use but run time processing uses a proprietary binary data format.

Event Modeller

The Event Modeller is the IDE. While they claim it is for business users, I believe it is to be used by application support working with the trading strategist. While not a full featured IDE one has come to expect from Visual Studio or Eclipse, it is indispensable, particularly in its autogeneration of proprietary MonitorScript. And there is no other IDE thjat generates Apama's Monitorscript.

Here is a screen shot taken from a StatArb example.

Like to See...

Beyond performance metrics which I have not had time to analyze yet, biggest thing I'd like to see are ability to process XML data using XML expressions (such as XPath) and some day to process unstructured text in real time.

Overall does its job of processing complex rules at high speed - which is what high frequency quant traders need.

CEP >= Stream Processing?
A lot has been written recently about complex event processing (CEP) and the question I have is whether or at least how it will be used, in the real world, beyond stream processing. Stream processing - which is not really new to real time developers (of applications like trading systems and security monitors) - is essentially geared to processing data (called events) in real time - in other words, before it is put in database or file system and its context is lost and, critically, time passes (read latency). What is new is that a distinct product category is emerging that offers stream processing in product form with admittedly some new spins (such as injectable scripts) and the general *-abilities that productized software offers.

Advocates, such as David Luckham in particular, suggests that CEP represents a whole new way of analyzing data to infer context where it may have been lost and, by analyzing event sequences across time, can be recovered. Seems like the top usage scenario proferred by most vendors putting themselves in this CEP space (e.g. Apama, Streambase and Aleri), however, is in low latency environments, particularly algorithmic trading. In this scenario, software processes make trading decisions automatically by interpreting price and other feeds within the context of trading strategies designed by the business (typically experienced traders). And latency is the enemy since each automated processes is competing with other automated processes - as well as human traders - for the same quotes. It is essentially a race to the order book.

While these products offer a host of other features (Stream SQL from Streambase is particularly interesting), they are basically each addressing this latency issue by offering - albeit in different ways - better and faster ways of processing streaming data. A worthy endeavor, to be sure, but it is stream processing.

It will curious to see what other types of applications develop and whether they fit the CEP moniker. I haven't seen it yet. The problems of discovering context for an event or other data (whether histroical or real time) do exist but can this crop of technology adequately address it?