Inside the complex world of event processing

By Pete Harris, President Americas and Editor-at-Large, A-Team Group

Complex event/event stream processing is one hot area right now, and it’s predicted to get hotter still. So to which applications is this technology best suited, and what are your options when it comes to CEP/ESP implementation?

Truth be told, software engineers in the financial markets have been writing code to handle events since the year dot. A price update from an exchange, a trading limit that gets broken, a stock split. All are examples of events that might cause some software process to take place.

Much of that event processing code is pretty generic. When the price of a NYSE stock hits a certain limit and so causes an event to be triggered, the process would be pretty much the same if the stock were a Nasdaq one, or one traded on an ECN. Thus, the emergence of generic event processing functionality and the birth of a software segment that’s expected to mushroom over the next few years.

That segment – known as event stream processing (ESP) to some and complex event processing (CEP) to others – was worth $99 million in 2007, according to Aite Group, which predicts it will grow to $460 million in 2010. Given the numerous applications to which this technology can be applied, across all asset classes, strong growth among ESP/CEP vendors seems highly likely.

More on those applications and vendors later. But first, some quick and approximate definitions:

Event stream processing refers to the tasks associated with handling continuous streams of data items, with a goal of identifying meaningful events within those streams. A market datafeed of trade and quote prices from an exchange or a feed consolidator is a common example of such a stream.

Complex event processing takes this concept further, by identifying several discrete events that, together, form a meaningful event. An example of a complex event might be: when the last trade price of Google increases three times successively by more than one per cent, within a 50 millisecond period, publish this as an event.

Technically, a CEP package will consist of a logic engine, capable of executing continuous queries (often expressed in an SQL-type language) and usually also supporting coded functions in C, C++ or Java, as well as a fast-access memory-resident database containing recent data items, linked to a persisted database (to handle large datasets and back-up) and often a third-party database for time-series data. APIs for linking with market datafeed handlers and messaging middleware will also be offered. The entire package will generally be tuned to offer very low latencies and to scale across multi-core or multi-CPU environments, and to offer fault tolerance and hot standby functionality.

In recent years, a number of vendors offering CEP packages have emerged, creating a fairly crowded marketplace – one that will likely see future consolidation despite the growth predicted. Thus far, those vendors are in varying stages of establishment, searching for market niches, exploring different business models and forming partnerships with complementary technology players and market giants.

Among those CEP vendors that might be regarded as firmly established are Aleri, Cicada, Progress Software’s Apama unit, Vhayu Technologies and Xenomorph. And more recently, they’ve been joined by the likes of Coral8, Skyler Technology, StreamBase Systems and – most recently – Truviso.

Add into that mix the embryonic CEP developments from IBM with System S and BEA Systems (soon to be merged into Oracle) with its Weblogic Event Server. And also add in high performance database vendors, including Kx Systems, Oracle (having acquired TimesTen in 2005), Sybase with its Risk Analytics Platform and Vertica (like StreamBase, founded by Dr Michael Stonebraker), which often provide persistence and historical times-series functionality to complement CEP engines. That adds up to a lot of companies to research when looking to buy a CEP solution.

And that assumes that a financial markets player is going to buy a solution. The Barracuda trade routing system developed by HSBC is one example of building CEP inhouse. When it was built, back in 2005, HSBC technologists didn’t think commercial offerings would cut it. Developed in about six months, Barracuda was rolled out in fixed income and emerging markets operations first, before being deployed more widely.

Also among the options for CEP is to go the open source route, with the Esper project, which is available for both Java and Microsoft .Net and enjoys commercial support from EsperTech.

Vendors have adopted different approaches in building their businesses around CEP. Focusing on the financial vertical, Vhayu has built a strong professional services business, designing and building customised solutions for customers, and has expanded its Velocity product, adding functionality for options data, and for handling of order books in a RegNMS environment. Perhaps more importantly, the company inked a reseller agreement with Reuters, which sells Velocity as the Reuters Tick Capture Engine. Between direct and reseller sales, Vhayu added 29 customers in 2007.

Many vendors – both established and upstarts – have built specific application offerings on top of their core CEP engines. Aleri has rolled out order management and liquidity management offerings.

Apama has built products for algorithmic trading and smart order routing in both the equities and foreign exchange markets. Cicada is focusing on a more fragmented post-MiFID trading world in Europe with its MarketPrizm product. Sklyler has introduced solutions for tick analysis, liquidity discovery and order management, while newest-on-the block Truviso has turned its attention to FX algorithmic trading in partnership with FX ECN Hotspot FX.

Indeed, partnerships have proved important for the CEP players. As well as reselling Vhayu’s software, Reuters also has partnerships with StreamBase (to integrate StreamBase’s Studio application designer with its Reuters Market Data System) and Skyler, to deliver its order book product to Reuters Datafeed Direct customers.

Meanwhile, Coral8 has worked closely with low latency datafeed handler and middleware vendors, including RTI and Wombat Financial Software (recently acquired by NYSE Euronext). And UK-based Xenomorph has partnered with vendors such as Microsoft and JWGIT Group for MiFID solutions, 4th Story for algorithmic trading and NumeriX for derivatives risk management.

Much of the focus of CEP deployment has been on meeting the increase in algorithmic trading, a trend that is still continuing as it spreads to new asset classes and new geographies. CEP engines allow algorithms to be designed, back-tested and deployed rapidly. As the various execution services groups within Wall Street’s sell side do continuous battle with one another over who has the best algorithm suite – today – leveraging CEP is a must.

But CEP is now being routinely adopted for other financial applications, across the entire transaction. In particular, CEP is lending itself to: market data cleansing and enrichment (filtering out and repairing price messages, and supplementing received data with standing data); auto-quoting and market making (responding to market prices and adjusting quotes as needed); order book aggregation and liquidity management (creating a virtual order book from several liquidity venues and determining how liquid an instrument is); and smart order routing (ensuring best execution in a RegNMS and MiFID market structure).

Moreover, CEP is being used in middle office functions, including risk analysis on an intraday basis, and for compliance and trade monitoring functions. Indeed, Apama was recently selected by the investment bank-owned Turquoise trading facility in Europe to implement a real-time and post-trade market surveillance system.

CEP’s promise is also being identified more widely as a technology that might form part of a generic IT infrastructure, and not simply as a capital markets point solution. This infrastructure play was very much the driving force for BEA to build a CEP engine to complement its WebLogic application server – already widely deployed in financial markets firms.

And as a player in the service-oriented architecture (SOA) space, the applicability of CEP to the run-time event management within SOA applications was likely also in BEA’s mind as it looked at the market for CEP. Within Oracle, with its own strong SOA offering, and approaches to scalability, leveraging BEA’s CEP development could make much sense.

Meanwhile, IBM is in the active research stage of its System S Project, which is building an application framework for streaming applications that might be run on a heterogeneous hardware and systems software base – everything from supercomputers and clusters, to hardware-accelerated commodity hardware. Financial markets applications, including low latency market data handling, are among the pilots forming a part of the research.

In short, CEP is in the right place at the right time, and it’s likely to be so for several years to come.