2006 - The Year of User Generated Content

It is hardly surprising to call 2006 the Year of User Generated Content, what with rapid adoption (and rich valuations) of YouTube, MySpace and the like - consumer-generated content. Couple that with business user generated content - where the content is commercial , even market oriented, and its impact on the year is even stronger. I would even go so far to say that electronic markets are based on user-generated content - where the content are commercial offers - ranging from lists of offers (e.g. Craigs List) to bids and offers, from ebay to Wall Street.

The technology impact of this is important - namely:

• I/O: Frequent updates as opposed to frequent reads add real architectural challenges. Typically scale out and caching solves the frequent read problem pretty well. Frequent writes requires a whole another approach - with risks of dirty data everywhere (particularly if performance is an issue - and where is it not?)
  
• Code segregation: For social networking applications likeMySpace, this enables users and contributors to self-style their pages. In the 1st generation of Web 2.0 this means segregating UI from styling (via CSS for example) but for 2nd generation Web 2.0 (should this be Web 3.0?), this concept extends further - to business logic, to data base queries, etc. 2006 was for 1st generation - expect 2nd generation to be significantly more complex - scaling to enable user styling was hard enough.
  
• Data synchronization: Related to the 1st issue. With high write rates, keeping data fresh is pitted against keeping it fast.
  

From a business perspective, what matters is monetization - and how does this content generate revenue? This need not be evil nor underhanded - infrastructure enabling such content generation and distribution is not free. For electronic markets, the content makes the market - the market generates revenue for the operator. For non transactional systems, revenue comes from leveraging the network effect. This network effect benefits everyone - users, aggregators and those that want to sell to the aggregated users alike.

And 2007? Is Social Commerce Coming?

I see very many of the same things happening in 2007 in the area of Social Commerce - where Web 2.0 meets E Commerce. While the details and challenges differ, my guess is there will be distinct technology and business model challenges.

But first, a Social Commerce overview:

While originally popularized by Yahoo Shoposphere and currently being pitched by Gartner and others, it seems to be a combination of social technologies, such as blogs and wikis and networking technologies and commerce, commercial transactions. Some categories are pretty well established:

• Affiliates and referrals (popularized by Amazon)
• Network marketing

while other are emerging, such as:

• Group Commerce, whether within groups (physical or virtual) or between them
• Social mashups involving commercial transactions

But however it surfaces commercially with whatever monetization strategy, in terms of technology, look for heavy borrowing from Web 2.0 (wiki's, blogging, Ajax) plus:

• Distributed transactions - like high frequency/volume of updates, this poses distinct scalability challenges - both in terms of managing state and transaction locks but also propogating transaction state information across a wide number of participants and systems. Under such conditions, a potentially huge number of events and messages could be generated by system and participants alike.
  
• Complex User Interaction models: Along with the potential need to coordinate transactions across independent participants, comes the need to coordinate the participant collaborations as well.
  
• Greater code segregation models: As discussed above, advanced Web 2.0 (Web 2.0 Gen 2) needs modularization beyond UI layout, it needs segregation of business logic from UI and business logic from data access. This is true for social commerce as well. In fact, social commerce is part of Web 2.0 Gen 2.
  
• More nimble UI's: UI's need to be aligned with the user category and be able to be adjusted as user categories change and new ones come on line. Consumer interfaces are obviously different than those for enterprise and SaaS (Software as a Service). This is nothing new - what is becoming increasingly important however are for applications and services to service multiple different user segments. Look to code segregation - discussed above - to be a key enabler.
  

Latency Kills

It has been a long while since last post but I would like to post a few times on an area I have hearing a lot about for last few months. It is not a new issue by any means but there has some recent reemphasis on it for a host of reasons. The issue is latency - so view this as a first with possibly more to come.

Remember the Highway Warning - Speed Kills?
Well, in electronic markets it is reversed - Speed Wins - Latency Kills.

From very high frequency markets, as is common in the financial markets, to broader albeit less latency-sensitive ones, such as eBay and shopping cart dependent Amazon, latency can kill business, kill profitability. I'll be continuing to watch this trend over the coming months but let's start by looking at the problem of latency generally and some of the common patterns that emerge.

Latency is a fact of life in computerized systems, particularly distributed systems - in fact is a fact of physics since computers and networks are at their core limited by the speed of light, the speed of electrons pulsing through microprocessors, network cards, optic fibers etc. While the speed of light sounds fast (and it is, more than 186,000 miles/sec as any high school physics book will tell you), its impact starts to add up the more the microprocessor needs to do, the more times memory needs to be accessed, the more times circuit boards are traversed, and, critically, the more physical distance needs to be traversed. And in today's globally networked systems, these distances can be long, sometimes thousands of miles. But even a relatively short hop (from say from a data center in Boston where a stock market order might originate to a data center in New Jersey where the order is executed) can take several ms.

While each component does indeed become optimized - microprocessors go faster and faster (remember Moore's Law), RAM and storage go faster and more optical fiber gets used - the real problem surfaces when all discrete, optimized components and systems are inter-connected as they are in today's Internet and today's electronic markets. Essentially they become co dependent.

Nothing controversial or dramatically insightful here. The main point of this is that there are many disparate sources of latency that multiply as systems and people are connected. And this latency has a business impact.

In high frequency financial markets, latency is measured in millseconds (one thousandth of a second) and even microseconds (one millionth of a second) and delays often mean lost business - business lost because someone else "hit the quote" first. Thousands of orders are at risk - and just one missed order may cost hundreds of dollars or even several thousand dollars.

In eCommerce applications, latency is measured in seconds for user responsiveness and subseconds for the servers servicing these user requests but the problem is compounded due to scale - millions of users. As for its impact, just think what often happens when a shopping cart takes too long to respond ... the shopper cancels the order or just moves on to a competitor's site. How much do these lost orders add up to?

So while the dynamics and requirements of different markets vary widely, because they are all based on interconnected, co-dependent systems and people, latency is an almost everpresent concern and needs to be squeezed out where possible and managed where it is not squeezed out. Common patterns, techniques and technologies have emerged to help solve this problem from a range of angles. Some emerging top contenders include:

• High performance stream processing to reduce the time it takes to process high speed streams of data (such as financial or geospatial data)
  
• Non invasive network level latency monitoring to determine how much latency each component in a distributed system contributes in an end-to-end transaction and ideally to route around the bottleneck
  
• Rich Internet Applications (RIA) using Ajax, Flash and other emerging technologies to reduce the number of round trips a browser needs to make with a web server
  
• Real time Content Delivery Networks (CDN) to move real time content as close as possible to where it is needed and to route requests for data to the closest possible source.
  

It'll be interesting to see how these trends and technologies develop, particularly as user-generated content continues on its exponential path and even trends to the "real time".