Real Time Cloud?

Can something like Twitter, the popular microblogging service, be truly real time - where all Tweets are distributed to web, cell phone and other followers in real time or nearer real time - and also handle spikes (like at a big event) as well as higher order business and filtering logic? Twitter is known to have stability problems that they refreshingly admit to. It is certainly a messaging system so a real time message-driven or event-driven architecture as part of the answer as opposed to its content management-oriented approach seems to make sense.

But what does this mean? And will it really help it scale while still providing the user experience it needs. Not sure what the " complexity and unpredictability" they talk about is, maybe there aren't off the shelf technologies but maybe the answer is in the cloud - cloud computing - particularly for handling spikes, flashes and even denials of service.

Notes from New England Cloud Computing Group
monthly meeting 12/2/2008

In addition to usual meet and greet, had the pleasure of presentations by Bret Hartman, from RSA who spoke about security as it relates to the cloud plus 2 from an innovative startup using S3 and EC2 - Vikram Kumar and Prasad Thammaneni from Pixily.

Probably biggest points of discussion were the economics of EC2 (including whether its use eliminated the need for costly planning steps, particularly capacity planning) plus the greater obstacles/hurdles enterprises (particularly security) need to overcome to adopt cloud computing vs startups/SMB's (Internal/private clouds could be a happy medium and even interim.)

I believe the hosts will be putting a video recording here.

Other topics/discussion points:

• Cloud computing is fundamentally different from enterprise computing, particularly from a security perspective, and requires new technology/approaches - this is Brett Hartman's contention. I agree. Large enterprises in particular have security policies that will not allow deployment to multi-tenant environments like Amazon. This could also be the consensus bottom line: Until security is woven into cloud computing fabric more completely, cloud computing remains an unrealistic - albeit compelling - option for enterprises where intrusion and data loss are real fears - and real risks.
  
• Start ups (like Pixily) and SMB's have different needs than enterprises and are (and will be in the forseeable future) much quicker to adopt cloud computing with its limitations - and budgetary attractions. Question remains as to whether a startup can scale sufficiently using cloud providers - this is where security rubber meets the road.
  
• Capacity planning is not eliminated in the cloud - but it is reduced and does change
• There is a need for use cases. Best use cases for Cloud tend to be consumer services. Whether a large bank, like Bank of America, can be supported by an multitenant cloud provider remains an open question.
  
• EMC's vision/roadmap for cloud as presented by Brett. Includes:
• Security that is dynamic (content-driven), policy-driven, transparent, risk-driven and information-centric (as opposed to network-centric)
• Products for partners to build cloud infrastructure, such as Atmos.
  
• Cloud services, including Decho (presumably Mozy)
  
• Pixily's positioning - as a consumer service for archiving paper-based documents