Chorus, Atoms and Chairman

Is this the new nomenclature for cloud computing, specifically databases for the cloud? No more records, transaction managers and the like. In one of the more interesting sessions I have seen, yesterday's presentation at MIT by Jim Starkey and Nimbus DB described his vision of a cloud database.

In NimbusDB speak, a chorus is (I think) a set of Peer-to-Peer database nodes all exchanging metadata messages about their data. The db is based on adding the network layer, along with memory, disk etc, to the pyramid of data access resources linking a set of nodes on the network.

Atoms are 50k chunks of data (Why not 64K?) for both metadata and application data.

The Chairman is responsible for managing read and write permissions to the atoms.

I definitely buy the theory and, having recently finished an book on Einstein, appreciate the references to Relativity. Not all database transactions (e.g. some reads) require immediate consistence. Eventual consistency (as long as eventual is within application specific tolerance windows) but not immediate. But I also have questions - specifically about time stamps, impact of version unavailability, whether chairman have live synchronized backups and performance impact of message chatter.

Cloud DB's

SQL-NoSQL, MVCC-Distributed MVCC, Relational-Non relational .....

The challenges are myriad for managing data in the cloud, particularly since traditional databases (Oracle, SQLServer, MySQL etc.) are difficult and expensive to scale and virtualize, requiring full backups for "instant" virtualization.

On the other hand, the choices are expanding. NoSQL, such as Google's BigTable and Amazon's SimpleDB, is gaining real adherents as it piles up successes (just don't ask for joins.) Joins in a massively distributed environment will probably require a distributed relational database based on MVCC or some other technique to ensure queries against a consistent snapshot. While not yet proven (AFAIK), this holds real promise. Many of the applications I am working on currently, including Innerpass' collaboration application, do require relational queries but they don't require subsecond response. And they could also benefit from the ability to handle spikes without provisioning db servers. Eventual consistency is actually a good match with asynchronous Ajax-based page refreshes where temporarily inaccurate or incomplete link lists can be tolerated.

My guess it will come down, like most (if not all) technologies and architectures before it, to application requirements and business imperative. First off, many apps won't need to change and won't change - it it ain't broke, don't fix it. Greenfield applications and, in some instances, applications that need to modernize - as they are exposed to larger user bases, for example, will need to consider cloud architectures. Technology adoption and adoption rates will be driven by the nature of the application, cost and time-to-market concerns. High volume, low data value sites will tend, from what I can see, toward non relational deployments, data warehouses to columnar and MapReduce hybrids and transactional applications will remain dependent on relational databases. The biggest questions are the in between ones. And the type of relational database will be very performance and latency dependent. Those with high latency tolerance may find distributed databases acceptable and even preferrable while those demanding higher throughput and consistency will stick with tried and true.

Boiling It Down

Buckets of ink and electrons have been spilled answering "What is Cloud Computing" - ranging from the overly stingent (requiring a specific hardware and/or software formalism) to overly broad (calling everything on the Internet a cloud application). Here is what it boils down to for me:

• Elasticity: The resources - hardware, software, network - that the application needs to run needs are elastic. As the application usage goes up, it should be able to access more of what it needs, from CPU cycles and horsepower to database "rows" and disk space. And as it goes down, these resources should be released for other applications to use. In other words, the elasticity of a single computer (where CPU, memory and other resources are shared across all the applications on that box) is spread across the data center, across clients and servers.
• Pay for what you use: Implicit - and a (perhaps the) primary benefit of elasticity is economic - aligning costs with use. As use (and resources) go up and down, so should the costs.
• Easy to Provision/Order/Pay For - This is about on ramping and adding resources. In a completely auto-elastic system, provisioning is implicit. But even if provision is not automatic, it still needs to be easy.

So .... cloud computing is both business model (yes whether public or private) and technology and these attributes generally cut across both. Elastic resources really includes the technology (enable it) and the ability to track what is used to support the business model, whatever its specifics.

On Developer Cloud Services

Developer cloud services are an increasingly viable options for architects, developers and managers. By developer services, putting my "developer hat" on, I mean services that developers access to provide necessary application functions as opposed, for example, to application level services (like creating a customer record) and lower level infrastructure services (like storage). In fact, developer level services sit between the other two as the (admittedly) simplistic diagram shows below.





Essentially, developer services are those functions (typically with an API) that developers need to build out applications and are typically managed by the platform (just as J2EE or .Net do). Some key services would include data/database, caching, logging, authentication, validation, messaging and text/XML processing services. These services could even include specific types of common data, like zipcodes and stock quotes. Storage services may belong here as well depending upon what type of interface and control there is (e.g. developer level vs storage administrator). Multitenancy, a critical consideration for public clouds, could generally be managed here either as a built in function going across other services or as a distinct service accessed by the developer as needed. While some are starting to emerge in platforms like Google AppEngine (BigTable, logging, etc.), it would be great to see these built out further. And they'd make the higher level, higher value application services (that we all need) easier to build - since less time and effort would be spent (wasted) on plumbing, necessary but reusable plumbing. Plus they'd hopefully be better implemented with more, better implemented. features, such as monitoring, reporting and analytics.

This in fact aligns pretty well with the IaaS, PaaS and SaaS breakdown folks are using - as mapped below.




One man's call ...

On Premises "Cloud"?

Is this an oxymoron? Or has its time come?

At a recent Silver Lining Cloud Computing meeting, we were treated to a couple interesting presentations, one by SnapLogic, the other by Tamale Software.

While SnapLogic offers technology to link private and public clouds/public SaaS offerings like SalesForce.com, Tamale offers an on premises appliance that they manage from their data center. Specifically tooled for their customers (securities analysts and portfolio managers), while it raises some cloud-oriented questions particularly around about scalability and handling spikes, it certainly helps alleviates concerns about security. Security is maybe the big issue for cloud computing, ensuring data privacy. I believe the economic drivers will still push many organizations to public cloud providers, but this "on premises" deployment model could be a good transitional state for some organizations and maybe even an end state for others.

In addition to cost-effective scalability, updateability is also an issue for on premise and on premise/cloud hybrids. To this end, Tamale has developed some interesting deployment technology called Delta. Not sure if this is code name or product name, but it essentially simplifies and automates the deployment of new components from their centralized data center to the customer's premises.

Definitely worth watching and looking into with more detail - addresses a big obstacle to the path to the cloud.

Real Time Cloud?

Can something like Twitter, the popular microblogging service, be truly real time - where all Tweets are distributed to web, cell phone and other followers in real time or nearer real time - and also handle spikes (like at a big event) as well as higher order business and filtering logic? Twitter is known to have stability problems that they refreshingly admit to. It is certainly a messaging system so a real time message-driven or event-driven architecture as part of the answer as opposed to its content management-oriented approach seems to make sense.

But what does this mean? And will it really help it scale while still providing the user experience it needs. Not sure what the " complexity and unpredictability" they talk about is, maybe there aren't off the shelf technologies but maybe the answer is in the cloud - cloud computing - particularly for handling spikes, flashes and even denials of service.

Notes from New England Cloud Computing Group
monthly meeting 12/2/2008

In addition to usual meet and greet, had the pleasure of presentations by Bret Hartman, from RSA who spoke about security as it relates to the cloud plus 2 from an innovative startup using S3 and EC2 - Vikram Kumar and Prasad Thammaneni from Pixily.

Probably biggest points of discussion were the economics of EC2 (including whether its use eliminated the need for costly planning steps, particularly capacity planning) plus the greater obstacles/hurdles enterprises (particularly security) need to overcome to adopt cloud computing vs startups/SMB's (Internal/private clouds could be a happy medium and even interim.)

I believe the hosts will be putting a video recording here.

Other topics/discussion points:

• Cloud computing is fundamentally different from enterprise computing, particularly from a security perspective, and requires new technology/approaches - this is Brett Hartman's contention. I agree. Large enterprises in particular have security policies that will not allow deployment to multi-tenant environments like Amazon. This could also be the consensus bottom line: Until security is woven into cloud computing fabric more completely, cloud computing remains an unrealistic - albeit compelling - option for enterprises where intrusion and data loss are real fears - and real risks.
  
• Start ups (like Pixily) and SMB's have different needs than enterprises and are (and will be in the forseeable future) much quicker to adopt cloud computing with its limitations - and budgetary attractions. Question remains as to whether a startup can scale sufficiently using cloud providers - this is where security rubber meets the road.
  
• Capacity planning is not eliminated in the cloud - but it is reduced and does change
• There is a need for use cases. Best use cases for Cloud tend to be consumer services. Whether a large bank, like Bank of America, can be supported by an multitenant cloud provider remains an open question.
  
• EMC's vision/roadmap for cloud as presented by Brett. Includes:
• Security that is dynamic (content-driven), policy-driven, transparent, risk-driven and information-centric (as opposed to network-centric)
• Products for partners to build cloud infrastructure, such as Atmos.
  
• Cloud services, including Decho (presumably Mozy)
  
• Pixily's positioning - as a consumer service for archiving paper-based documents