On My Watch

2011-06-01T19:00:00.012-04:00

4.5

4.5 out of 10.

That's how firms rate their use of analytics, according to Michael Hopkins, editor-in-chief of MIT Sloan Management Review as described here. Moreover, Brad Peterson, Schwab CIO says get marketing to pay rather than IT. He used mobile app in his example thinking, I think, about capturing mobile data for analysis but why not use marketing budgets to better drive marketing analytics more broadly. Better yet, align costs to revenue and profits. And enable ROI calculations then maximize ROI.

Here are a few places to start:

Move beyond web and email analytics. While hits, visitor counts, email opens and the like are critical, they are not enough. Marry this data with target company organizational structures, social graphs, CRM data and other internal and external data to get a full picture of the market or slice into territories.
Let people drive the analytics based on their roles and needs. Sales reps can mine the data to look for past deal patterns and possible triggering events in search of next deal. Marketing can analyze previous campaigns and plot the next one. Managers, directors and C-Level can watch for longitudinal trends.
View marketing data through different lenses. For example, sales interested in this quarter, marketing next and biz dev 2-3 quarters away in adjacent markets.

In short, capture whatever data you can from the market and from the field and let your servers and analytics algorithms do the work to crunch the data for you. And use the analytics in a feedback loop: Learn from how the system is used to determine results and let the measurable actions of those responsible for quota or P&L drive ROI calculations.

And turn 4.5 into 9's and 10's.

2011-05-01T14:40:00.006-04:00

Real Time Web and Marketing Automation

I am reposting graphic from earlier post but applying to marketing automation. Although quite generic and almost ancient history (2008), it surprisingly pertains I think. But Real Time Web is People-to-People and so is B2B marketing.

Traditional marketing automation concerns itself primarily with managing campaigns (landing pages, etc) and scoring, routing and nurturing marketing leads or, more generically, contacts. Making real time means:

1:1 marketing - letting customers find and communicate with marketers while actively engaged. And vice-versa.
Meaningful bidirectional notifications - keep customers informed based on their interest and marketers informed of customer actions and, critically, relationships
Viewing marketing and sales process as well as a series of events - behavioral events, conversion events, inquiries, transactions
Searching lead and other internal marketing data for contact, lead, prospect and customer actions - or of those in their business network. On demand and continually. So while customers are googling you or more usually solutions to their problem or deals, marketers can be looking for them.
Correlate events and data across prospect network to uncover hidden triggering events.

2010-03-19T08:05:00.005-04:00

Chorus, Atoms and Chairman

Is this the new nomenclature for cloud computing, specifically databases for the cloud? No more records, transaction managers and the like. In one of the more interesting sessions I have seen, yesterday's presentation at MIT by Jim Starkey and Nimbus DB described his vision of a cloud database.

In NimbusDB speak, a chorus is (I think) a set of Peer-to-Peer database nodes all exchanging metadata messages about their data. The db is based on adding the network layer, along with memory, disk etc, to the pyramid of data access resources linking a set of nodes on the network.

Atoms are 50k chunks of data (Why not 64K?) for both metadata and application data.

The Chairman is responsible for managing read and write permissions to the atoms.

I definitely buy the theory and, having recently finished an book on Einstein, appreciate the references to Relativity. Not all database transactions (e.g. some reads) require immediate consistence. Eventual consistency (as long as eventual is within application specific tolerance windows) but not immediate. But I also have questions - specifically about time stamps, impact of version unavailability, whether chairman have live synchronized backups and performance impact of message chatter.

2009-12-01T16:07:00.002-05:00

Cloud DB's

SQL-NoSQL, MVCC-Distributed MVCC, Relational-Non relational .....

The challenges are myriad for managing data in the cloud, particularly since traditional databases (Oracle, SQLServer, MySQL etc.) are difficult and expensive to scale and virtualize, requiring full backups for "instant" virtualization.

On the other hand, the choices are expanding. NoSQL, such as Google's BigTable and Amazon's SimpleDB, is gaining real adherents as it piles up successes (just don't ask for joins.) Joins in a massively distributed environment will probably require a distributed relational database based on MVCC or some other technique to ensure queries against a consistent snapshot. While not yet proven (AFAIK), this holds real promise. Many of the applications I am working on currently, including Innerpass' collaboration application, do require relational queries but they don't require subsecond response. And they could also benefit from the ability to handle spikes without provisioning db servers. Eventual consistency is actually a good match with asynchronous Ajax-based page refreshes where temporarily inaccurate or incomplete link lists can be tolerated.

My guess it will come down, like most (if not all) technologies and architectures before it, to application requirements and business imperative. First off, many apps won't need to change and won't change - it it ain't broke, don't fix it. Greenfield applications and, in some instances, applications that need to modernize - as they are exposed to larger user bases, for example, will need to consider cloud architectures. Technology adoption and adoption rates will be driven by the nature of the application, cost and time-to-market concerns. High volume, low data value sites will tend, from what I can see, toward non relational deployments, data warehouses to columnar and MapReduce hybrids and transactional applications will remain dependent on relational databases. The biggest questions are the in between ones. And the type of relational database will be very performance and latency dependent. Those with high latency tolerance may find distributed databases acceptable and even preferrable while those demanding higher throughput and consistency will stick with tried and true.

2009-08-03T15:53:00.003-04:00

Boiling It Down

Buckets of ink and electrons have been spilled answering "What is Cloud Computing" - ranging from the overly stingent (requiring a specific hardware and/or software formalism) to overly broad (calling everything on the Internet a cloud application). Here is what it boils down to for me:

Elasticity: The resources - hardware, software, network - that the application needs to run needs are elastic. As the application usage goes up, it should be able to access more of what it needs, from CPU cycles and horsepower to database "rows" and disk space. And as it goes down, these resources should be released for other applications to use. In other words, the elasticity of a single computer (where CPU, memory and other resources are shared across all the applications on that box) is spread across the data center, across clients and servers.
Pay for what you use: Implicit - and a (perhaps the) primary benefit of elasticity is economic - aligning costs with use. As use (and resources) go up and down, so should the costs.
Easy to Provision/Order/Pay For - This is about on ramping and adding resources. In a completely auto-elastic system, provisioning is implicit. But even if provision is not automatic, it still needs to be easy.

So .... cloud computing is both business model (yes whether public or private) and technology and these attributes generally cut across both. Elastic resources really includes the technology (enable it) and the ability to track what is used to support the business model, whatever its specifics.

2009-06-19T16:31:00.006-04:00

My (Personal) Activity Monitor?

Like waiting for my jetpack (still waiting...) I am also waiting for an activity monitor - like Business Activity Monitor (BAM) but for me - a Personal Activity Monitor (PAM). After years of promises and slew of technologies - both corporate (data warehouses, real time XML ..) and consumer (IM, RSS readers, ...), I know everything is possible and many things done. But I am just looking for an easy way to track things potentially complex things about me and my accounts, not just online social networking and communication accounts (Facebook, Twitter, LinkedIn, GMail, Skype ...) but transactional accounts (e.g. bank accounts). It is my information (so I should know it) but I don't always - and more importantly I want to monitor transactions and transaction status - and be alerted if something goes wrong or something is just not right.

I have wanted this for some time but, recently, there was fraud activity on my personal bank account. So now I really want it. I only found out because I happened to be log onto account for another reason - and I saw suspicious activity that I should have been notified of. (Essentially several unauthorized transactions over a 2 day period on the opposite - West - coast for unauthorized withdrawals 4-5 times bigger than either my wife or I usually take out). And if it didn't trigger bank algorithms, I should have been able to set up my own. Now, I did get the money back (but it cost me several hrs, some embarassment of bounced check on a closed account and still chasing the fee reimbursements), but it sure would have been better to catch it before it happened - or at least before it happened second or third time. Save the bank the money too since they had to cover it (and ultimately the country since this is a bank receiving billions in gov't bailout money). And the fact that the bank's systems were not set up just strengthens the impetus to push the filtering algorithms out to the users.

My credit card company allows me to configure various alerts, such as bills due or thresholds met and my brokerage account lets me set up simple trading triggers. But noone (at least none of my providers) let me set up more than simple monitors (like a threshold alert) that would catch a pattern preceding my bank account fraud. The amounts alone should have triggered an alert, or at least a watch. And noone lets me input other data (my travel schedule, for one thing, would have said I wasn't in California on those days) to process against the data they already have.

Providing consumers the ability to set up filters like this would actually save institutions/businesses (like banks) money (in analysis and engineering overhead) AND produce a better result I think - more customized, more able to take advantage of what I know about myself - more situationally aware.

Not sure exactly what it is (control of a Facebook plus SECURE access to real transactional data?) but I want my PAM...

2009-06-17T14:23:00.001-04:00

SSO for Clouds
How close are we?

Now that SFDC supports SAML 2.0 (plus oAuth) standards for Single Sign On, that joins them up with Google AppEngine as 2 of the major cloud providers on a common path. This should bode well for SSO I would think. However, I think Amazon, particularly for S3, needs to get on board, since they are by far the biggest player in IaaS with the greatest number of independent developers, for the tipping point to come. EC2-based apps can obviously be SSO-enabled by the app developer deploying on it, but doesn't t it make sense to have this facilitated for app developers who are not SSO experts) ? And SSO, as I said, for S3 in particular opens up a lot of options for collaborative applications, options for collaboration that is also secure and managed.

2009-06-08T13:24:00.005-04:00

Got to be a better way

Last night finished Meltdown: A Free-Market Look at Why the Stock Market Collapsed, the Economy Tanked, and Government Bailouts Will Make Things Worse by Thomas E Woods.

Is Deflation OK? Woods says it is but my text books say no. Otherwise, some of prescription sounds pretty darn good, particularly, as a techie and modernizer, the first:

Let Them Go Bankrupt - 'What's special about banks?" Can't we find other ways to connect depositors and borrowers without propping up the losers? Banks are intermediaries. Doesn't technology make this process easier and easier - including proper regulation - and lower the costs? I am not a banker and sure they'll be pain but there already is (to the tune of $billions) and can't us (we?) technologists build the connections and controls.
Abolish Fannie And Freddie - What are they even doing now?
Stop the Bailouts - How can more debt solve the problem of too much debt? Or at least "Where's mine?"
End gov manipulation of Money - not sure about this one. Again, is deflation OK and time to go back to the gold standard? Above my pay grade for sure.
Put the Fed on The Table - As far as what I can tell, Fed is probably one of the most rational and least political institution now but it is hardly ever mentioned as part of the problem. Certainly better than the over leveraged banks, regulators (particularly Office of Thrift Supervision), Congress, dead beat borrowers and activists pushing for loans to people who can't pay but didn't easy credit get us into this fix? (Actually I don't really fault the banks too much (some of my best friends are bankers - some even design financial products) - they just operate in the environment the govt created but take a look at some of those complex loan contacts used for ARM's - yikes)
Close Those Special Lending Windows - Again don't really understand the detailed pros and cons but do the Term Auction facilities hurt or help?
End the Monopoly Money - Back to the gold standard? Let me check my gold chain inventory....

I''ll get off my soap box now - but there has got to be a better way than flushing billions/trillions away - a billion here and billion there.... Guess I need to read Von Mises next.

2009-05-11T08:27:00.011-04:00

On Developer Cloud Services

Developer cloud services are an increasingly viable options for architects, developers and managers. By developer services, putting my "developer hat" on, I mean services that developers access to provide necessary application functions as opposed, for example, to application level services (like creating a customer record) and lower level infrastructure services (like storage). In fact, developer level services sit between the other two as the (admittedly) simplistic diagram shows below.

Essentially, developer services are those functions (typically with an API) that developers need to build out applications and are typically managed by the platform (just as J2EE or .Net do). Some key services would include data/database, caching, logging, authentication, validation, messaging and text/XML processing services. These services could even include specific types of common data, like zipcodes and stock quotes. Storage services may belong here as well depending upon what type of interface and control there is (e.g. developer level vs storage administrator). Multitenancy, a critical consideration for public clouds, could generally be managed here either as a built in function going across other services or as a distinct service accessed by the developer as needed. While some are starting to emerge in platforms like Google AppEngine (BigTable, logging, etc.), it would be great to see these built out further. And they'd make the higher level, higher value application services (that we all need) easier to build - since less time and effort would be spent (wasted) on plumbing, necessary but reusable plumbing. Plus they'd hopefully be better implemented with more, better implemented. features, such as monitoring, reporting and analytics.

This in fact aligns pretty well with the IaaS, PaaS and SaaS breakdown folks are using - as mapped below.

One man's call ...

2009-03-08T14:51:00.002-04:00

Scuttlebutt - 2009

Trying to recover (along with everyone else) from market bloodbath - so this is what it feels like to be sliding into a Great Recession - I am rereading book on Warren Buffet - The Warren Buffet Way (by Robert Hagstrom Jr). This book talks about the legend's investment strategy as well as that of the two who influenced him - Benjamin Graham and Philip Fisher. While Graham was the dean of financial analysis, Fisher, the author of Common Stocks and Uncommon Profit was more qualitative, focusing more on manager quality than quantitative metrics.

Anyway, Fisher was a big advocate of what he called business "scuttlebutt" - essentially using the business grapevine - to find and evaluate the quality companies and stocks. This involves talking to suppliers, customers, company employees, and people knowledgeable in the industry, and, eventually, company management. Fisher even teaches how to the correct, company-specific questions. Common sense I guess but Peter Lynch, from Fidelity fame, is a big advocate of this approach - kicking tires, eating donuts at Dunkin Donuts, etc,

Fisher acknowledges the scuttlebutt method is a lot of work. It was in 1958 when it was written - requiring lots of driving and calling around. But this is 2009 - isn't this a perfect use of tools likeTwitter? Twitter is hot, Twitter is cool, Twitter evaluation is sky high - but Twitter believe it or not , is not for everything. But it or perhaps one of its current or future competitors could be perfect for this, for business scuttlebutt. Or at least a good start.

StockTwits may be closest thing so far although don't really offer much in the way of managing this sort of analysis.

2009-02-12T09:52:00.003-05:00

On Premises "Cloud"?

Is this an oxymoron? Or has its time come?

At a recent Silver Lining Cloud Computing meeting, we were treated to a couple interesting presentations, one by SnapLogic, the other by Tamale Software.

While SnapLogic offers technology to link private and public clouds/public SaaS offerings like SalesForce.com, Tamale offers an on premises appliance that they manage from their data center. Specifically tooled for their customers (securities analysts and portfolio managers), while it raises some cloud-oriented questions particularly around about scalability and handling spikes, it certainly helps alleviates concerns about security. Security is maybe the big issue for cloud computing, ensuring data privacy. I believe the economic drivers will still push many organizations to public cloud providers, but this "on premises" deployment model could be a good transitional state for some organizations and maybe even an end state for others.

In addition to cost-effective scalability, updateability is also an issue for on premise and on premise/cloud hybrids. To this end, Tamale has developed some interesting deployment technology called Delta. Not sure if this is code name or product name, but it essentially simplifies and automates the deployment of new components from their centralized data center to the customer's premises.

Definitely worth watching and looking into with more detail - addresses a big obstacle to the path to the cloud.

2008-12-05T21:05:00.014-05:00

Real Time Cloud?

Can something like Twitter, the popular microblogging service, be truly real time - where all Tweets are distributed to web, cell phone and other followers in real time or nearer real time - and also handle spikes (like at a big event) as well as higher order business and filtering logic? Twitter is known to have stability problems that they refreshingly admit to. It is certainly a messaging system so a real time message-driven or event-driven architecture as part of the answer as opposed to its content management-oriented approach seems to make sense.

But what does this mean? And will it really help it scale while still providing the user experience it needs. Not sure what the " complexity and unpredictability" they talk about is, maybe there aren't off the shelf technologies but maybe the answer is in the cloud - cloud computing - particularly for handling spikes, flashes and even denials of service.

2008-12-03T20:16:00.009-05:00

Notes from New England Cloud Computing Group
monthly meeting 12/2/2008

In addition to usual meet and greet, had the pleasure of presentations by Bret Hartman, from RSA who spoke about security as it relates to the cloud plus 2 from an innovative startup using S3 and EC2 - Vikram Kumar and Prasad Thammaneni from Pixily.

Probably biggest points of discussion were the economics of EC2 (including whether its use eliminated the need for costly planning steps, particularly capacity planning) plus the greater obstacles/hurdles enterprises (particularly security) need to overcome to adopt cloud computing vs startups/SMB's (Internal/private clouds could be a happy medium and even interim.)

I believe the hosts will be putting a video recording here.

Other topics/discussion points:

Cloud computing is fundamentally different from enterprise computing, particularly from a security perspective, and requires new technology/approaches - this is Brett Hartman's contention. I agree. Large enterprises in particular have security policies that will not allow deployment to multi-tenant environments like Amazon. This could also be the consensus bottom line: Until security is woven into cloud computing fabric more completely, cloud computing remains an unrealistic - albeit compelling - option for enterprises where intrusion and data loss are real fears - and real risks.
Start ups (like Pixily) and SMB's have different needs than enterprises and are (and will be in the forseeable future) much quicker to adopt cloud computing with its limitations - and budgetary attractions. Question remains as to whether a startup can scale sufficiently using cloud providers - this is where security rubber meets the road.
Capacity planning is not eliminated in the cloud - but it is reduced and does change
There is a need for use cases. Best use cases for Cloud tend to be consumer services. Whether a large bank, like Bank of America, can be supported by an multitenant cloud provider remains an open question.
EMC's vision/roadmap for cloud as presented by Brett. Includes:

Security that is dynamic (content-driven), policy-driven, transparent, risk-driven and information-centric (as opposed to network-centric)
Products for partners to build cloud infrastructure, such as Atmos.
Cloud services, including Decho (presumably Mozy)

Pixily's positioning - as a consumer service for archiving paper-based documents

2008-10-06T16:31:00.010-04:00

Social Media and Engagement

Measuring Social Media

This post from Online Marketing Blog has several useful online marketing links but two, in particular, taken together reinforce a thread I have begun to follow - that is Engagement Metrics and Social Media. Looking at #4 (linking to 237 examples of brands using social media) and #10 (linking to a white paper from WebAnalyticsDemystified ) suggests a possible formulation for measuring the impact of community on engagement and on community engagement itself.

Engagement Metrics

The white paper, Measuring the Immeasurable: Visitor Engagement, postulates a possible formula to measure engagement:

where:

Ci = Click Depth Index capturing the contribution of page and event views

Di = Duration Index capturing the contribution of time spent on site

Ri = Recency Index capturing the visitors “visit velocity”—the rate at which visitors return to the web site over time

Bi= Brand Index capturing the apparent awareness of the visitor of the brand, site, or product(s)

Fi=Feedback Index capturing qualitative information including propensity to solicit additional information or supply direct feedback

Ii= Interaction Index capturing visitor interaction with content or functionality designed to increase level of Attention the visitor is paying to the brand, site, or product(s)
Li=Loyalty Index capturing the level of long-term interaction the visitor has with the brand, site, or product(s)

I am not sure how practical these are in practice but, upon initial read, they seem to address at least some of the shortcomings of traditional engagement measures, such as session duration, page views per session. But even it is the best, most workable model for online media, the question to me is whether it sufficiently covers social media and communities. For example, are blog comments properly accounted for? What about friends lists and social graphs?

Which leads to the second item - the 237 examples of brands. More to the point, 237 initiatives are listed, some more effective than others but, at the end of the day, only metrics can tell us which ones are which and these metrics need to align with each community's and social initiative's objectives.

Social Media

Take one of them - Oracle Community . Oracle, as another example of a software company looking to bridge their disparate developer and partner communities, puts a number of communities - Oracle Technology Network, Oracle Partner Network, Oracle Ace Program, OSpace, Oracle Customers and others - under this umbrella. Then add the Open Source communities not included here, such as Berkeley DB.

Bottom line is the need to manage these disparate yet interrelated Oracle-focused communities. How well do these engagement metrics apply to the properties under this umbrella? Although there is obviously significant community overlap (e.g. between developers/architects and partner channel managers), there are also significant community differences. And the metrics (both what is collected and how they are collected) need to reflect both the similarities and differences.

2008-09-28T08:50:00.008-04:00

Community Metrics and Instrumentation

Why Communities Fail suggests traditional metrics used to measure communities are wrong:

"The third problem with online communities is how businesses go about measuring the success of their communities. Businesses say that their primary objectives are generating word-of-mouth marketing and increasing customer loyalty. Yet the metric that businesses use most often to measure success is the number of visits to the site. Moran points out that there isn’t much of a connection between what businesses want and what they’re measuring. Better metrics might be rankings in Google or the number of inbound links"

So what are the right metrics then? Yes, of course what is measured should be aligned with the goals of the community. But how? How should a community be instrumented?

In my view, the top metrics (selected from a larger list courtesy of GetClicky ) to choose from include:

links
links-outbound
site-rank
visitors
visitors-unique
actions-average
time-average
bounce-rate
visitors-online
feedburner-subscribers

And inbound link statistics, in particular, are a good indicator of real, organic community traction and are applicable to nearly all community types. Plus tools like GetClicky provide them - as shown here

And additional ones to watch include:

Twitter-use - since its use is rapidly increasing and creates a useful snapshot in time
Cross-community links, such as Facebook for consumer-focused communities, LinkedIn for business/executive-focused communities or even Apache mailing lists for developer-oriented ones.

But, I have yet to see a really good model for which to use when and which ones to really bank on.

2008-09-25T10:40:00.010-04:00

Real Time Web Architecture

Kind of an eye chart, but here is an architecture for real time web apps, a possible reference architecture.

Note the inclusion of CEP (like Apama, Streambase or Esker) and Data streaming (like Cometd) along with traditional web and mashup servers. I think that is where the future resides - being able to handle both client pulls (dynamic and static) as well as real time data feeds. Real time data feeds traditionally have been feeds (like securities prices) that are limited to certain markets but increasingly they are broader, mass market feeds - such as Feedburner, Facebook, Twitter and phone/sms - integrating both system-generated and human-generated messages.

Programming languages will be all over the map (from compiled C#, Java, etc) to open web scripting (Python, Javascript etc.) to messaging and proprietary optimized event processing languages like StreamSQL and Apama's monitor script.

Depending on feedback, I hope to describe these components in more detail in future posts and also keep this updated to reflect technology and market developments.

2008-09-19T14:49:00.018-04:00

Web 2.0 Expo

Didn't exhibit or speak but did walk the exhibit floor at Web 2.0 Expo in NYC yesterday.

It was surprisingly vibrant coming in the midst of the financial meltdown - just blocks away.

Source: Wall Street Journal Digital Network

Although there was some good gallows humor in among the crowd.

In my adhoc "survey", nearly all said traffic was either excellent or good. Although obviously anecdotal only, there was quite a bit more activity in Web 2.0's half of the Javitz Center than Interop's which was dead.

This vibrancy is either a sign of an innovative space with a good value proposition or a bunch of ostrich's hiding their head in the sand.

In any case, while there were definitely a number of me-to technologies, what struck me was the focus on business models and user experience as opposed to technology - big change from years past.

Can't mention everything (maybe I'll do a more complete overview if I have time) but here are a couple worth mentioning from my loop around the exhibitor floor.

Sightix stood out to me - in terms of both business model and technology. I wanted to show a good example here but unfortunately couldn't get online demo to work very well (not sure what I am suppose to select) or how to create my own account. (Maybe it is just me ....)

In any case, what I like about it is the focus on social relatioonships and the contextual linkages between people - not just that someone is part of your network but also the how and why. Are they a friend, a customer or your boss? (I think Sightix can show this.) This opens up a lot of possibilities - socially rich ERP/CRM is one good example. Business model was also EASY - along the lines of "Try it - if you like it, pay us on a CPM basis" - making it easy to experiment.

Another was Sun - a couple social computing initiatives in particular - Zembly.com and Sun's offering around GlassFish and Shindig (for OpenSocial) - that deserve a closer look. I also had an interesting discussion with Angelo Rajadurai (~~Rajadouri~~) regarding how to "integrate" - or better yet - cross fertilize (my words) disparate but related communities. Is this the new (or is the new new) Sun?

2008-09-12T14:37:00.010-04:00

Community Lifecycle

The following diagram (borrowed from Forrester's Jeremiah Owyang) shows some steps a successful community goes through - from Conception to Maturity - as measured by member activity.

Nothing controversial here - Strategy, Research, Launch, Kickstart, Growth, Ongoing Management, Continuous Improvement - but my biggest question ( I am not a Forrester client) is how member activity is measured. Is it total hits or some sort measure of community engagement - such as number of inbound links? In my view and experience, defining the proper metrics is not only critical to defining and measuring success but also in advancing the community across the lifecycle. You want to avoid this:

That is volatility - or worse which is decline. There can obviously be volatility on an hour-to-hour, day-to-day or even week-to-week basis but generally you need an up trend to advance the community from strategy to the continous improvements associated with maturity. And since you can't fix what you can't measure, you need the right metrics.

2008-09-12T14:24:00.009-04:00

Private Communities

In my experience, private communities, while sounding exclusionary - like private gated communities in Florida, are really just purposeful communities - pulling together people for a reason - one that helps and/or rewards all participants.

So they are different from those communtiy efforts seeking to cast the widest net - and the most eyeballs.

A good example might be an M&A group evaluating a company's books, a project team organizing a project or a product management comittee. In terms of customer-focused private communities, - "10 Tips For Building Successful Online Customer Communities" - from Communispace - offers a reasonable starting list. Finding the social glue (#3 on their list) may be the most important - providing sustainability and engagement - and also requires real well-intentioned effort.

2008-01-17T07:04:00.006-05:00

Just returned from Skype Developer Conference in NYC. Also made a presentation on Innerpass' architecture and how it is integrated with/utilizes Skype.

Integration BTW is achieved with a Javascript abstraction. Depending upon OS and browser, Javascript calls either an ActiveX control or a java applet. This symmetry, in my opinion, makes development, debugging and maintenance easier.

For ActiveX, Skype4Com was used.

For Java, Skype4Java I extended/bug fixed Skype4Java and wrapped in a signed applet.

Probably the most interesting topics, from my perspective anyway, was how to deliver Skype functionality to Web browsers - sans Skype client - and utilizing Skype on a server. I am eager to hear more, particularly on client-less call participation. The big question for this scenario was obviously the codecs (how they would be downloaded, how they would run, etc) - although Skype client obviously does more than that (e.g. call routing, call state management) .

2007-07-20T09:15:00.003-04:00

Using XAP Tree

Although I am no longer employed by Nexaweb, I will probably continue experimenting with and possibly using Apache XAP in real projects.

For one community project, while still with Nexaweb I put together an overview of the XAP Tree and posted it here.

The XAP Tree is one of the UI components included in the XAP Build. This document described the components and how it fits with the rest of the XAP framework - shown below.

While I didn't test performance, particularly with large complex trees, it can be built declaratively either statically or dynamically using virtually any server side framework and thus most likely fit into your existing web development methodologies and technologies. It can also save you a lot of time.

2007-06-08T06:55:00.003-04:00

Eclipse DTP

I spent some time with John Graham the lead at the Eclipse Data Tools Platform (DTP) at Eclipse last week. I learned quite a bit about DTP's current state and some future possibilities. Nexaweb uses it for design time binding of its rich application platform to SQL databases. DTP helps make it incredibly easy to select the target database using Eclipse GUI and then generate all the binding metadata.

I usually use Apache Derby or MySQL with it but can also use commercial rdbm's, such as SQL Server.

2007-03-18T13:57:00.000-04:00

If you are going to AjaxWorld next week in NY and want to see something fresh, check this out. Colleagues of mine sure (from Nexaweb) but I am pretty jaded myself. Don't want to spoil their presentation, but should at least get you thinking about how modern applications should be constructed, deployed and managed. Dare I say a better way - particularly as browser and other Internet access technologies improve (as well they should!) and proliferate.

Monday 3:40-4:10 pm.

2007-03-05T16:20:00.001-05:00

Main Takeways from Eclipse Con (Santa Clara, CA 3/4-8)

Eclipse development is about we (us/community), not me or I. Developers need to be social. As someone who has reverbed between engineering and sales/marketing, sure. But for some it may be new. But I am also no longer a committer anywhere - so it is easy for me to say ;-)
Less is more - particularly when it comes to preso's. As an example, James Governor's (from Redmonk) session was minimalistic - some slides had one word on them - but stayed with me.
Got an updated perspective on Eclipse - kind of like a trade association with bits. Some pioneering still to do but I think they are on the right track. One big question I think is how far can an IDE-focused organization go? Developers want tooling but people/enterprises want OS's, platforms, applications, solutions.
Spent time with some of the Apache Maven folks and an interesting company (Simula Labs) they are part of. They cracked open Apache XAP (which BTW had its first incubator release - o.3.0 - Applause!) and worked on a Maven build for it. I look forward to the results and insights (is Maven good for Javascript?).
Eclipse and more generally Open Source business models need more attention. Probably shouldn't rival technical sessions in number, but more depth and analysis would be helpful.
Look forward to followups at AjaxWorld and Open AjaxAlliance F2F in 2 weeks in NY. Ajax was BIG topic here - faster growing than Java - Eclipse-Ajax is even tracking with Visual Studio-Ajax, according to some job-based analysis provided by Don Smith of Eclipse. The game is changing....

2007-02-18T18:07:00.001-05:00

I need to write an ROI calculator as part of some demos at Nexaweb. I may as well show the process. Since this is not intended to be a robust mission critical application, I am really looking for how easy it is to go from concept to development and to iterate (since I am sure there we won't get it right the first time)

So let's start with the ROI logic in - surprise - an Excel spreadsheet. (Excel being where many of these rules and formulas are created.) .

Here is the spreadsheet.

Since Nexaweb technologies is based on declarative XML (XAL), the approach I decided to take (with the help of Fred Mikkelsen) was to use Excel itself to generate the basic XAL markup which would in turn form the basis of the Nexaweb ROI calculator (without the formulas).

So, while formulas would not be preserved, cells and tabs would be. The rest of the development then would be to implement the formulas (such as sum) and spread sheet like functionality (such as auto update when a user exits a cell) with Javascript (added to Nexaweb platform as MCO's) and Nexaweb event handling and UI update technologies.

2007-02-05T18:09:00.001-05:00

Was Steve Jobs right?

Was he right when he dissed Java recently (setting the Java world aflutter)?

Well noone wants to bet against the Legend himself, certainly not a mere mortal like me with far fewer 0 (zero)'s in my bank account (and far fewer bank accounts I am sure), but, let me raise some ahem points:

What are you going to do with all those existing Java applications? We still have Fortran and Cobol.

Why (back to this century) isn't there room for more than 1 "rich client" technology in the browser, phones and end user devices in general? iPhone may support Javascript but can one technology be the be all and end all? (I don't care how cool - or is it hot - Ajax/JS is. It is NOT easy to build robust complex apps in. Things that are easy in Java - like avoiding name space collisions to take a simple example - is too hard in Javascript. And things that are hard in Java - like building a ui table that handles 100,000 rows - is virtually impossible in Ajax.) . Maybe a phone doesn't need to support high performance applications (yet!), but does it make sense to limit client technologies to the flavor of the month? I think not.

Wouldn't it be better to assume there will multiple client technologies? (Ajax - XML/HttpRequest this year but what about next or the next 2? ) There are too many devices. (Heck even Apple might come up with a better rendering technology for cell phones.) So Java vs Ajax vs html/dhtml vs Flex is the wrong question. Wouldn't it be better to plan for multiple uses, faces or skins for an application or for multiple client technologies? Wouldn't it be better to code the UI or at least the user interactions and page flows independent of the client technology du jour and plan for both change as well as innovation? The reality of multi-faceted, dynamic and truly reusable/extensible applications is here - or at least the potential for it is.

And all these changes or lack of easy change cost - in real costs, in time, in opportunities lost.

And, unlike Steve, I can't afford to waste any of my 0's.

2007-01-25T18:34:00.001-05:00

Open Source Software (OSS) and Ajax

In looking at all the open source activity around Ajax (here is a good list including Dojo, Google, Microsoft Atlas, Yahoo - from last year but a good place to start - to which I'd also like to add Apache XAP - Full disclosure: I have been a consultant to Nexaweb - contributors of the technology comprising this Apache incubator project - since the beginning of the year), it got me to thinking - is Open Javascript any different from other open source?

So I went back and started looking at all kinds of data points, ranging from the canonical (such as Cathedral and Bazaar) to the pragmatic discussions of commercial business models (such as Taber) and lots in between (including Greg Stein's Apache vs Eclipse, and looser models like SourceForge) and considered the nature of the language itself.

So where do I come out? I guess the following themes emerge:

Open source is innovation. Since many of the most talented developers and experts are engaged in open source for a whole host of reasons, solving the problems that invariably need to be solved (such as name space collisions, memory utilization, debugging to name a few) virtually require their involvement - particularly if we want these problems to be solved sooner rather than later. Honey attracts the bees - if you want bees, go to where the honey is.
Code is already visible. Isn't Javascript already "open source" by its very nature? Should code be obfuscated or should this be embraced? Embrace it.
Javascript is social. As a key underpinning to social neworking, it is already part of the architecture of participation, of community. Leverage community.
There is a lot to do. To make Ajax really work well, particularly for businesses, more invention and problem solving needs to be done. Again, go to where the bees are.

2006-12-30T10:23:00.001-05:00

2006 - The Year of User Generated Content

It is hardly surprising to call 2006 the Year of User Generated Content, what with rapid adoption (and rich valuations) of YouTube, MySpace and the like - consumer-generated content. Couple that with business user generated content - where the content is commercial , even market oriented, and its impact on the year is even stronger. I would even go so far to say that electronic markets are based on user-generated content - where the content are commercial offers - ranging from lists of offers (e.g. Craigs List) to bids and offers, from ebay to Wall Street.

The technology impact of this is important - namely:

I/O: Frequent updates as opposed to frequent reads add real architectural challenges. Typically scale out and caching solves the frequent read problem pretty well. Frequent writes requires a whole another approach - with risks of dirty data everywhere (particularly if performance is an issue - and where is it not?)
Code segregation: For social networking applications likeMySpace, this enables users and contributors to self-style their pages. In the 1st generation of Web 2.0 this means segregating UI from styling (via CSS for example) but for 2nd generation Web 2.0 (should this be Web 3.0?), this concept extends further - to business logic, to data base queries, etc. 2006 was for 1st generation - expect 2nd generation to be significantly more complex - scaling to enable user styling was hard enough.
Data synchronization: Related to the 1st issue. With high write rates, keeping data fresh is pitted against keeping it fast.

From a business perspective, what matters is monetization - and how does this content generate revenue? This need not be evil nor underhanded - infrastructure enabling such content generation and distribution is not free. For electronic markets, the content makes the market - the market generates revenue for the operator. For non transactional systems, revenue comes from leveraging the network effect. This network effect benefits everyone - users, aggregators and those that want to sell to the aggregated users alike.

And 2007? Is Social Commerce Coming?

I see very many of the same things happening in 2007 in the area of Social Commerce - where Web 2.0 meets E Commerce. While the details and challenges differ, my guess is there will be distinct technology and business model challenges.

But first, a Social Commerce overview:

While originally popularized by Yahoo Shoposphere and currently being pitched by Gartner and others, it seems to be a combination of social technologies, such as blogs and wikis and networking technologies and commerce, commercial transactions. Some categories are pretty well established:

Affiliates and referrals (popularized by Amazon)
Network marketing

while other are emerging, such as:

Group Commerce, whether within groups (physical or virtual) or between them
Social mashups involving commercial transactions

But however it surfaces commercially with whatever monetization strategy, in terms of technology, look for heavy borrowing from Web 2.0 (wiki's, blogging, Ajax) plus:

Distributed transactions - like high frequency/volume of updates, this poses distinct scalability challenges - both in terms of managing state and transaction locks but also propogating transaction state information across a wide number of participants and systems. Under such conditions, a potentially huge number of events and messages could be generated by system and participants alike.
Complex User Interaction models: Along with the potential need to coordinate transactions across independent participants, comes the need to coordinate the participant collaborations as well.
Greater code segregation models: As discussed above, advanced Web 2.0 (Web 2.0 Gen 2) needs modularization beyond UI layout, it needs segregation of business logic from UI and business logic from data access. This is true for social commerce as well. In fact, social commerce is part of Web 2.0 Gen 2.
More nimble UI's: UI's need to be aligned with the user category and be able to be adjusted as user categories change and new ones come on line. Consumer interfaces are obviously different than those for enterprise and SaaS (Software as a Service). This is nothing new - what is becoming increasingly important however are for applications and services to service multiple different user segments. Look to code segregation - discussed above - to be a key enabler.

2006-12-11T11:05:00.001-05:00

Latency Kills

It has been a long while since last post but I would like to post a few times on an area I have hearing a lot about for last few months. It is not a new issue by any means but there has some recent reemphasis on it for a host of reasons. The issue is latency - so view this as a first with possibly more to come.

Remember the Highway Warning - Speed Kills?
Well, in electronic markets it is reversed - Speed Wins - Latency Kills.

From very high frequency markets, as is common in the financial markets, to broader albeit less latency-sensitive ones, such as eBay and shopping cart dependent Amazon, latency can kill business, kill profitability. I'll be continuing to watch this trend over the coming months but let's start by looking at the problem of latency generally and some of the common patterns that emerge.

Latency is a fact of life in computerized systems, particularly distributed systems - in fact is a fact of physics since computers and networks are at their core limited by the speed of light, the speed of electrons pulsing through microprocessors, network cards, optic fibers etc. While the speed of light sounds fast (and it is, more than 186,000 miles/sec as any high school physics book will tell you), its impact starts to add up the more the microprocessor needs to do, the more times memory needs to be accessed, the more times circuit boards are traversed, and, critically, the more physical distance needs to be traversed. And in today's globally networked systems, these distances can be long, sometimes thousands of miles. But even a relatively short hop (from say from a data center in Boston where a stock market order might originate to a data center in New Jersey where the order is executed) can take several ms.

While each component does indeed become optimized - microprocessors go faster and faster (remember Moore's Law), RAM and storage go faster and more optical fiber gets used - the real problem surfaces when all discrete, optimized components and systems are inter-connected as they are in today's Internet and today's electronic markets. Essentially they become co dependent.

Nothing controversial or dramatically insightful here. The main point of this is that there are many disparate sources of latency that multiply as systems and people are connected. And this latency has a business impact.

In high frequency financial markets, latency is measured in millseconds (one thousandth of a second) and even microseconds (one millionth of a second) and delays often mean lost business - business lost because someone else "hit the quote" first. Thousands of orders are at risk - and just one missed order may cost hundreds of dollars or even several thousand dollars.

In eCommerce applications, latency is measured in seconds for user responsiveness and subseconds for the servers servicing these user requests but the problem is compounded due to scale - millions of users. As for its impact, just think what often happens when a shopping cart takes too long to respond ... the shopper cancels the order or just moves on to a competitor's site. How much do these lost orders add up to?

So while the dynamics and requirements of different markets vary widely, because they are all based on interconnected, co-dependent systems and people, latency is an almost everpresent concern and needs to be squeezed out where possible and managed where it is not squeezed out. Common patterns, techniques and technologies have emerged to help solve this problem from a range of angles. Some emerging top contenders include:

High performance stream processing to reduce the time it takes to process high speed streams of data (such as financial or geospatial data)
Non invasive network level latency monitoring to determine how much latency each component in a distributed system contributes in an end-to-end transaction and ideally to route around the bottleneck
Rich Internet Applications (RIA) using Ajax, Flash and other emerging technologies to reduce the number of round trips a browser needs to make with a web server
Real time Content Delivery Networks (CDN) to move real time content as close as possible to where it is needed and to route requests for data to the closest possible source.

It'll be interesting to see how these trends and technologies develop, particularly as user-generated content continues on its exponential path and even trends to the "real time".

2006-01-06T08:09:00.009-05:00

Using Apama Trading Strategies Platform

I have recently had the occasion to pound on the algorithmics trading platform from Apama - both for use at a major Wall Street sell side shop and as a platform for an enterprise risk management system my company is exploring.

The Apama platform offers several components, including:

Event Modeller - an IDE for developing applications. (See screenshot below)
Event Manager - the event processing engine. They call it a correlator since it "correlates" events.
Smart Blocks builder - for building reusable components
Dashboard generator - to generate the UI's
Some out of the box trading analytics(VWAP, EWMA, etc) , utilities (e.g. timer) and calculators (P&L, position, etc)
Some sample trading strategies, such as Statistical arbitrage (StatArb)

Event Manager

The Event Manager is the guts of the system. It processes events guided by a proprietary event definition and scripting language (monitorscript). Events are defined using a simple text-based scheme. For example, here is the definition for a stock quote provided in one of the samples:

event StockTick {
string stockCode;
wildcard integer bidQuant;
wildcard float bidPrice;
wildcard integer askQuant;
wildcard float askPrice;
wildcard integer lastQuant;
wildcard float lastPrice;
}

One of the nice nice things about defining events or objects in this way is the ease in which they are defined. There is also no run time casting like a compiled language - which has both pros and cons. I tend like simplicity - must be simple minded.

Monitor script is more complex. It defines relationships between events. It is designed for speed, to enable processing of large volumes of events looking for potentially complex time-based relationships at high speed. All in flight events are kept in memory to speed access and processing. There are several key constructs but I'll call out 2 in particular:

event handlers. This is fundamental to Apama and where it is optimized. The language sets filters and actions on events. The filters can be very complex.
Routers and Emitters - are used to generate new derivative events

There is a Java interface to monitorscript that most Java developers use but run time processing uses a proprietary binary data format.

Event Modeller

The Event Modeller is the IDE. While they claim it is for business users, I believe it is to be used by application support working with the trading strategist. While not a full featured IDE one has come to expect from Visual Studio or Eclipse, it is indispensable, particularly in its autogeneration of proprietary MonitorScript. And there is no other IDE thjat generates Apama's Monitorscript.

Here is a screen shot taken from a StatArb example.

Like to See...

Beyond performance metrics which I have not had time to analyze yet, biggest thing I'd like to see are ability to process XML data using XML expressions (such as XPath) and some day to process unstructured text in real time.

Overall does its job of processing complex rules at high speed - which is what high frequency quant traders need.

2005-10-21T11:34:00.001-04:00

CEP >= Stream Processing?
A lot has been written recently about complex event processing (CEP) and the question I have is whether or at least how it will be used, in the real world, beyond stream processing. Stream processing - which is not really new to real time developers (of applications like trading systems and security monitors) - is essentially geared to processing data (called events) in real time - in other words, before it is put in database or file system and its context is lost and, critically, time passes (read latency). What is new is that a distinct product category is emerging that offers stream processing in product form with admittedly some new spins (such as injectable scripts) and the general *-abilities that productized software offers.

Advocates, such as David Luckham in particular, suggests that CEP represents a whole new way of analyzing data to infer context where it may have been lost and, by analyzing event sequences across time, can be recovered. Seems like the top usage scenario proferred by most vendors putting themselves in this CEP space (e.g. Apama, Streambase and Aleri), however, is in low latency environments, particularly algorithmic trading. In this scenario, software processes make trading decisions automatically by interpreting price and other feeds within the context of trading strategies designed by the business (typically experienced traders). And latency is the enemy since each automated processes is competing with other automated processes - as well as human traders - for the same quotes. It is essentially a race to the order book.

While these products offer a host of other features (Stream SQL from Streambase is particularly interesting), they are basically each addressing this latency issue by offering - albeit in different ways - better and faster ways of processing streaming data. A worthy endeavor, to be sure, but it is stream processing.

It will curious to see what other types of applications develop and whether they fit the CEP moniker. I haven't seen it yet. The problems of discovering context for an event or other data (whether histroical or real time) do exist but can this crop of technology adequately address it?

2005-10-14T17:41:00.000-04:00

Since this is first post in this blog, I'll use it to say this blog will be mostly about, for lack of a better term, business technology, the use of technology (particularly software) to improve business performance and business-IT alignment - you know work stuff. So topics (read: acronyms/buzzwords) like SOA, EII, BPM, CEP, RSS, Real Time Enterprise, etc. as well as specific applications within various verticals will certainly be inbounds. But I am sure other random topics (personal and otherwise) will creep in as well from time to time.

One topic I have been thinking a bit more than usual about lately has been RSS and blogging - mostly in how it relates - or at least could relate - to SOA and other application data flows. And mostly because a smart guy I know sort of led the horse to water. Some intriguing relationships certainly are possible. Treating RSS as at least a semi-structured feed can certainly help integrate and coordinate human activity with business applications and human-readable information flows with application-readable ones. And maybe even smart or at least interesting things could be done.

Obviously weblogs (recently acquired by Verisign) is helpful for any budding "real time enterprise". On a side note, I really love weblog, once again, like RSS, for its simplicity. Rest and XML-RPC - no complex subscription protocols. Thanks again, Dave.

And still watching next gen (Web 2.0) search engines like Sphere and others because it seems increasingly likely that the 1st gen search providers (we know who they are) will need to start tweaking their algorithms to account for freshness/timeliness AND content quality/relevance as blogs and other content are ranked - or else open the door for new entrants.

Building this out has interesting possibilities ... And a likely topic for future posts.

One unrelated note to close on: Adam Bosworth's piece on intelligent reaction is worth a gander, particularly for those interested in user-driven innovation but really for all since it is applicable to building a demand-driven and reactive business, in general. And maybe not so unrelated to RSS after all....