Analytics and the use of the Cloud

Posted: March 1st, 2010 | Author: julio.luis.miyares | Filed under: Cloud Computing, Cloud Map Reduce |

If understanding Data is the true measure and worth of collecting it in the first place, what is now standing in the way at a minimum of the capability to crunch the data using a slew of tools adapted for that purpose? The answer is nothing , well kind of. With the publicly available services in the Cloud such as Amazon’s MapReduce, the inherent capability it there for all but the most uninitiated to make sense of the mountains of data point collected by a typical high volume website.

Of course one can just plugin Google Analytics onto their website and they are good to go with a robust solution that captures, aggregates ,slices and dices and presents pretty reports via a web form about a host of activity on a website. Nevertheless there are always instances where the need to unlock additional understanding inherent in all that data requires some additional processing and some additional tools.

One of the biggest costs for large scale data processing has usually been the need to have an operational infrastructure in place to excise, collate, aggregate and slice the mountains of data that are generated by your typical website or application usually measured in gigabytes and terabytes. Of course all that operational infrastructure is always bound to some strategic and business critical processing leaving precious little horsepower available for for those extraneous random questions that come up on a recurring basis for any business. By the time the request for that question to be answered is queued up and delivered by the typical IT group ,it is highly likely someone more nimble somewhere else has already answered the question and gotten a lead on adapting their business with the answer in hand.

With the wide availability of Cloud Services such as Amazon’s Map Reduce, the timeline to answer the question can be markedly reduced. Admittedly you still need Technologists either in house or outsourced that understand the components of Map Reduce and how to actually leverage it’s power for the needs of the business but the need to wait for precious cycles on the IT’s host complex, partnering Software engineers and Operational Admins to make the environment available with the hosts and necessary software and connectivity to actually run jobs can be short circuited to almost zero time.
You also need the Analysts that can come up with pertinent questions specific to the business they are in and with an understanding of the data elements and relationships between them that are currently captured or can be captured and can help answer specific questions. The technology is not smart enough to do it by itself.

A non technical question , are mature companies with a long standing streak of being in business availing themselves of this form of Cloud services computing? I would not expect them to have the same drive to operate on a shoestring as the majority of startups do but are Cloud services of viewed as an valid option by most IT groups in mature organizations?

Yes, more then a handful of companies outsourced their Development but especially Networking over the course of the past few decades leaving those esoteric functions to entities like MCI that had scale and a knack for staying on top of the evolving technologies shuffling bits around.