Courtesy of Computer World UK

The unprecedented growth in the volumes and complexity of business information is ushering in the era of "Big Data". The phenomenon is not a "thing" but instead a strategic approach that crosses many IT borders and so poses one of the biggest challenges currently facing CIOs from companies of all sizes.

IDC defines the Big Data world as: "A new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high-velocity capture, discovery, and/or analysis. Big Data is a horizontal cross-section of the digital universe and can include transactional data, warehoused data, metadata, and other data residing in ridiculously large files."

As the amount of data held in these ridiculously large files continues to explode, the process of analysing large data sets will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office.

"Leaders in every sector will have to grapple with the implications of Big Data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future," the McKinsey report Big data: The next frontier for innovation, competition, and productivity, predicted.

The scale of this "exponential growth" in data is staggering. The fifth annual study of the digital universe by analyst company IDC revealed that in 2011, the amount of information created and replicated surpassed 1.8 zettabytes (1.8 trillion gigabytes) - growing by a factor of nine in just five years.

"Like our physical universe, the digital universe is something to behold - 1.8 trillion gigabytes in 500 quadrillion "files" - and more than doubling every two years. That's nearly as many bits of information in the digital universe as stars in our physical universe," said IDC senior vice president John Gantz.

This growth in business data is due to multiple factors, with the main culprits being the falling cost of data storage hardware, the increasing density of storage, increased digitisation of business processes together with the growing prevalence of storage-intensive data such as video and machine-generated log files.

The net result of this data explosion is, according to Andrew Buss, service director at Freeform Dynamics, that many organisations, large and small, are becoming increasingly overwhelmed by storage - by the amount they have to store, by getting it to the systems that matter, and by the huge variety of the types of data being held. He pointed out that, while many companies may not think of it as such, this is very much a "Big Data" problem, whatever the scale.

"Many of you may think that Big Data is yet another hyped-up fad that will pass with time or impact only the largest or most demanding organisations with massive amounts of data. However, the reality is that it is a question of scale and the ability to deal with it in terms of investment capability, existing infrastructure and skills," Buss said.

"Companies of all sizes are feeling the effects of the data explosion - a mid-size organisation may struggle to deal with a few tens of terabytes of data, while this may be thought of as relatively simple by a global company regularly dealing with petabytes or even exabytes themselves."

And it is not just the volumes of data that pose a management headache for CIOs - it is the changing nature of the underlying file types. According to Clive Longbottom, service director at analyst company Quocirca, the biggest problem comes from the fact that the type of data that businesses need to manage and interpret is changing.

"When it was rows and columns of figures held in a standard database, life was - relatively - simple. It all came down to the speed of the database and the hardware it was running on. Now, more and more binary large objects reappearing in the databases - and these require different approaches to be able to identify and report on what the content actually is and in identifying patterns and making sense out of what this means to the end user," Longbottom said.

"Even worse is the fact that less information is making it into standard databases - yes, there's still an increasing amount of numerical and textual data being created that resides within a database, but this is being outstripped by the amount of information that is being created in a more ad hoc manner with files that lie directly in a filing system."

Throwing more storage at the problem or improving the performance of existing storage systems will not solve the problem, Freeform Dynamic's Buss warned. "Merely investing in bigger and faster storage arrays, multi-gigabit networks and lots of flash cache may alleviate the symptoms - but like putting lipstick on a pig - the underlying issues remain. Dealing with the problem of Big Data requires a change in mind-set from trying to manage storage systems to trying to manage information."

It is apparent that, in the face of this assault, traditional storage and management technologies are simply no longer able to cope. Forrester notes that, at extreme scale, traditional data management and business intelligence (BI) become "impractical". This results in organisations losing control of their data and failing to harness it effectively to gain insight that could be used to drive greater business performance.

Brian Hopkins, principal analyst at Forrester, explained that Big Data helps firms work with extremes to deliver value from data cost-effectively. However he cautioned that CIOs must understand that the world of Big Data is not business as usual: "In fact, Big Data will disrupt the data management landscape by changing fundamental notions about data governance and IT delivery. Take the time to understand Big Data as well as its implications and begin a balanced approach that considers more than just the technology hype," Hopkins said.

As firms seek value from digital and analogue sensors, social media, financial systems, emails, surveys, and customer call centres - to name a few - new tidal waves swell on the horizon. All of this data promises to open up new frontiers of business opportunity - if we can figure out how to turn it into insight. Business-as-usual BI and IT delivery approaches, however, will not generate the future results your business expects."

For companies that can tame this Big Data tiger, the rewards can be substantial. A December 2011 poll of 63 companies with at least 25 terabytes of active business data conducted by Aberdeen Group found that leading companies - the top 50% in terms of data performance - were able to meet their user demand for information more than four out of every five times. Such companies also had a data infrastructure the enabled employees to spend half as much time looking for information, and access almost three times as much business data as followers.

McKinsey predicts that leveraging Big Data successfully will underpin new waves of productivity and growth for companies of all sizes across all key verticals. For example, it estimates that a retailer using Big Data to the full has the potential to increase its operating margin by more than 60%.

In order to begin to address these issues McKinsey advises that multiple factors will have to be addressed. Policies related to privacy, security, intellectual property, and even liability will need to be modified up in a Big Data world. Looking across technology, people and processes, corporates will need to put the right talent and infrastructure in place but also shape workflows and incentive packages to optimise the use of Big Data.

The value of such a holistic approach was echoed by Forrester. In the analyst company's June 2011 Global Big Data Online Survey, 70% of respondents said that Big Data is or will be a collaborative effort between business and IT. The poll revealed Big Data teams of skilled technicians are working increasingly closely with business executives to discover new uses for more data. In order to succeed in this brave new world of Big Data CIOs must, according to Forrester partner with business peers to identify opportunities and solutions.

"While business executives at some firms immediately get it, we expect the concepts of Big Data to be foreign to most. CIOs who understand the state of the art and are perceived as strategic partners will be well positioned to help executives justify investments and reap new benefits. Conversely, CIOs without a seat at the business strategy table may struggle," the Forrester survey warned.

Forrester also advises CIOs to turn over part or all of Big Data solutions delivery to business leaders. A recurring theme in its research was the level of business involvement in Big Data solutions. CIOs are also warned to be ready to support rapid growth as commonly firms beginning a Big Data solution as a pilot with a few terabytes of planned growth, were found to often find themselves with more than a petabyte very quickly.

According to Gartner, enterprise architecture (EA) practitioners have a major role in ensuring their organisations maximise the business opportunities posed by Big Data. The analyst group notes that Big Data can makes organisations more productive by enabling people to derive value from the increasingly diverse data sources, and thus to identify previously unseen opportunities.

"Traditional approaches to EA are significantly impacted by Big Data," said David Newman, research vice president at Gartner. "For the EA practitioner, the balance shifts from a focus on optimisation and standardisation within the organisation, to lightweight approaches that focus on harmonisation and externalisation across the broader enterprise ecosystem. Big Data disrupts traditional information architectures - from a focus on data warehousing (data storage and compression) toward data pooling (flows, links, and information shareability)."

Newman added that careful planning must be undertaken to determine the best tools and techniques for analysing complex datasets. These include skills in statistics, machine learning, natural-language processing and predictive modelling. In addition, practitioners must help teams understand how to use Big Data visualisations techniques, such as tag clouds, clustergrams, history flows, animations and infographics. Teams should use low-cost, open source tools in early pilots to demonstrate the feasibility of Big Data projects.

Instead of "Big Data" Quocirca suggests that CIOs aim for an "unbounded data" strategy - the capability to pull together data and information from a range of disparate sources and to make sense of it in a way that an end user needs.

Quocirca recommends that organisations seek data management solutions that can deal with different data types, including text, image, video and sound - from both within and outside of and organisation's environment. According to the analyst company, it is essential that a successful approach uses meta data and pointers to minimise data replication and redundancy, and not simply create a new, large data warehouse that will only make the existing problem worse. Such a system should be able to present findings back to business decision makers -not only based on what has already happened - but also incorporate the capacity predict with some degree of certainty what may happen in the future. The system should also be inherently fault tolerant and flexible enough so that more resources can be apply easily to the system as required to cope with future demand.

It is clear that the brave new world of disruptive Big Data has the potential to give CIOs big headaches, but IDC's Gantz remains upbeat. In fact, he argues that Big Data is putting CIOs on the threshold of a major period of exploration of the digital universe.

"This period of 'space exploration' of the digital universe will not be without its challenges. But for the 'astronauts' involved - CIOs and their staff - it represents a unique, perhaps once-in-a-career opportunity to drive growth for their enterprises," Gantz said.

"They will need to lead the enterprise in the adoption of new information-taming technologies, best practices for leveraging and extracting value from data, and the creation of new roles and organisational design. Each step will require organisational change, not just a few new computers or more software. The success of many enterprises in the coming years will be determined by how successful CIOs are in driving the required enterprise-wide adjustment to the new realities of the digital universe."