Courtesy of btob

For marketers, Big Data is front and center today. It's seen as essential in getting the most out of direct marketing programs, and those unable to extract the right information from these massive databases may be at a competitive disadvantage.

"In the past, it's been prohibitively expensive to actually make sense of [so much data], to infer opportunities," said David Cummings, CEO of marketing automation company Pardot. "Now, tools exist to see, for example, website traffic patterns hour by hour, minute by minute or Tuesday at 2 p.m."

While that may be possible, marketers continue to feel overwhelmed, as reflected in IBM Corp.'s "Global Chief Marketing Officer Study" released last fall.

Based on interviews with 1,734 CMOs from 64 countries in 2011, the study identified Big Data as the greatest marketing challenge of all, cited by 71% of respondents. Also, 79% believed the level of data complexity will be high or very high over the next five years; but almost as many said they felt unprepared to handle it.

The difficulty in coping with Big Data is felt even among those companies IBM characterized as "outperforming organizations." Here, 65% reported being unprepared to cope with the explosion of data, compared with 77% of "underperforming organizations."

"You can use some tools to get some basic intuition about data; but, once you start collecting enough data, the problem lies in interpreting it," said Ezra Fishman, director-marketing with video hosting and analytics company Wistia Inc. "It becomes exponentially harder, because each part of the data tells its own story; how it combines tells a story that makes sense to me."

Fishman said every touch on the Wistia website and inquiries about its products are being captured by Pardot technology, augmented by advanced reporting and data warehousing from GoodData Corp. That is followed by analyses that attempt to connect the various activities.

But he acknowledged that ever-growing data sets can be "a mess."

"We have at our fingertips all this data, and in that mess there are answers to such issues as setting up our website optimally, how best to offer free software trials, the optimal pricing for our plan and so forth," he said. "The answers are hidden but are there. We are confident it's just about interpreting the data."

Big Data, and the technology necessary to parse it, is already making for inroads in operational efficiencies. Last month, LexisNexis Risk Solutions began migrating its insurance-support products to an in-house Big Data processing platform to help insurers assign premiums more accurately, as well as better understand their customers and risk throughout the policy life cycle and drive better profitability.

In March, the federal government launched the "Big Data Research and Development Initiative," a $200 million research project to better understand science research opportunities, national defense, energy efficiency, healthcare and education.

Vendors are responding to the Big Data conundrum. Next month, Yahoo Inc. will roll out Genome, a solution to manage Big Data for advertisers, building on data from Yahoo's Interclick behavior identification and targeting service combined with data gleaned from its multiple Yahoo sites and third-party data compilers. The goal is to mine these data with predictive modeling to determine optimal audiences.

And last month, database marketing company Infogroup announced plans to use a data-integration platform from Talend to integrate client data with "a single universal identifier." Also this spring, Lyris Inc. began using Talend solutions to streamline data management for online marketing campaigns.

One missing element, however, is adequate staff to make sense of all this, said Phil Fernandez, CEO of marketing automation company Marketo Inc.

"The talent shortage is especially acute in the marketing department," Fernandez wrote on the Marketo blog. "The practice of marketing is now becoming more and more data-driven; but there are still not enough marketing executives who are truly experienced, or even comfortable, with making sense of the data and actually putting it to work to drive revenue growth."