What, exactly, is "big data"? In a recent interview (video posted below), Duncan Stuart, director of research for TMT at Deloitte Canada, defined it as 5 petabytes or more.

Many in the industry define big data not just by volume, but also by velocity and variety. But 5 PB is a good, simple threshold for current measurements. It's also a moving target - bear in mind that 5 PBs may be what you find in a tablet computer three years from now.

Such large, multi-petabyte sites are likely to be proliferating. In a survey I helped conduct last fall as part of my work with Unisphere Research/Information Today Inc., nine percent of the companies participating reported data stores exceeding 1 petabyte. For comparative purposes, a petabyte is 1,000 times bigger than those 1-terabyte databases that made news just a decade ago.

In its report, Deloitte spells out the challenges with sizing the big data market - there are varied definitions of what big data is, it is still early in the adoption cycle of big data technologies, and most of the companies who are doing big data do not disclose their spending.

Nevertheless, Deloitte pegs the size of the big data market at about $1.3-$1.5 billion in 2012. The consultancy also predicts that this year, we'll see big data experience accelerating growth and market penetration:

• "As recently as 2009 there were only a handful of big data projects and total industry revenues were under $100 million. By the end of 2012 more than 90 percent of the Fortune 500 will likely have at least some big data initiatives under way."
• But the industry is still in its infancy, Deloitte cautions. "Big data in 2012 will likely be dominated by pilot projects; there will probably be fewer than 50 full-scale big data projects (10 PBs and above) worldwide."

There are compelling reasons for companies to pursue big data. "Big data can see through time, big data basically allows you to see everything all at once, and in much finer detail," says Stuart. "Instead of looking at my customer's behavior once a month, I can look at it every minute of every day. That kind of insight is very, very powerful. It allows me to serve my customer better - either very large or very fast or both, requires the big data toolset."