Courtesy of Information Week

There's an argument for integrating master data management with big data, but much work needs to be done before that happens.

As the discussion around big data gains momentum and substance, questions are being raised about how master data management (MDM) fits into the picture. From what I've seen, the amalgam of master data management with big data is a lot like the Sherlock Holmes story about the dog that didn't bark--what's noteworthy is that there's nothing of note.

I heard (or rather, didn't hear) something similar at the recent MDM and Data Governance Summit in San Francisco, where I presented as well as was part of a panel on master data management and data governance. The message in a nutshell, is that there's definitely a use case for integrating master data management with big data, but there hasn't been much distance covered yet.

The value proposition for bringing master data management into big data analytics is essentially no different from the standard MDM use case: providing identity (i.e. uniqueness) to entities.

Take the product marketing team that's interested in collecting and collating comments made by consumers across the Internet, in discussion forums, personal blogs and other hard-to-decipher places. The data it collects follows a typical big data pattern:

1. Large in size and can quickly go into hundreds of terabytes and beyond.

2. Semi-structured. (Take the phrase "unstructured data" with a grain of salt, because the contents may be free form, but there's almost always some sort of interesting structure around it that can be leveraged for analysis.)

3. Is usually high, coming in fast and furious, which poses a challenge to conventional data extract, transform, and load (ETL) architectures and relational databases.

The product is the central point of intersection of master data management and big data. There's potentially important information in all the consumer feedback, but unless we tie in the free-form comments to our product catalog, what use is all that information?

This requires a classic both-ends-to-the-middle approach--technologies and techniques (like natural language processing) to help make sense of the free-form comments at one end, and an accurate and consistent product hierarchy (the product master) at the other end. MDM is clearly an enabler here.

I've heard similar stories elsewhere. At a recent user group conference for a large data warehousing appliance vendor, a pervasive message was the need for mastering identity to support consumer analytics.

At the conference, I had an interesting chat with the founder and CEO of Reltio, a venture capital-funded big data initiative, (don't bother visiting the website since it's in stealth mode right now). He had a sound understanding of the challenges of integrating MDM with big data, having previously led product marketing and strategy at a leading MDM vendor. His thoughts around using MDM to support big data analytics resonated with the emerging theme: MDM helps lay the foundation for big data analytics, but we're in the early stages of defining this integration framework.

Beyond big data, the conference featured the usual (and successful) mix of decision-makers, practitioners, and vendors. For example, an interesting presentation on a data governance approach formulated on the basis of Malcolm Gladwell's tipping point theory (loosely defined as a quick and unexpected change that dramatically transforms a situation) can be leveraged to drive data governance deeper and wider in the organization.

The challenges of data governance are fairly ubiquitous. The most common is a shortage of budget and resources, though not, in a curious twist, particularly driven by a lack of executive-level interest in data governance. The mantra for data governance seems to be "doing more with less".

In the products section, two offerings caught my eye, neither of which are new, but seem to have evolved into mature products worthy of attention.

Orchestra Networks' MDM/DG product offers a pleasant change from the integration-heavy architectures of the dominant MDM platforms out there, e.g. IBM, Informatica, Oracle, etc. Orchestra Networks starts with a facility to model your data and take it from there, which presents a much cleaner and more manageable approach, particularly, I suspect, for departmental and smaller MDM implementations.

Talend continues its foray into open source and the cloud. It's a product range that's expanding and maturing fast, and seems to be at or close to the point (call it the "tipping point") where it merits serious consideration in your MDM strategy, especially if you aren't already locked into the IBM, Oracle, or SAP stack. Keep in mind, of course, that "open source" doesn't always equate to "free."