Courtesy of Information Week
There's an argument
for integrating master data management with big data, but much work
needs to be done before that happens.
As the discussion around big data gains momentum
and substance, questions are being raised about how master data
management (MDM) fits into the picture. From what I've seen, the
amalgam of master data management with big data is a lot like the
Sherlock Holmes story about the dog that didn't bark--what's
noteworthy is that there's nothing of note.
I heard (or rather, didn't hear) something similar at the recent
MDM and Data Governance Summit in San Francisco, where I presented
as well as was part of a panel on master data management and data
governance. The message in a nutshell, is that there's definitely a
use case for integrating master data management with big data, but
there hasn't been much distance covered yet.
The value proposition for bringing master data management into
big data analytics is essentially no different from the standard
MDM use case: providing identity (i.e. uniqueness) to entities.
Take the product marketing team that's interested in collecting
and collating comments made by consumers across the Internet, in
discussion forums, personal blogs and other hard-to-decipher
places. The data it collects follows a typical big data
1. Large in size and can quickly go into hundreds of terabytes
2. Semi-structured. (Take the phrase "unstructured data" with a
grain of salt, because the contents may be free form, but there's
almost always some sort of interesting structure around it that can
be leveraged for analysis.)
3. Is usually high, coming in fast and furious, which poses a
challenge to conventional data extract, transform, and load (ETL)
architectures and relational databases.
The product is the central point of intersection of master data
management and big data. There's potentially important information
in all the consumer feedback, but unless we tie in the free-form
comments to our product catalog, what use is all that
This requires a classic both-ends-to-the-middle
approach--technologies and techniques (like natural language
processing) to help make sense of the free-form comments at one
end, and an accurate and consistent product hierarchy (the product
master) at the other end. MDM is clearly an enabler here.
I've heard similar stories elsewhere. At a recent user group
conference for a large data warehousing appliance vendor, a
pervasive message was the need for mastering identity to support
At the conference, I had an interesting chat with the founder
and CEO of Reltio, a venture capital-funded big data initiative,
(don't bother visiting the website since it's in stealth mode right
now). He had a sound understanding of the challenges of integrating
MDM with big data, having previously led product marketing and
strategy at a leading MDM vendor. His thoughts around using MDM to
support big data analytics resonated with the emerging theme: MDM
helps lay the foundation for big data analytics, but we're in the
early stages of defining this integration framework.
Beyond big data, the conference featured the usual (and
successful) mix of decision-makers, practitioners, and vendors. For
example, an interesting presentation on a data governance approach
formulated on the basis of Malcolm Gladwell's tipping point theory
(loosely defined as a quick and unexpected change that dramatically
transforms a situation) can be leveraged to drive data governance
deeper and wider in the organization.
The challenges of data governance are fairly ubiquitous. The
most common is a shortage of budget and resources, though not, in a
curious twist, particularly driven by a lack of executive-level
interest in data governance. The mantra for data governance seems
to be "doing more with less".
In the products section, two offerings caught my eye, neither of
which are new, but seem to have evolved into mature products worthy
Networks' MDM/DG product offers a pleasant change from the
integration-heavy architectures of the dominant MDM platforms out
there, e.g. IBM, Informatica, Oracle, etc. Orchestra Networks
starts with a facility to model your data and take it from there,
which presents a much cleaner and more manageable approach,
particularly, I suspect, for departmental and smaller MDM
its foray into open source and the cloud. It's a product range
that's expanding and maturing fast, and seems to be at or close to
the point (call it the "tipping point") where it merits serious
consideration in your MDM strategy, especially if you aren't
already locked into the IBM, Oracle, or SAP stack. Keep in mind, of
course, that "open source" doesn't always equate to "free."