A key component for Master Data Management (MDM) is data cleansing which has multiple disciplines such as address cleansing or PIM (product information management). My expertise is in PIM, therefore my meetings have been focusing on data in the ERP and Inventory system.

My latest meeting was with an informed Material Manager, he understood the concepts of master data management, after the introduction meeting, he stated that “We had a data cleansing project and it did not work, I ended up going back and correcting the data.” Through the discussion, I came to believe that the data cleansing company, extracted the data and attempted to auto classify a half million records. As a purchaser of these types of services, I asked what was the process for mapping and quality checks?

The business issue is the buying team’s inability to utilize spend analytics and the solution is that the data needs to be referenced to the UNSPSC^® (The United Nations Standard Products and Services Code^®). The scope of the project is mapping the purchasing data to the UNSPSC^®. In my experience, I have identified four general levels of PIM data cleansing, 1) auto mapping 2) auto mapping with a manual review 3) verification and 4) enrichment. The cold hard facts are “buyer beware”.

The detail of the levels are:

1. Auto mapping: if you have a large collection of data, automation is a requirement however there are some issues. First, auto mapping incorrect, incomplete and inconsistent data will result in a system that will still have incorrect, incomplete and inconsistent data. The quality of the auto mapping is dependent on the structure of the data. If the data is structured to a noun or class, the auto mapping process will have high quality rate. If the data is set up as “free text”, the results will be dismal. This method will not address duplication or data quality in your system.
2. Auto mapping with a manual review: this process will take the results of the auto-mapping process and add a step of a manual review of the data. The question of the review, will all records be audited in the review, or is the process to review just the records that when the auto mapping just failed? How will consistency of the audit be managed? Again there are still the inherent issues as described in the auto mapping process.
3. Verification: In order to improve data quality, the data cleansing process requires verification with the manufacturer (service or product). The verification process assures that the purchasing record is set up to the correct manufacturer (referenced to the supplier via the contract), part number for restock ordering, UOM (Purchasing Unit of Measure), description with correctly classified i.e. BEARING, TAPER and the UNSPSC^®. Our process is to request the manufacturer to provide the UNSPSC^®. If the manufacturer cannot provide the UNSPSC^®, the item is correctly classified; the auto map to the UNSPSC^® will be successful. The verification process positions the data to identify duplication, manufacturer obsolescence and inaccurate data requiring additional information from the business to reconcile.
   
4. Enrichment: The fourth level of data cleansing quality, in addition to verifying, the data is enriched, this can be obtaining a price, warranty with the terms, additional description attributes, ECCN (Export Control Classification Number), recommended repair spare part information, eCl@ss, NSN (National Stock Number) or any other data element your business requires.

The conclusion is asking the right questions of how my data cleansing project will be implemented and managed are essential to making it a successful data cleansing project.

Thank you for your comments in “Something beyond MDM is coming your way – would MDM 2.0 fly?” and starting the discussion to expand the definition of MDM to include data integrity, data quality, entity resolution, matching, data integration, governance, metrics and analysis. The topics discussed should also include work flow (management of data and analysts), translation management, data structuring, data profiling, duplication removal, data change management, verification contact management, etc.

The MDM and PIM software industry needs to take a step back to understand actual day to day business requirements of data management to achieve Master Data Quality. Lesson one is that data is created and supplied by many sources in many different formats at various quality levels. Data is created by engineering, submitted by integrators, manufacturers and suppliers. To add to the complexity of the information flow, data is introduced into businesses systems in different departments (engineering or purchasing or maybe plant from maintenance) with different data requirements to meet the needs of that job function. Now the next dynamic is mashing new data to existing legacy data in a number of systems to ensure no duplicates are created, managing obsolete / recommended use and functional equivalents. The old philosophies of a PIM or MDM software to “hold, provide search functionality and maybe a shopping cart” isn’t going to meet the true requirements of the new definitions of Master Data Management.

To meet the new definitions the MDM or PIM software needs to provide horse power to electronically and intelligently processing data to identify exceptions for manual intervention by an analyst. Data should be processed one time to ensure that the data record will be enriched to meet the requirements of the enterprise and then the record is moved to a maintenance program (managed also by the MDM or PIM software). The processing of data needs to be efficient and cost effective, from my perspective the cost of data management should be covered by the cost saving achieved by MDM management.

I look forward to the discussions as the definition of MDM is expanded to include data quality, data governance, data provenience as the software industry provides the intelligence, functionality and business processes to cleanse, enrich and management data for my client to ensure their ability to make confident business decisions based on data integrity and accuracy.

Here is to the future of PIM and MDM!

Jackie Roberts