Lately I have had a number of meetings with material and purchasing managers and I have come to two distinct conclusions from the feedback. First, businesses recognize the importance of data quality and have attempted to work on improving their information with either implementing an internal program or hiring a company to provide data cleansing services. The second conclusion is that the activity of Data Cleansing has such an incomplete and broad definition, I reference the blog post by Koa Beck in Gartner Releases Its Magic Quadrant for Master Data Management, “while we continue to monitor the aggregate MDM market, we still believe that it is premature.”
A key component for Master Data Management (MDM) is data cleansing which has multiple disciplines such as address cleansing or PIM (product information management). My expertise is in PIM, therefore my meetings have been focusing on data in the ERP and Inventory system.
My latest meeting was with an informed Material Manager, he understood the concepts of master data management, after the introduction meeting, he stated that “We had a data cleansing project and it did not work, I ended up going back and correcting the data.” Through the discussion, I came to believe that the data cleansing company, extracted the data and attempted to auto classify a half million records. As a purchaser of these types of services, I asked what was the process for mapping and quality checks?
The business issue is the buying team’s inability to utilize spend analytics and the solution is that the data needs to be referenced to the UNSPSC® (The United Nations Standard Products and Services Code®). The scope of the project is mapping the purchasing data to the UNSPSC®. In my experience, I have identified four general levels of PIM data cleansing, 1) auto mapping 2) auto mapping with a manual review 3) verification and 4) enrichment. The cold hard facts are “buyer beware”.
The detail of the levels are:
- Auto mapping: if you have a large collection of data, automation is a requirement however there are some issues. First, auto mapping incorrect, incomplete and inconsistent data will result in a system that will still have incorrect, incomplete and inconsistent data. The quality of the auto mapping is dependent on the structure of the data. If the data is structured to a noun or class, the auto mapping process will have high quality rate. If the data is set up as “free text”, the results will be dismal. This method will not address duplication or data quality in your system.
- Auto mapping with a manual review: this process will take the results of the auto-mapping process and add a step of a manual review of the data. The question of the review, will all records be audited in the review, or is the process to review just the records that when the auto mapping just failed? How will consistency of the audit be managed? Again there are still the inherent issues as described in the auto mapping process.
- Verification: In order to improve data quality, the data cleansing process requires verification with the manufacturer (service or product). The verification process assures that the purchasing record is set up to the correct manufacturer (referenced to the supplier via the contract), part number for restock ordering, UOM (Purchasing Unit of Measure), description with correctly classified i.e. BEARING, TAPER and the UNSPSC®. Our process is to request the manufacturer to provide the UNSPSC®. If the manufacturer cannot provide the UNSPSC®, the item is correctly classified; the auto map to the UNSPSC® will be successful. The verification process positions the data to identify duplication, manufacturer obsolescence and inaccurate data requiring additional information from the business to reconcile.
- Enrichment: The fourth level of data cleansing quality, in addition to verifying, the data is enriched, this can be obtaining a price, warranty with the terms, additional description attributes, ECCN (Export Control Classification Number), recommended repair spare part information, eCl@ss, NSN (National Stock Number) or any other data element your business requires.
The conclusion is asking the right questions of how my data cleansing project will be implemented and managed are essential to making it a successful data cleansing project.
