Comments on: Data Cleansing to Achieve Information Quality

By: Recently Read: March 22, 2010 « Reblogger Memo Links

Recently Read: March 22, 2010 « Reblogger Memo Links — Tue, 30 Mar 2010 18:25:40 +0000

[...] Data Cleansing to Achieve Information Quality – Jackie Roberts raises some interesting questions regarding the efforts needed to cleanse data though multiple stages of analytics and processes to achieve appropriate information quality. [...]

By: William Sharp

William Sharp — Thu, 11 Mar 2010 15:57:18 +0000

@Jackie & @Ken … Were you guys on the same projects I was a few years ago? I was an ETL developer on several data warehousing projects when I developed this philosophy. I’ve even been asked to reto-ETL a broken warehouse were the original ETL developers forgot to add — primary keys! I’ve learned to never rule anything out as far as data is concerned. I’ve seen every special character on the keyboard in a dataset before. Which leads me to believe someone loaded their delimiters.

I think people take data for granted like we do hardware. Not to mention, how often and long do people talk about the UI features they want? It’s more tangible to business users, somehow?

In the end, it’s always going to be there and we need to be slick in the ways we try to prevent it and clean it up when it does happen.

By: Ken O'Connor

Ken O'Connor — Thu, 11 Mar 2010 15:02:38 +0000

Hi Jackie,

I completely agree with your reply to William.

The attitude too often taken is: “Our job is to implement the new system, not clean up someone else’s mess”.

Ken

By: Jackie Roberts

Jackie Roberts — Thu, 11 Mar 2010 14:37:15 +0000

William, I believe that the topic of verification or sometimes referred to a “catalog @ source” is a critical step of data quality but is usually overlooked during the traditional data migration phase of a software implementation which results in bad data moved from a legacy system to a new system . . . only providing the business with limited data accuracy for reporting.

By: William Sharp

William Sharp — Thu, 11 Mar 2010 14:05:59 +0000

1. “does an address or a product detail meet the requirement if only classified?” – not in my opinion due to the fact that without verification the data cannot become information.

2. should verification at source (contact for address or manufacturer / supplier for product) be required at initial setup of the data in the system or maintenance scheduled as part of the data governance program? – yes, otherwise what are we governing and by what laws/rules.

3. Is the data incomplete? – yes, often address data is incomplete or invalid. Either is insufficient to become information.

4. Does the MDM process include a question / answer scenario to complete the data? – I believe so. This is the voting portion of the governance initiative, functioning as the business’ chance to manage its own information.

Great post Jackie!! Keep it up! I’ll be mulling these questions over and forming a blog post soon!