During my morning data quality, MDM and data cleansing reading, I happened upon this on a help site and the million $$ question:
I have a scenario to build a data flow task for Data Cleansing.
Logic 1 to be build:
Source data would be like 1050 and I should convert it to 1.050
Source data would be like 085 and I should convert it to 0.85
Profiling, structuring or normalizing data without any referential information risks errors in business use, especially if the data is use for purchasing or maintenance. If the goal is to automate the data normalization, the data needs to be referenced to metadata, 1050 could be a part number? Or a quantity? It could be an attribute representing a measurement such as length or diameter. Is it an inch or foot or meter?
Tags: Data Cleansing, Data Profiling, data quality, DATAForge, dataquality, manufacturing, masterdata, mdm
Totally agree Jackie…
Before considering any form of “Transformation”, the “business rules” for the field should be fully understood:
I would want to have:
1. Business name of the data field(s):
2. Permitted values:
3. Business meaning of each permitted value:
4. Interdependencies with other data:
5. Field precedence:
The “business rules” should be centrally maintained, and visible to the business.
All “Transformation” rules should also be centrally maintained (and not just coded into an ETL script).
Actual transformations, and exception processing (when input data could not be transformed) should be recorded in an audit trail.
Regulators will increasingly look for a “Data Audit Trail” – be warned.
Ken