For the love of Data (Quality)

"Cleaning the lake or reducing the pollution from the factory" - is an analogy used by Thomas Redman. It perfectly paints a picture of data quality issues 'we' all face in our data management projects. 

In projects we often have to struggle against forces that 'just wanna create the freekin report'. Whether the data is wrong is of no concern. In these instances the goal apparently is the information system ('our DWH is running', 'The report is build' or 'SAP is live') and not the data. Put in other words; data is often treated as a by- product and the information system is the main product.

Lets take a closer look - by using (among others) Richard Wang's analogy with a manufacturing process¹:

I have never seen any manager, CEO or foreman happy with a succesful implemented assembly line but a lousy product. Have you? In software engineering I sometimes have the feeling we lost touch....

The Information System - be it a Data Warehouse, a report or an ERP - is not the purpose, it is a means to an end. And the end should at least be sufficient data quality (where data quality is defined in the perpective of the customer, fit for his/her task). 

The cool thing about the (somewhat) oversimplified analogy of Richard Y Wang is its usefulness for another reason. It stresses the system perspective you gotta have on dealing with data quality issues. You cannot go about 'cleaning your lake while the factory is still polluting'. Producing quality Information Products is executed by means of a system. Now, do not translate this 'system' into 'information system'. This system consists of people, processes and technology. Dealing with data quality issues requires a system perspective to really add value in terms of better quality products and a 'greener' environment.

So - do not blame your ERP department for creating bad data

So - do not blame your report builder for creating useless reports

So - do not blame the person entering the data

Maybe, something to consider; who do you think is accountable for organising the 'system'? Yes - management should embrace quality in its DNA...

I know I am corny - management should have read Deming, Juran, Crosby in their MBA's. Knowledge that is like half a century old.

 

^{1 - Richard Wang - A Product perspective on Total Data Quality Management - feb.1998, Communications of the ACM}

Comments

[Hennie]

Hi Ronald,
Off course, there's a need for good data for analytical purposes. In too many projects i've seen that the reporting guy had a problem with the quality of the data. For example, in one project the BI department needed reliable data and they had to pay (and not for low price) to the applicationowner to fix the problem. So, for who is this a problem? Fixing dataquality afterwards costs huge amount of money, effort and time.

Too many software products are bought just for the operational process and the analytical process is not taken into account. The BI department is not (always) involved in selection processes. So I think the polluting will continue in the coming years but may be the trend is going downwards, I hope.

Hennie

[Cheap Jerseys]

Business sweetens pleasure, and labour sweetens rest.