The costs of poor data quality

A bridge or a shopping complex simply cannot be built with any structural defects.

The potential consequences are obvious, and our inherent understanding of the time, effort and costs of having to work around such problems over the years, and to ultimately have to redo the initial effort, usually ensure an acceptable level of quality is adhered to in the first place.

Unfortunately not so in the data creation business, aka "your business" - we all, to a lesser or greater degree, quite happily allow incorrect, invalid, inconsistent and "just plain wrong" data to be entered into company databases.

The consequences go largely unnoticed, or are simply accepted as part of normal practice. If we get a purchase order wrong, we just redo it. We accept the fact that we need to spend time every day in hunting down correct, trustworthy information to complete or fix a business transaction. We never realise which opportunities we have missed out on, because our reports were wrong. These things all cost money. Because high quality data is in general not expected in the first place, poor quality data is tolerated.

Most organisations, without really realising it, work around poor data quality daily. In fact, some companies even have dedicated people or teams whose main task it is to deal with the problems caused by poor data quality. Unfortunately, the names given to such departments make them seemingly acceptable practice - in the retail industry we have people dedicated to resolving "out of stock" situations, in financial services we often have "special projects" who are tasked with handling incorrectly calculated charges, invoices, interest and premiums, for example.

There are therefore clearly many direct and indirect costs associated with poor data quality, but if no one is looking for them, they are not obvious. In general, the effects of poor data quality manifest further down the line in other departments or at higher levels in the organisation, or, at worst (and, sadly, frequently so) within the customer base.

There are many direct and indirect costs associated with poor data quality, but if no one is looking for them, they are not obvious.

Bryn Davies, Sybase South Africa, technical director

For example, an irate car hire customer calling to complain about an incorrectly recorded mileage for which she has been billed. There is also usually a time lag between when poor data is captured, and when its consequences are felt. For example, incorrect recorded package dimensions only get noticed when the crates don't fit into the designated warehouse racking. It is this latent effect of poor data quality that makes it difficult to quantify the resultant costs, but on the other hand, the evidence has proven time and again that tackling data quality ultimately leads to improved business process efficiencies and lower costs (see my previous Industry Insights).

So what are the types of costs that result from a lack of quality in corporate databases? At a high level, the two major categories are: (a) the costs of redoing transactions due to initial errors, and (b) the costs of lost opportunities.

The first category is often referred to as "scrap and rework", a term analogous to the manufacturing industry where it is clearly costly to have to scrap semi-fabricated units or to rework raw materials, due to initial errors.

The same is true for data: it takes people's time and other costly resources to fix what continually manifests as business transaction errors. To make matters worse, this often occurs only once the company has been alerted by the customer on the receiving end of the mistake, thus incurring additional costs in ensuring ongoing customer loyalty.

There are other types of more pernicious consequences, with attendant costs, to low quality data: frustrated employees who have to deal with the resultant inefficiencies, a need to perform frequent verification of figures in reports, re-coding of computer applications and database fixes, costs of multiple redundant copies of the same data (often "private" copies due to mistrust of data maintained in corporate systems), and of course diminished business agility. Finally, the ultimate costs in today's regulated environment are fines and jail terms due to serious compliance issues. Data quality is not just about clean data, it is about good business.

Properly implemented data quality initiatives will spearhead better practices, more efficient processes, better application design, and more positive customer, supplier and partner interaction. I often get asked what the costs of implementing a data quality programme in a company might be. This is the wrong question: what should be asked is "what are the costs of NOT implementing a data quality programme?"