What`s driving the need for data quality?

In a 2001 report by The Data Warehousing Institute (TDWI), 48% of the respondents claimed their companies had "no plans" for a data quality initiative. A more recent survey by TDWI in March 2006 revealed that the percentage of respondents without a plan had dropped dramatically to 24%.

Why the change of heart? As usual both carrots and sticks have had their say.

Perhaps one of the biggest sticks has been substantially increased regulatory compliance requirements: there is nothing more effective than the threat of hefty penalties and/or a stint in jail to get resources mobilised.

While many have yet to make the connection between data quality and compliance, those who have done so have ensured they not only meet compliance requirements, but have also created a much more efficient organisation, resulting from all the attendant benefits of high quality data (see my previous Industry Insights on ITWeb).

Banks, in particular, have had to invest heavily in Basel II initiatives, while ensuring data being used to assess risk is also completely trusted. Others might soon need to comply with Sarbanes-Oxley, Anti Money Laundering, Financial Intelligence Centre Act, and Regulation of Interception of Communications and Provision of Communication-Related Information Act, all of which need to be underpinned by high quality, trusted data.

The recent realisation that an organisation`s data is an asset that needs much better management has resulted in a flurry of interest around topics such as data governance, enterprise information management, master data management (MDM) and customer data integration, also sometimes referred to as "single view of the customer".

In the retail and manufacturing sector, for example, a great deal of work is being done to ensure all trading partners use aligned product data, and so projects for internal data alignment and global data synchronisation are being planned.

But just as tidying up the garage results in more order, so too will any of these initiatives demand that organisations first identify the dirt within the data and clean it up before trying to improve the way it is managed. Therefore any attempts to "govern" data or to achieve a consolidated view of whatever the organisation considers critical to survival and growth, be it customers, products, suppliers or business events, will require a data cleansing exercise as a prerequisite.

For years data has simply been "mapped then moved", only to cause significant problems in the new system because of quality levels unable to support enhanced functionality.

Bryn Davies is regional manager for Sybase SA`s Cape Town office.

It is by now well known that business intelligence (BI) is a top priority for most organisations, and it has become equally well known that dirty data leads to dirty decisions - the need for higher quality has been driven by frequent boardroom frustrations of inaccurate, ambiguous and generally mistrusted information. But because BI relies not only on the data itself but also on its meaning and presentation, it has also become necessary for organisations to re-assess how they model, store and present their information.

The subject of metadata management therefore, for years pushed to the back of the queue, is finally nearing the entrance, as companies start to formally define and align the semantics of the data they maintain.

Once again data quality enters via the side door, as companies start out to improve the accuracy of their reporting and BI systems, and wind up tackling this deeper, more fundamental issue.

Another more recent trend has been to profile and cleanse data within legacy systems, before moving it to, for example, a new enterprise resource planning (ERP) package.

For years data has simply been "mapped then moved", only to cause significant problems in the new system because of quality levels unable to support enhanced functionality. This is a bit like moving house but taking all the junk too! Better to thoroughly clean, consolidate and enhance data before trying to shoe-horn it into its new home, and so data profiling, standardisation and matching has become common practice as sub-projects of ERP implementations.

Finally, seeing the carrot, some more mature organisations are tackling data quality in its own right, without a parent project or another driver such as compliance.

In the end, all will recognise that data is the foundation of our businesses, and that high quality data will lead to lower costs, loyal customers, increased opportunities and better business overall.

So whether it`s an MDM project resulting in a data quality initiative, or a pure data quality drive leading to sound data governance, data quality is the common thread that binds these all together. Whether it is perceived as a carrot or a stick, all paths eventually lead to an imperative to do something about improving the quality of the only re-useable resource any organisation possesses - its data.