Five years on: Governance still biggest hurdle to BI

Almost half a decade ago, businesses realised that effective data governance was key to a solid business intelligence (BI) strategy. However, today, governance is still the biggest hurdle to BI.

KID director Mervyn Mooi noted in 2015 that BI projects built on ungoverned, unqualified data/information and undermined by shadow BI would deliver skewed and inaccurate information.

In a study done by Forbes in 2016, it warned that data governance required a healthy balance between consistency and flexibility.

In the years that followed, we saw the emergence of more automation, artificial intelligence (AI) and dedicated data stewards – ideally, to rectify this.

However, today, while organisations are moving at the speed of business and data is being created at a rate of 2.5 quintillion bytes per day, it is still not an easy task to ensure good, clean, quality data is available for accurate BI reporting.

With compliance being a requisite in almost every industry, companies face a massive whip should they not comply with certain reporting standards.

Now, as it was then, governance is more important than ever before. Having a solid governance framework in place is vital to ensure the data being used to make critical and non-critical decisions is as accurate as it can be.

How can a business make a decision if it has no trust in its data/information, or where the data is coming from?

In the world we live in, where mission-critical or even life-threatening decisions need to be made quickly, data/information needs to be available immediately − and that data needs to be trustworthy and correct.

With self-service reporting now available to almost everyone, being able to report information and present data accurately is critical. Having a strong governance framework in place ensures the data used can be trusted and that all the relevant controls, checks, standards and legalities (compliance) are adhered to.

Modern day governance tools enable organisations to map and link all data/information and business and technical process artefacts or assets (metadata) to each other, to the respective systems, role-players or stakeholders and also to compliance/regulatory rules and clauses as they relate to each other.

Using in-built AI or manual interrogation or discovery methods, this repository of mappings immediately enables true lineage, landscape oversight and proving of compliance.

How can a business make a decision if it has no trust in its data/information, or where the data is coming from?

It also surfaces opportunities for efficiencies; ie, overlaps in artefacts, manual lineage and gaps, change tracking or role actions.

The governance model is premised upon oversight and control over data management (including data quality, verification, security) processes, roles and compliance to external regulation or internal standards.

Governance guidelines are available from many sources on the web and specifically in tool vendor documentation.

To make governance easier, it is recommended that organisations make relevant such guidelines within a site-specific framework of relevant standards that are realistically applicable or pragmatic; ie, articulated.

Data governance sets a blueprint of controls for the proper management of data assets across the organisation.

Even with governance processes and tools in place, effective governance can be challenging because data changes all the time. Customers may move and organisations may change their reporting, branches or processes.

Therefore, linking and change tracking is crucial in ensuring data integrity and accurate current and historic reporting.

Matching and merging of historic data to ensure design and storage conventions are aligned and all data is accurate according to set rules and standards may require up to a year of careful design and architecture to integrate data from various departments and sources, in order to feed the BI system.

Another challenge is the existence of ‘rebel’ or shadow data systems, in which departments start working in silos, creating their own spreadsheets, duplicating data and processes, and not inputting all the data back into the central architecture. This obviously results in huge and unnecessary costs.

Even data as apparently straightforward as a customer’s ID number may be incorrect – with digits transposed, coded differently or missing. Data stewards are still required to carry out certain manual verifications to ensure the data is correct and remains so.

Stronger governance helps ensure not only consistent but also reliable and optimised results.

It all needs to come through the central architecture or common model regulated by framework standards.

In this way, the entire ecosystem can be governed effectively and data/information can be delivered efficiently and in a trustworthy manner, also making management easier and more cost-effective.