Conquer data basics before tackling emerging trends

2023 is expected to see significant growth in a number of emerging data trends, including the use of machine learning (ML), artificial intelligence (AI), edge and composable data analytics, DataOps, in anticipation of powering it with quantum computing.

However, none of these technologies and models can be successfully deployed unless organisations attend to the most critical basics of managing and harnessing data / information: metadata management and data governance.

Globally, and in South Africa, we see a great deal of interest and growing adoption of these technologies and models.

ML and AI capabilities are maturing, as can be evidenced in many organisations around the world. AI / ML has found its place in many business applications and the online market.

Manual coding, scripting and modelling are fast being replaced by RPA tools and standard AI applications that enable process/workflow automation and data sciences, offering a variety of functionality, such as bots, multi-languages, efficiency monitoring and predictive/prescriptive analytics.

Hybrid cloud services and cloud computing usage is on the rise for agility, and to support the work-anywhere workforce. However, organisations are also realising the cloud is not a silver bullet.

Where data is housed depends on a variety of factors, such as data security / privacy, hosting and computing costs, organisational policy and size.

DataOps, a methodology using agile, iterative approaches to data lifecycle management, is enjoying increasing focus. But DataOps should not focus on speed and agility alone: it should rightly be bundled together with data stewardship, to ensure fluent data with the right governance and curatorship assured throughout the entire journey of data.

Every emerging capability, tool and solution – from AI and ML through to quantum computing − will be metadata-dependent.

Edge and composable data analytics (on the consumer “edge” side) is really only possible when there is data fluency; ie, a state or capability where data is automatically and seamlessly acquired, curated, processed and provisioned on time (or on demand) with the appropriate security / privacy, and assuming that consumers have the right analytical tools.

This means the data platform to effect all this back-end work has to:

• Be proactively resourced in terms of compute, bandwidth and storage scalability.

• Have all the processes integrated with all the necessary controls and interfacing instructions.

• Have all necessary roles assigned along the full lineage of the data journey to assure solid administration and strict oversight.

All of these components together require DataOps, and the right degree of process orchestration with simple and flexible configuration, change, administration and execution capabilities.

Quantum computing presents exciting possibilities for all computing and storage platforms. There are expectations of unprecedented data capacity thresholds and processing and analytical performance.

Quantum computing will take us into the realm of the next generation of computing, networking and storage, and it offers the promise of solving big data management and analytical constraints.

It is, however, still in its infancy and far from ready for normal commercialisation for businesses, but its future is bright and it is expected to drastically change the world.

Good governance and metadata management will be the foundations that allow organisations to tap into the opportunities presented by these emerging technologies and trends.

As the rate of uptake accelerates, these technologies and trends could overtake efforts to get governance right, so organisations must move quickly to set the necessary foundations in place.

Every emerging capability, tool and solution – from AI and ML through to quantum computing − will be metadata-dependent. The full data / information landscape is encased in metadata placeholders, so organisations need to get control of their metadata management in order to unleash the power of new technologies.

Tied to this, data governance is crucial within DataOps and for the successful deployment of AI and ML, enablement of fluent analytics and the management of data/information platforms and quantum computing.

If companies don’t have control of metadata and solid data governance in place, they won’t be able to map data components and elements to business requirements / use cases / processes, and to standards and compliance rules and regulations.

This has become increasingly important as organisations strive to overcome a superficial ‘check-box’ approach to compliance. Not being able to evidence implementation of standards and proving compliance is a key risk within organisations today.

In South Africa, we see growing requirements from auditors and regulators for organisations to ‘prove’ compliance by linking and mapping the data, to what happens on the floor, to a compliance rule.

For organisations, doing this mapping not only makes it easier to prove compliance; it also lends itself to efficiencies. If carried out from the metadata level, from the bottom up, it reveals gaps, overlaps and inefficiencies.

While emerging technologies promise new data paradigms, getting governance right has to be the first priority for organisations in 2023. Likewise, the new technologies and trends must incorporate data controls / governance and stewardship capabilities to remain secure, credible and compliant.