Traditionally, management information systems, business intelligence and analytical projects have been planned mainly with business data in mind, focusing on the transaction or process that occurred.
But organisations are now looking to a far richer source of information – the metadata behind the business data – which complements and enhances all data, providing more detail and underpinning veracity and governance.
Metadata is usually generated from systems and applications automatically as data is created or generated from business/technical processes. As a complete record of what, where, when and how a business process occurred, metadata seldom lies and is an invaluable source of information.
Even system and application designs or specifications are eventually digitalised. What this means is that the veracity of metadata in the main part is very high once it is generated, hence there is a huge reliance on metadata for sensing, policing, governance, monitoring/surveillance/tracking and forensic solutions.
A world of use cases
But metadata can be deployed for a great deal more than audit and forensics. It can enhance analytics across the board, and is already in use to support a myriad of applications today.
For example, in data discovery: the first step is to find out what’s out there. In order to understand the type of data, you require metadata to discover the boundaries, scope or areas where you could find this data, and to support categorisation of the data as big data, structured or unstructured data.
Another example would be a scholar using a cellphone to look for a bus route. They would input parameters to find the bus route, including the place of pickup and the destination. This is all reference data – which is classed as metadata – and based on those parameters, off the logic goes to find the bus route, line, time and cost of a bus trip.
Metadata enables everything in the digital world and organisations are realising the huge value of metadata and are starting to use it more.
In the world of social media analytics and sentiment analysis, the understanding of customer data has traditionally been built around the transactions customers make – what is bought, the amount, pricing, date and call centre engagements.
However, this understanding can be enriched by analysing the technical metadata behind that transaction – how many times such a transaction was made, on which platform, how long it took for the customer to make a decision from the time they went to the app to the time they submitted the transaction, and even whether customers moved to a competitor after viewing a page.
Metadata enables far better insights into the behaviour of an entity (customer, supplier, third-party, service provider, person on the street, scholar, etc) which will enable a provider/seller to plan and provide a better experience for those entities.
Metadata enables everything in the digital world and organisations are realising the huge value of metadata and are starting to use it more.
A worthwhile investment
Harnessing metadata can require some investment, however. Metadata as a data entity in its own right has a significant storage requirement, which could be significantly more than that of the actual business data.
A single record of a client transaction may need 20 fields, each of which has metadata around it, recording information such as where the record sits in the table, the server it is stored on, who may access the record, and widgets or logic to execute and store, expose and update the record.
Like traditional business systems’ processes, interaction between a single user using a mobile app and a service anywhere generates more metadata than the actual business or user (transactional, interaction, queried) data. This also holds true data generated on social media platforms.
The volumes of metadata behind internet of things (IOT) devices are even greater. According to IDC, there will be over 55.9 billion connected devices worldwide by 2025, with the amount of data generated by IOT devices expected to top 79.4 zettabytes.
These devices range from surveillance cameras and autonomous navigation devices, through to medical and industry terminals and household smart devices. If RFID tags are included in the tally, the number of IOT devices in use by 2025 is expected to top 150 billion.
Rightly so, metadata is classed as being Big Data, given the fact that metadata volume, velocity and variety is just as pervasive (if not more) than business, interactive and social media data.
Despite the daunting volumes of metadata in existence, we must conclude that outside of the normal analysis of data content, sensing and tracking, there is a huge reliance on metadata, and this is going to grow, adding to costs and changing storage needs.
Computing resource-wise, projects would traditionally budget only for the data storage space, data processes and user interfaces necessary to deliver information (the solution) to the end-users, forgetting that the “unseen” metadata that is also generated and required to be managed for the storage structures, processes and interfaces.
Usually, organisations’ IT units carry the burden and cost of the latter, especially in traditional large companies that can afford to have their own IT competency/platform. But in these modern times where third-party cloud/bureau/IT platforms are used or subscribed for, such costs (capital, operational, metadata) are factored in for the account of the client (organisation), actually making projects more expensive.
However, because the value of metadata management and consumption to achieve greater customer and supplier insights is unquestionable, a mindset change must occur, with organisations accepting these additional costs as a worthwhile investment.
At the same time, a mindset shift must occur within data sciences: instead of the traditional approach looking at business content only, my recommendation is that data sciences incorporate the discipline of meta analytics and the metadata tools to report and expose metadata behaviour.
Share