From sensors gathering climate information, to posts on social media sites, from purchase transaction records to cellular location data, and from digital media such as images and video to machine data recorded by factory and plant instruments, most organisations are drowning in data.
We're creating around 2.5 quintillion bytes of information every day, according to IBM. And as much as 90% of the data in the world right now was created in the past two years. Financial market data, server logs, medical imaging files, metrological data, and genomic research information - there's just no end to it.
Far from slowing down, the pace is accelerating as the types and sources of data organisations have access to multiply in number. That means the question for most enterprises is no longer how to gather and access data, but how to store, manage, analyse and put it to work in an efficient manner.
That's where big data technologies - tools for storing, managing and analysing huge data volumes, created at a high speed and in range of structured and unstructured formats - are starting to come to the fore. These technologies are meant to help organisations sift through the chaff of useless and duplicate data to find the insights that will enable them to build better businesses.
This trend is fuelled by technologies that are starting to hit a sweet spot in price/performance. Cloud architectures, cheaper commodity servers and storage subsystems, open source software, noSQL databases that free data from the constraints of traditional relational database structures, in-memory computing and powerful analytics tools are just some examples.
Data, data everywhere
Big data brings together a number of other trends in the market, says Mary Brady, principal consultant at HP South Africa. Mobility, social media, the processing power of the cloud, and continuous improvement in storage technologies are just examples of the megatrends enabling huge volumes of data to be captured and stored at rapid speeds.
Most South African companies are still just experimenting with big data technologies, looking to understand how big data should slot into their world views, say vendors interviewed for this article. But the same is true of organisations in the rest of the world.
We're a bit behind trends overseas, but local companies are waking to the realisation that we have the same opportunities as organisations in the rest of the world.
According to the Big Data @ Work Study, conducted by the IBM's Institute of Business Value and the Sa"id Business School at the University of Oxford, most enterprises are currently in the early stages of big data planning and development efforts.
Most are focused either on understanding the concepts (24%) or defining a roadmap related to big data (47%). However, 28% are implementing big data pilots, proofs of concepts (POCs) or have already implemented big data solutions at scale.
"Most local organisations are still catching up to international trends," says Lee Naik, managing executive at Accenture South Africa. "But some clients are asking how to leapfrog their journey into big data."
"We're a bit behind trends overseas, but local companies are waking to the realisation that we have the same opportunities as organisations in the rest of the world," agrees Willem van Aswagen, business intelligence lead at Avanade South Africa.
Against the backdrop of tough market conditions, enterprises are eager to use data to make better decisions.
Although South African companies have produced and gathered high volumes of data in varied forms for many years, they didn't have the tools they needed to drive the value from it in the past, says Johannes Kanis, Server and Tools business lead at Microsoft South Africa. Storage was expensive and analytics tools weren't powerful enough to unlock the value trapped in the data. But today's lower computing and storage costs, paired with better analytics engines, mean it's possible to tap insights from data in a cost-effective and scalable way, he adds.
Big data strategy
Vendors say the best place to start thinking about a big data strategy is to look at how to gain new insights from existing internal data, and then extend into new sources of data over time. Many organisations already have massive data sets they're not really leveraging to full potential, such as detailed point-of-sale transactions in retailers or call records among telcos.
Nicholas Bell, MD at BusinessIntelligent, says one impact of big data is that it has sparked new energy around data and analytics among South African organisations. Against the backdrop of tough market conditions, they're eager to use data to make better decisions, he adds. They need to take stock of their existing data and examine where big data can deliver a return on investment.
"Companies should not be overwhelmed or intimidated by the concept of big data, but rather see it as an opportunity to expand on their existing insights," says Van Aswagen. Before embarking on a big data project, companies should take stock of the information existing within their own environment and identify questions they cannot answer using only their internal data sources, he adds.
Once they've done so, they should identify external data sources as well as the technologies they need to capture big data and integrate it into their existing data platforms.
For many organisations, the greatest value associated with big data is in the area of customer analytics, says Ocea Garriock, technical professionals leader at IBM Software. By using big data to truly understand customer preferences and anticipate future behaviour, companies can personalise customer interactions and improve customer satisfaction.
Big data can also be a catalyst for operational business intelligence that enables companies to act on real-time data. Big data tools have a major role to play in operational areas such as real-time fraud detection, loyalty settlements in customer rewards programmes, or alerting customers when their telecoms bills look dangerously high, says Marc Scheepbouwer, MD of Zetta.
Realignment of power
Despite the opportunities hidden in big data, most organisations face major barriers as they try to get to its value. One of them lies in building the right combination of business, technical and analytical skills required to deliver on the promise of big data, says Garriock.
Another major challenge lies in the fact that data has traditionally being tightly coupled to applications and departmental business leaders, rather than being made available to the organisation as a whole, says Naik. Data needs to be treated as a shared enterprise asset that crosses department boundaries.
Most companies still run multiple disparate databases that don't talk to each other, says Gary de Menezes, regional director at Oracle South Africa. Even after years of consolidation and integration, most companies are not as far advanced in consolidating middleware, databases, and applications as they need to be to squeeze the maximum value from data mining and analytics, he adds.
That means organisations need to spend time getting their enterprise data management models right if they're to get real value from it. "The rate of innovation outstrips the ability of most South African companies to make use of the latest tools and technologies," De Menezes says. In other words, to get true value from big data, enterprises need to get basic data right first.
Share