Masters of data

Two years ago, the business intelligence (BI) market was, if not full, then certainly bristling with a number of different solution providers. Today, there are far fewer. Oracle bought Hyperion, IBM acquired Cognos, and SAP forked out for Business Objects, reducing the number of vendors to a few large players and a handful of specialists.

The specialists are happy with the uncertainty and the inevitable lag in R&D and product cycles that happen when one large competitor swallows another, but the big players are hopeful they're one step closer to offering an end-to-end solution.

Fewer choices certainly make BI easier to sell. Marc Scheepbouwer, MD of Intellient, says although BI sales are now somewhat easier upfront, they also tend to be far more about strategy alignment than finding the right product for the right business need.

"Acquisitions and consolidation in the market has meant there are fewer political hurdles within organisations. Customers are having to consolidate their BI and vendor strategies," he says.

"It's become more of a platform sell than a point solution [sell]. Customers have SAP or Oracle or Microsoft or someone else's strategy. Either we're going to bump heads with our competitors in the market or we're going to have to come up with a hybrid solution. The consolidation of the BI market has made it a lot less complex and a lot easier to move faster with customers. If they're SAP-aligned, then you need to be aligned to that architecture.

"Someone made a point to me the other day that the tools are advancing, but sometimes the customers aren't. There are a lot of issues around the use of internal information. It's a protected asset in companies. If someone steals money from the company, they get fired. If they withhold information, they get promoted. That's a shift we're all part of - that information is being kept close. Eventually that information could become a product that gives you a competitive advantage. Information as an asset is very powerful in the hands of many," he notes.

Simon Carpenter, director of strategic initiatives at SAP, agrees there's a shift in the power of information and says SAP's acquisition of Business Objects is an admission that it needs to look wider than it already does.

"The fact is that we've gone through this wave of putting in ERP, CRM and automating business processes. Through it all, we've created huge amounts of data that has been locked down inside these large assets. The next wave is about making that information more democratic and more widely available, internally as well as outside our own organisation to our distribution partners and channel. We have to make decisions quickly and that means getting information to whoever needs it on whatever device they have."

John Olsson, sales and marketing director of Ability Solutions, says the new landscape is to be expected. "It's a reflection of the maturity of the market. The days are long gone where someone bought an ERP solution and later decided they really needed to think about putting in some BI to make sense of it. Acquisitions have come about because the major vendors want to be able to deliver an end-to-end solution."

People think in terms of relationships and associations rather than hierarchies.

Keith Jones, MD, Harvey Jones

Despite greater maturity and some easier choices for large customers, some issues simply won't go away. The quality of the underlying data remains critical which, in some sectors such as government, has been accurately described as "filthy and disorganised".

"Data quality is absolutely crucial for quality BI solutions," says Merlin Knott, sales director at Microstrategy.

"After all, the end result of BI is quality information and one version of the truth. The problem is that big corporates very often operate as federations and data standards vary. It becomes a nightmare to get quality data out of them. Moreover, the data hasn't been treated with the respect it needs. If data as an asset is not treated as such at a strategic level and filtered down as such, then it's going to fail at an operational level."

Paul Morgan, MD of Asyst Intelligence, points out that the main challenge is how to get data to be treated as it should at an operational level.

"A data capturer is only interested in a delivery address because that's what his commission depends on. E-mails, postal addresses and phone numbers may not be relevant to what he's doing right at that moment. So, it's all very well to have performance management systems that measure how good your data capturers are, but it's a very difficult problem to incentivise people to capture information relevant to a marketing campaign."

Jeff Solomon, corporate performance manager at i5 Group, says data management is really a journey and not a destination. "You need to push it much higher up the balanced scorecard than it is currently. People need to be accountable for their piece of the quality of the data. There are two real issues to focus on: one is how to incentivise operations; those clerks doing the job need to be incentivised by quality. When that's done, the quality does jump. The other thing is that quality means different things in different areas to different people. Once you've defined it properly, then you're in a position to go ahead."

Microstrategy's answer in one case was to use BI tools to drive better data quality. "A large customer of ours had a very federated environment with lots of strategic acquisitions," says Knott. "They therefore needed to focus on data management as a core strategy. The way they did that was to use BI to drive it. It became both their change management tool and their process management tool. Managers were held accountable based on the information they brought through the BI system. They couldn't bring spreadsheets along to their meetings and wing it."

Davide Hanan, MD of QlikView SA, agrees that BI tools can be part of the solution to bad data quality. "We find that the market wants to put data quality as a precursor to BI," he says. "But we feel very strongly that BI is part of the solution. BI is what makes your data visible and it can assist in highlighting problems that may be there. At least if your data is rubbish, a BI tool will highlight that and you will be able to act more quickly. BI is part of the process that can accelerate that."

Data quality is absolutely crucial for quality BI solutions.

Merlin Knott, sales director, Microstrategy

Master data management (MDM) is one discipline that, when properly applied, can give better BI results. Instead of multiple copies of your information sitting at a services organisation such as a bank, MDM says a single, pristine and comprehensive record should be at the centre of the system.

Ryan Jamieson, director of IS Partners, says MDM is necessarily broader than BI. "MDM sits outside the realm of BI, which feeds off MDM, but so do other operational systems. It's an expression of business: how is the data structured? What are the attributes that enhance the data and how can we use that data? It goes beyond just the analysis into every single operational system. The key challenges are: do you throw all the data in there or just the common data? And how do you identify the 'golden record'?"

Leonard Browne, principal consultant at SAS Institute, notes that the concept of the "single version of the truth" has been around as long as data warehousing. "What BI brings is the concept of identifying the master record," he says. "The trick is to get all the operational areas in their silos to conform to it. To do that, you need to make sure that any new software you buy doesn't try and identify a customer within its own application. There needs to be an interface to the existing record."

The main challenges to MDM are not technical ones. SAP's Carpenter says IT shouldn't be lumped with the implementation. "As long as MDM remains the province of IT, business doesn't give a toss. Someone has got to take stewardship. We owe it to ourselves to get this message across to business that this is a critical asset. Unless you have a clear view of who your customer is, what they are buying and how they are buying, how do you drive up-sell and cross-sell in something like banking products? One of the banks we're working with is putting a lot of effort into getting a single, consistent view of its customers across all the products it has bought."

Hubert Wentzel, divisional director of EOH Consulting, agrees, especially about the banking environment's unique problems with data management. "Whether this is BI, MDM or whether it's driven group- or enterprise-wide, it has to be driven by some kind of strategy," he says. "There has to be some kind of consistency. The problem that all the banks are facing is that they treat all their product systems as independent applications and no one ever talks to anyone else and everyone has their own version of the truth. There's no incentive for anyone to change that. There's been some success moving capturing clerks out of their roles temporarily and into others to show them how important their jobs really are and what kind of knock-on effects their mistakes can have."

Data management is also complicated by the fact that reams of unstructured data finds its way onto many companies' systems every day. To some, like QlikView's Hanan, this isn't so much a problem as a pointer to the future of BI itself.

"Structured data is something we've created," he says. "It starts unstructured, but we impose structure because that's the only way we can analyse it. You start from a simple transaction and the result of that is a single transaction in the ERP system, and additionally, into a data warehouse. The tools we have now require some structure. But as the tools become more intelligent, you'll require less structure. We need to see the flexibility of the tools coming down to a lower level. Our Outlook inboxes contain a wealth of information, for example. The challenge is for tools to be able to extract that information without imposing additional structure."

Acquisitions mean there are now fewer political hurdles to overcome.

Marc Scheepbouwer, MD, Intellient

One of the ways that has received some high-profile attention in the last 18 months is search, as popularised by Google's enterprise appliance tools. Knott says there could be plenty of mileage here.

"A lot of vendors are working towards a scenario in which you can type: 'Give me all the sales reports for the western region' into a search browser and it will bring up the reports for you. You're starting to get an interface that is more intuitive rather than being forced to use what a BI system says you should. BI is making a decision using the available information at the right time, so if the systems are more aligned to how we think, it's going to be far better," he says.

Keith Jones, MD of Harvey Jones, agrees. "People think in terms of relationships and associations rather than hierarchies, even though we structure data into hierarchies. For example, which products are we selling to which customers in Gauteng and who else is buying those products? There should be no difference between structured and unstructured information."

Ironically, it can be easier for users of a BI system to get information about the marketplace and competitors using Google or Reuters than it is to get specific information about their own sales figures for a region, but EOH's Wentzel says this is simplistic.

"We tend to assume that whatever we get with Google is fact, but we don't trust our own information. We have to check ourselves to make sure it's 100% accurate. Sometimes, we don't always need it to be 100% accurate to make a good business decision. Is 80% accurate good enough to make a good business decision fast or do we have to wait until it's 99% accurate?"

That depends on the industry and the application. Says Knott: "When you're driving your car and want to know whether you're going to make it to the next petrol station, then you don't need to take a pipette out and measure the millilitres. You can just take a look at the gauge. However, if you're going into a customer like a telco where they're starting to derive quantities like customer lifetime value, then accuracy and detail is that much more important."

Faster and more intelligent tools will, no doubt, be welcomed by BI users. But for now, has the end of the acquisition season been good for the people who actually have to buy BI systems: the customers?

Charles Teversham, director at Alchemex, says yes. "Customers do benefit, for instance, from the acquisition of a BI company by an ERP vendor. It gives the combined company the ability to offer enhanced solutions for both customer bases. You can deploy it far quicker than if it was a generic offering."

SAP's Carpenter says for SAP and Oracle and others, mergers and acquisitions have naturally been driven by the companies in question needing to grow. "There's no getting away from that. We believe if we get it right, then we're going to be able to reduce a lot of cost for our existing customers too."