Data warehousing is dead?

The statement is often made in the industry that data warehousing is dead. What complete and utter unadulterated nonsense.

The need for build once, use many, or a single version of the truth, has nothing to do with technology. Historical trending and slowly changing dimensions were created out of business needs. Data needs to be collected, cleaned, combined, aggregated and prepared for business by IT. That is simply not going to change in the short- to medium-term.

The business intelligence (BI) market of the future will allow for all transactional data to be tagged and indexed at source, and then all BI will be done on the fly using in-memory engines. But until all sources can offer this kind of tagging, industry is stuck with the world it is in.

The engines that will be able to crawl through the transactional data and index it are coming, but not likely to offer proper commercial value for a few years. The definition of the term "data warehouse" may change, and what is included in the warehouse will certainly change, as will the toolsets it uses.

If it is virtual or “real”, is it still a data warehouse? I don't know, but I am not going to lose sleep over this. I am sure a techie will set up a blog around this between Idols auditions.

Business says it wants self-service, but this is not actually true; what it wants is to do the least work it has to in order to obtain the data it wants in the format it wants, and quickly.

Lazy? Efficient? Semantics. As the market matures, other technology sets will come in to play, like in-memory, which is actually quite old, but it seems its time has come.

Having more tools to do the job makes the job easier. Using the right tool for the right job just makes sense. The more tools available, the better and faster the job can be done.

So, how the job is done may change. BI appliances, data warehouses, data marts, in-memory, MOLAP and ROLAP are all tools that support the data gathering and preparation process. For larger customers going forward, they will all play a role as they all do different jobs. The type of businesses will push more towards one set than the other.

At the one end of the spectrum, administrative bureaucracies will rely on printed green stripy paper, 132 characters wide and two-year data warehouse projects that cost a fortune.

Adhocracies will rely on Excel and then move to in-memory. The “right” solution lies somewhere in the middle. There has to be somewhere where the data is gathered and prepared, then the right vehicle to share this data must be chosen.

The more tools available, the better and faster the job can be done.

Keith Jones is MD of Harvey Jones.

The good news about the big vendors is that most of them can do most of the job now, so laying the base platform should not be that restrictive. The solution should be able to service the immediate BI need and then be easily extended to integrate with or offer a full CPM (corporate performance management) solution.

If the one platform cannot easily extend to budgeting, forecasting, predictive analytics, data mining and all the things needed to be done in a growing BI environment, then it is a quick fix and is taking the user down a technology cul-de-sac. Users will be in the buy and bin cycle that makes vendors happy.

The market is quite gun-shy now about where to go. Everyone has been acquired and what seemed like a good strategy two years ago is not viable now.

So, how can a business move forward without going down the same road? The large players have entered the application market with their acquisitions, but choosing Microsoft, Oracle, IBM or SAP as a platform is actually not a bad option. They will be around, and from a platform point of view, will probably do quite a good job.

It is just what happens beyond this that people tend to lose their sense of logic. The annual BI Survey gives a resoundingly simple answer: “Buy the right tool for the job.” This means see it working and doing what the business wants. The single, one-stop-shop solution sales pitch is really nonsense. The products are not integrated or don't have the functionality to do the job. They may one day, but by the time they get there the market will be somewhere else.

I am always stunned by how many businesses are prepared to risk it all for a technology vision. If someone sold me a car and I said “Does it go?” and the response was “No, but it will do soon,” I would be quite annoyed. I guess the difference is all the cars go these days, but some need pushing or towing and others can be driven by the owner. Companies need to decide what they want.

* Keith Jones is MD of Harvey Jones.