Welcome to the open source era

Research indicates open source is gaining momentum, and there are significant amounts of money coming from the investment community into open source vendors.

A study by Ventana Research [1] reveals midsize and large companies are increasingly evaluating and deploying open source business intelligence (BI) software.

Many have had positive experiences with the software and intend to expand their use of it. Although open source BI software is not as mature and feature-rich as equivalent software from commercial BI vendors, organisations find it acceptably functional for their intended uses.

However, during a recent BI conference I attended, some well-known IT and BI specialists were asked what they think of up-and-coming open source BI tools. Unfortunately, the event was sponsored by a non-open source BI vendor, so the replies were rather subdued, with a few main points being raised, among them:

1) If you want 'proper' support you need a 'proper' vendor.
2) Open source BI tools are not yet mature enough to really compete.

I don't want to make the 'paradigm' argument here, but what I wanted was to find out for myself, if one can get past the initial distrust of software that is being maintained by a 'community' of 'contributors', whether some of the BI offerings that are available in open source support or disprove the above notions aired by a panel of BI experts.

I decided to stick to data integration tools, even though there are numerous open source projects that also cover reporting, OLAP, dashboarding, meta data, data mining and other BI areas. I decided this for no other reason than it is, in my opinion, the most technical part of the 'backend' of any data warehouse or BI implementation.

I visited the community pages of some open source BI vendors, downloaded the data integration (DI) tool, and started experimenting. I soon discovered their 'forums', where experts were at hand to give advice and point one to the necessary online resources. I also registered on the community 'issue and enhancement' request system, and could not only log new enhancement requests here, but also view all the current work that was in progress, complete with specifications, test scenarios, etc.

As a technical person, I could not help but wonder why commercial BI vendors can't be this open with what they are busy with. I have also never experienced a commercial vendor with as lively and helpful online forums and support as I experienced here. I was pleasantly surprised!

In my techie heart, I felt this support was more 'proper' and real than most I have experienced from commercial vendors, but of course lacked the formality that large organisations feel comfortable with.

Having said that, I quickly learned the most popular of these open source BI tools do in fact also have formal support 'subscriptions' of various levels, as well as commercial versions of their software that boast a few extra features for the 'enterprise-level' user.

As a technical person, I could not help but wonder why commercial BI vendors can't be this open with what they are busy with.

Cor Winckler is technical director at PBT Group.

That brings me to the second point. Can these tools compete with mature commercial DI products?

Again I was pleasantly surprised. I compared two of the open source DI tools available, and in both, on opening the very portable GUI front-end, I found a plethora of functionality that one would expect from any traditional ETL or DI tool, and more. Since a community contributes to the functionality in an active way, the collective functionality that results contained surprisingly (or perhaps by now, not so surprisingly) innovative options.

The ability to extend the functionality easily is another inherent attribute of most of the open source tools on the market, not just these BI tools. In addition to this, the architecture of the tools was well documented and scalable.

If I had to build something on my own account, I would not hesitate at all to make use of these innovative and surprisingly powerful open source tools.

With decreased costs and increased functionality, the popularity of open source technology as a driver for lowering the cost of traditionally expensive BI implementations will certainly gain increased momentum in the market and could be the next big untapped opportunity.

For many IT organisations, open source products will be simply considered alongside full commercial approaches, and standard selection criteria will be applied to both.

As we move forward, it will be common practice for organisations to use a mixture of commercial and open source products. I welcomed myself to a new era.

[1] Ventana Research, open source BI usage, 2006

* Cor Winckler is technical director at PBT Group.