Using BI to drive BI

The Pareto principle is named after Italian economist, Vilfredo Pareto, who observed in 1906 that 80% of the land in Italy was owned by 20% of the population. Hence, it is more commonly known as the 80/20 rule, where for many events, 80% of the effects are a result of 20% of the causes.

Typical BI reporting often produces extremely large and “sparse” reports.

David Logan is principal consultant at PBT Group.

Using this principle in designing and building business intelligence (BI) systems can result in significantly improved performance, relevance, and correspondingly, business value of the product delivered. Two obvious areas that spring to mind are the extraction, transformation and loading (ETL) and presentation (reporting) components of BI.

Analysing the underlying information for 80/20 patterns (such as 80% of a company's transaction volumes come from 20% of its customers) during the BI design phase can create rapid return on time invested.

Given the trends of dramatically increasing data volumes that have to be processed, an example of using the Pareto principle would be to note that, typically, most of the transactional data processed daily are recent transactions, whereas a smaller volume of data can often be older data, which has only been processed recently.

The interesting thing here is that the smaller volume of older data often creates a disproportionate processing load on the ETL process, as it involves recalculations of amounts for days - which have already been assumed to be “complete”.

In some cases, the pattern is typical and repeated only for certain products sold by a business. Think of the example of mobile roaming information, where a customer makes calls while overseas, whereby the information has to flow from a foreign mobile provider to the local one - which can involve some delays. However, only a small proportion of total call volumes are roaming. By using BI to analyse the patterns of information being loaded, the ETL process could be designed to intelligently and dynamically “split” and process information according to the known business patterns, and apply more processor load to the intensive processing tasks and less load to the less intensive task. In this case, real BI has been used, ie, not just technical expertise, to produce the best ETL performance. In addition, by automating this data profiling into the process, the process itself will “adjust” itself to changing business behaviour.

Typical BI reporting often produces extremely large and “sparse” reports. By this, I mean that a typical Excel spreadsheet can contain hundreds of rows indicating sales by customer; however, 20% of the customers produce 80% of the volume, with the others being relatively low numbers.

Observing business customers at work with these reports often involves the user repeatedly filtering the information to focus on his 20% of customers. Using BI in advance can produce far superior results. By dynamically aggregating low volume customers into a single category, the report becomes much more relevant and consumable.

In addition, due to the lesser data volumes, the performance time of queries and the ability of the end-user to visualise key trends is significantly enhanced. Even the movement of customers in and out of the 20% provides immediate insight to the business user.

Taking the above into consideration, when companies fail to use BI when designing BI systems, they not only miss out on the huge benefits of doing so, but ironically, they fail to “eat their own cooking”. Careful analysis of most BI systems will result in opportunities to “use BI to drive BI”.