Agile for all

In many organisations, business intelligence (BI) has become a curse, with long-running initiatives that do not seem to deliver any business value, often accused of flushing thousands of rands down a black hole.

Keeping information fresh is really necessary to adjust an organisation's actions to match market demands.

Martin Rennhackkamp is COO of PBT.

To keep up with changes in its environment, the business has frequent and rapid changes in information and usage requirements. This should be addressed in an orderly way, however, in a way that does not stifle creativity and enthusiasm. Additionally, it must be done fast enough not to stall the business' initiatives in any way.

An agile approach to BI is the way to deliver real, valued information to the business much faster. In organisations that have not embraced an agile approach, this has to be carefully engineered to gain the advantages such an approach holds, but still ensuring it fits in with a more rigid program management culture.

For the business, the BI competency must behave like a service centre - rapidly serving pressing business demands. However, internally it should operate with an agile approach to ensure the rapidly constructed components slot coherently into the correct places of the overall solution.

The most complex part of agile BI is, in fact, 'agile data warehousing', whereby parts of the data warehouse must be designed, modelled, implemented and populated in much quicker timeframes compared to traditional times, while still keeping to a conformed model as a blueprint.

Agile data warehousing is certainly an efficient method of enabling efficient reporting systems - ultimately allowing for the rapid changes in the data warehouse to keep up with the demands for information around customers, products, and financial information.

With this being said, industry experts highlight several main principles that the concept of agile data warehousing should address, as briefly outlined below.

Agile data warehousing without the backing of an agile-oriented team proves difficult. Therefore, the team must be selected very carefully, understand changes, and be able to cope with a continuous stream of new tasks and requirements.

Metadata-driven development tools are known to speed up the introduction of changes to an existing data warehouse, support business development and accelerate implementation of new agile data warehouses. As such, the metadata files should be accessible to other programs, so that a broader spectrum of tools is available for use.

Keeping information fresh is really necessary to adjust an organisation's actions to match market demands. With regards to agile data warehousing, frequent source system feeds assist with gaining maximum return on investment, which is key to any business model.

Keeping the above methodologies top of mind, I generally prefer having an 80% straw model in place before colouring in the details during each sprint. However, in order to do this, one needs a good understanding of the business and the data that represents its business processes. Just-in-time data modelling can get very tricky if the subject areas and where they integrate with each other are not fully understood.

In line with this, it is crucial to get the conformed dimensions right from the start. The term 'conformance' may have originated from dimensional modelling, but the concept of one single instance of each key entity occurrence on the correct level of detail is crucial, regardless of which model the data is represented on. With agile data warehousing, this concept proves even more important, because the various parts of the data warehouse have to be slotted around these key entities as the company builds and populates the detail during each sprint. With agile implementation cycles, a business has even less time to rework and repopulate a data model if the conformed entities are incorrect.

When applying an agile approach, there needs to be a lot more interaction with the business. Each component has to be reviewed and agreed upon with the business as it is in the development and deployment phase. This on-the-fly testing and roll-out approach keeps the business interested, involved and, above all, co-responsible for the outcome. For most technicians, these faster and more interactive cycles make the work a lot more interesting, challenging and, at the same time, exciting.

If a company can manage agile implementation sprints through a properly conformed data warehouse, the benefits should be quickly realised, not to mention keeping life inside the team much more exciting than coding one boring ETL load after the other, which will result in greater productivity.