I have always really enjoyed working in the data industry. By data, I mean data warehousing, big data, business intelligence and data analytics.
A well-run data implementation can have a fundamental impact on a business. It can turn a poorly-performing business into a profitable one. It can give a business that is stagnating a competitive-edge, making it once again relevant and exciting.
At the risk of sounding cheesy and clichéd, data-driven solutions can even change lives.
In some industries, a data-driven solution can have a direct, tangible improvement on the lives of every-day people. In the medical industry, data-driven solutions can improve the quality of patient care and the speed of diagnostics and treatment.
In the financial services industry, for example, these solutions can improve the speed at which claims are processed, improve the management of people’s savings and reduce the cost of policy administration.
In general, data-driven solutions can positively impact the quality of the customer experience, enabling long-lasting business-to-customer relationships that benefit both parties.
While these types of successes are sadly not everyday occurrences, when it does happen, it can be extremely rewarding, and it stays with you for a long time. I personally find that these successes re-energise me and motivate me to achieve still more.
From a technology perspective, we have never had it so good.
Given all of this, you might find it surprising for me to point out that these solutions, and the data industry in general, often have a bad reputation.
The perception of business representatives in many cases is that these solutions take a very long time to deliver, are difficult to evolve and respond to change, and from a technical perspective, are often of poor quality, not built to standard and difficult to maintain.
There is also the concern that these solutions are extremely expensive to implement. Of most concern to me, is that often the company feels these solutions do not address its core business challenges.
I encounter these issues regularly in my engagements as a data consultant. I experience the legacy of solution quality issues constantly, where my teams must often deal with the fallout, having to re-engineer these solutions.
I often deal with frustrated, jaded business representatives who have long ago lost faith in data-based solutions, and I need to try to re-energise these representatives. I need to coax them and encourage them to believe in the principles again, and give these solutions a second, third or even fourth chance.
I believe that sooner or later, something is going to give. Either business will completely lose faith in the industry, which will result in many not investing in these data solutions. They will go back to small, independent silos, point-to-point type data solutions. Or, business will continue the status quo, allocate a default budget to build the “obligatory” data solution, but have no faith in it, and return to implementing “shadow BI” solutions.
Honestly, I don’t know which of these options are worse.
In the hopes of starting a data revolution, it is my vision to propose some potentially radical concepts − a different way of thinking that can hopefully address some of these key challenges to business and to this industry that I am so passionate about.
At the heart of everything, is people. The people are where the opportunities are. However, the people are also where the problems lie.
The fact is, we have great software out there. All the key, flagship data platforms and tools fundamentally do the job. We simply can’t say that any of the leading software tools and products on the market right now are bad − they are all good, and one couldn’t really make a mistake choosing any of them.
From a technology perspective, we have never had it so good. Infrastructure costs keep going down, we have great quality infrastructure to choose from, and with infrastructure automation and the cloud, the ability to deliver solutions on technology has never been better. Cloud and the open-source community also mean there is access to cheaper software.
Given all of this, how can we still have problems with solution cost and quality? How can we still have solutions that don’t deliver the value that business needs? How can we still have solutions that are difficult to maintain, change and evolve?
Unfortunately, the answer is once again: the people. Fundamentally it comes down to how people bring their own inherent bias to platform, software, solution design and implementation choices. How they define business strategy, budget and priorities. How they implement architectural and governance oversight on solution delivery.
I would like to propose and advocate that we are data professionals, but as data professionals we do not have a professional body with real teeth and clout. We are data professionals; however, there is nothing enforcing our behaviour, our engagement with business, the nature of work that we do, or how we perform that work.
In my articles to follow over the next few months, I am going to unpack what I think is needed to address these concerns.
I freely admit there is no silver bullet out there. Implementing an enforceable set of professional standards with a business that is already wary and cautious of our industry, will not be an easy task.
However, in the true spirit of agile, I believe we simply need to take it one step at a time and see how far we get. As often is the case with agile, we might never get to our ideal state. Recognising where we need to be, and taking small and relevant steps in the right direction, will yield significant improvements to the current status quo.
Share