When the cloud first started being widely used, I was initially struck by such a sense of…freedom. It was like we were finally unshackled. Myself and my fellow consultants and client partners were finally free to unleash our creativity in the cloud; no longer bound by so many restrictions as we had been in the on-premises world.
Those restrictions were many and varied. The availability and cost of infrastructure and software were key restrictions. With both, there were often a blend of capex- and opex-related costs locked into contracts. This made it difficult to change the direction a company was taking.
The result of this was that data engineers and business intelligence (BI) developers were closely bound to product families, and never got to venture too far out of that product family. We would create data engineering solutions in one related data and ETL product family and build BI solutions in another.
In some cases, we were able to do the data engineering, data storage and even BI solution development all within one product family. This, of course, resulted in trade-offs being made. It was difficult to have access to the best software for each component of a solution, due to the need to maximise the investment in expensive platforms and product families.
Many of us got so excited that we could choose to virtually build our solutions in anything…that we quite literally did choose to build it in anything.
Even obtaining complementary software, that was not duplicated by existing software, was challenging, due to the need to secure funding and ensure there was budget for the maintenance of the complementary software.
Then one day, all that changed with the emergence of the cloud. In our excitement, like children at long last let out from behind their parent’s shadow, we revelled in our newfound freedom, and gloried in excess. Where previously we had been bound by strict contracts, by limited infrastructure and large capex budget requirements, now we were not.
This was, and still is, a good thing. However, I do believe that as an industry, we got a bit carried away by that freedom.
When we moved away from mainframe technology and development styles some decades ago, and moved into servers and more modern software development, there was an immediate disregard for the lessons learned in the mainframe world.
Stability and SLA adherence, downtime, production failures were tightly controlled by rigid SDLC processes. When the development industry changed and moved away from the mainframe, many of those practices and methodologies were discarded in favour or new ways of doing things. The result was a very real decrease in stability of production solutions.
In much the same way, I believe that when we embarked on our cloud journey, we were so focused on the benefits of the cloud that we lost sight of some of the benefits (even unintended ones) of being on-premises with limited software and infrastructure.
In the on-premises world, we had far less diversity in our solutions. The OS systems and database platforms our solutions were based on, the languages they were written in, were more consolidated, and less varied. This meant a greater concentration of infrastructure and software, that could be leveraged across a greater number of use cases.
Because of this, there was also a greater consolidation of similar skills, resulting in developers and analysts using far less widely diverse skills to build and maintain solutions. This meant the cost and effort in finding these skills was bearable.
What happened when we dived into the cloud though, I believe that many of us got so excited that we could choose to virtually build our solutions in anything…that we quite literally did choose to build it in anything.
We never looked to what the team or even individual to the left and right of us were doing − we just dived in and started building. Often, we did this using technology and software, for no other reason than that we could, or we wanted to.
Now, when we look at a modern company’s cloud real estate, we see a large, diverse set of cloud-based servers and services, an even wider base of development styles, languages and implementations, across an ever-increasing technology and software base.
This enormous diversity, if managed and targeted correctly, could be a powerful set of tools. But I believe it is not being used as skilfully as it could be. So far, it has been used a bit too wildly for my liking.
It is becoming far harder to secure and staff resources for a team, due to the ever-increasingly diverse set of skills that are required. These skills need support. You can’t just get a single person with a skill, as that person can go on leave, get sick, etc.
The list of core skills and competencies keeps increasing, which either increases the size of the team to undesirable levels, or it requires every team member hired to be a rare unicorn-type resource that is highly sought-after, and thus difficult to secure or retain.
In addition, management of the estate is becoming difficult. We have too many servers and services. We are not able to properly register them and manage them because they are constantly being created and destroyed on the fly. We don’t know who has what and we are getting less return on investment, because more often we are working in smaller communities doing their own thing.
This means that we are not sharing our resources efficiently and are paying high prices for services that we could be sharing and benefitting from economies of scale.
From a data perspective, I find myself saying more often: “Data doesn’t like to move!” And yet, in the cloud, we are moving data multiple times, across multiple technologies, using multiple services and languages.
The scope of this keeps growing, and so I am starting to tell my clients: slow down. Take a look around. Who else is doing this? What technology are they on? Can we reuse it? Can we store the data centrally, and let people connect to it directly? Why are four people sitting next to each other all using a different database platform?
Looking back on the early years of cloud exploration, I can’t help but laugh now at how naïve we were. And shake my head at all the mistakes that we made. What is so frustrating is that we had already learned these lessons before in the on-premises world, and despite this, we must once again learn those same lessons, in the cloud.
If we do not, I fear that the cloud, despite its vastness and limitless potential, will become just as limiting to us as the on-premises world was, even if for different reasons.
Share