I must confess that I have been guilty of making a fundamental mistake when evaluating cloud data partners. My clients often express a desire to retain alignment with their existing database, ETL and analytic software. This software is inevitably associated with one of the main cloud platform vendors.
During these engagements, I made the mistake of assuming the best fit was to align with the base products that are provided by the cloud platform vendor. I did this, as we believed this would minimise the impact of the migration to the cloud, and because we somehow felt the cloud vendor would “prefer” if we stay with their own base products.
There was also a perception that this would reduce the complexity of the resulting solution, and its associated costs. Experience has taught me otherwise.
The main cloud platform vendors have significant strategic partnerships with vendors offering key data platform services. While these vendors might provide competing services to their own services, their core focus is still the cloud platform itself. This is resulting in some surprisingly honest and straightforward advice when it comes to product selection.
Sometimes, when trying to reduce complexity, we end up making our lives more complex and difficult.
Another aspect to consider is that no matter what companies might think and hope for, to get the most out of a cloud migration, they can’t really plan for a “lift and shift” style migration. Rather, they should reconsider the solution from the ground up.
Trying to “minimise the impact of cloud migration” is simply not viable. This will only constrain and limit thinking. I would rather encourage companies to re-imagine what their solution could look like in the cloud, with access to a wide variety of best of breed, fit for purpose tools that were designed and built with a cloud-first mentality.
My advice, and based on experience, is that organisations shouldn’t minimise the impact of the migration to the cloud, and instead embrace that change and consider how it can help achieve goals.
Sometimes, when trying to reduce complexity, we end up making our lives more complex and difficult. We buy into a new technology, but still try to implement it in the old ways of the past.
So again, instead of putting limitations on the scope of the technical offering in the cloud, rather keep a completely open mind for all technology choices. Companies might be surprised to find that while their initial ramp up might have seemed complex to initially implement, the resulting solution is now simpler and easier to operate.
When I expanded my horizons and started focusing on data service offerings beyond the scope of the cloud platform vendors’ core products, I was amazed at the depth and breadth of services available.
These services differ drastically to many of the traditional offerings that were migrated and modified to operate in the cloud. They have been designed from the ground up, to be cloud and big data first. Their architecture, data storage, processing and management are built on completely different paradigms to what we are used to.
A good example of all of this, is a data platform called Snowflake. It has been designed first and foremost to be a cloud-driven data platform. I was quite taken with its architecture and offering when I first investigated the platform.
Snowflake allows companies to store data centrally in databases. On top of this, there are multiple virtual warehouses. The database storage is where data is stored and managed, while the virtual warehouses are where data is processed and are effectively compute nodes.
It allows companies to scale data clusters and compute clusters separately, according to storage and processing needs.
Snowflake automatically manages many of the tasks the organisation might be used to doing itself. Things like partitioning, compression, encryption and query optimisation are handled automatically.
The database layer can be accessed by multiple virtual warehouses, concurrently, with no contention. The virtual warehouses provide support for relational, dimensional and data vault methodologies. This means data can be loaded and stored once and this data can be exposed multiple times into different virtual warehouses, using different data model design methodologies for different use cases, at the same time.
Furthermore, the data and compute clusters are priced separately. Each has multiple tiers of features, meaning organisations can customise data and compute separately, select the required features for each and pay accordingly.
This concept also extends into the security layer, where different levels of security features are available, once again allowing enterprises to customise this according to their own needs and pay accordingly.
It's not my intention to push customers away from the products of the main cloud platform vendors. I do believe, however, that there is more on offer, more available than what we are used to from our standard vendors.
It is a brave new world out there, and the people leading the charge are very often the newer, less established vendors.
Share