In today’s corporate landscape, there is a progressive move to cloud-based data platforms and data lakes. This migration carries with it several pros and cons, both for the organisation implementing the move, as well as for the data teams on the ground who will need to implement it.
One of the benefits of moving processes into a cloud-based platform or data lake is the scalability that is offered.
It is also a way to ensure all data housed in the organisation is in one place and accessible to all – potentially leveraging processes between different teams to ensure the same approaches and sources are used, thereby creating one version of the truth throughout the organisation.
Any process pushed up to the cloud which has not gone through at least one governance gate could result in unnecessary, costly cloud computing increases.
Another benefit is assisting to avoid unnecessary duplication of effort and data between teams in the organisation – with the knock-on being the reduction of data storage requirements as duplication is eventually stopped.
In line with the nature of these cloud platforms, there is an increased amount of governance required to ensure the processes and data sources housed on the platform are functional, relevant and appropriately managed.
Any process pushed up to the cloud which has not gone through at least one governance gate could result in unnecessary, costly cloud computing increases.
Any migration of on-premises data systems to the cloud is a huge undertaking which impacts everyone in the organisation involved with data. There are a number of tips which could assist the organisation and the teams on the ground to streamline the process and reduce frustration and blockages.
Tips for organisations
Communication and governance: For organisations, the main tips relate to information and guidance provision. At a group level, there must be clear direction and communication in terms of timelines for the migration, as well as what the change means for the data workers in the organisation.
All governance changes also need to be provided in a central location and in a simple and understandable format. This information should clearly indicate the governance gates which will need to be passed in the process, the specific individuals responsible for each gate and what is expected to be provided from the teams for each step.
Should the data teams require additional assistance or input, say from an architect, to highlight how the processes or data will slot into the bigger picture, the contact details of a dedicated person who can assist with this should be given.
Documentation and accessibility: Any changes in terms of tools available or coding practices required on the new platforms need to be clearly documented for the data teams and made available prominently in the organisation.
The process to access the data in the new environments also needs to be streamlined and simplified for data teams. Ideally, the organisation should provide a like for like access management approach, where if the individual or team had access to data or process on-premises then that should be mirrored automatically on the new platform.
The connection method to the new platform should also be made as accessible and simple as possible to promote use and exploration of the new platforms throughout the process. Making connecting to the new platform too onerous will mean that teams cling to the on-premises sources as they are easier to access and use.
Tips for data teams
For the data teams that need to move their in-team procedures and data builds to the new data structures, there is a bit more work to be done.
Planning and strategising: Firstly, take stock. Understand what the portfolio of work looks like, how many processes there are, which sources are used, what history has been created that might need to be considered as a data exception and pushed up to the platform directly, how the processes tie together and whether the data sources needed are already available on the new platform.
Declutter the data: As part of this, do some house cleaning and ensure you retain only those processes and builds which are still relevant and used. Sunset any processes which are no longer contributing or have become outdated. Archive any data artifacts or features which will not be carried forward.
Stage the migration: Be methodical in transforming code to new standards and take the migration one process at a time. Use the new governance structures to your benefit to ensure your processes are robust and optimised, and also to evaluate if there is anything already on the platform that you could leverage to improve or expand processes as part of the migration.
Any migration to cloud-based platforms will cause discomfort and confusion within the organisation. Appropriately managing information provision and ensuring there is a strong guidance base in place will assist the organisation and the data teams to navigate the changes more smoothly.
The more support offered, and the more streamlined the access controls are made, the less negative impact there will be on the users.
Ultimately, the migration to a cloud-based data platform can unlock exciting opportunities for the organisation and its data teams. However, the management of the migration will define the attitude towards the new structures within the organisation, and as such, is the most crucial aspect of any migration.
Share