The only constant in life is change. Digital transformation, driven by consumer demand, is forcing many companies today to accelerate change.
Typically, at some point, this change involves a migration of data from older legacy based IT systems to more functionally-rich and digitally enabled solution. "It's so easy to get caught up in the excitement of exploring new functionality, defining new processes and getting users engaged that your data can be forgotten," says Barry MacDougall, Service Delivery Executive at JMR Software.
He says: "Data migration is often viewed as a small IT-led requirement of the overall project and of little interest to business. Yet, so many projects fail due to a number of common causes. Getting it wrong can be costly in many ways. Gartner says 'Through 2019, more than 50% of data migration projects will exceed budget and/or result in some form of business disruption due to flawed execution'."
MacDougall offers the following five key lessons around data migration:
1. Agree your data migration strategy and start early
This may sound obvious, but setting out the strategy is often left too late. It's also critical to have business input early regarding the migrating of data. Ultimately, they own it and are creators and consumers of the data in your business. Data migration planning and development should be initiated at the start of the overall project and run as a parallel stream to application development or configuration.
2. Be disciplined in your approach to data migration
Data migrations essentially sound easy. After all, it's just extract, transform and load, right? On paper that may be correct, but if you're not extremely disciplined in your methodology, your project will most likely fail. Introduce discipline by doing the right thing at the right time, every time. Don't be tempted to skip steps. You need to agree upfront what constitutes success, establish this early on, but also set the benchmark high and agree on measurable criteria.
3. Establish what data you should migrate
If left unchallenged, the tendency will be for your business stakeholders to try and take everything. However, this not only increases risk and cost, but it's also most likely unnecessary. At the data mapping phase of your project, it's important to examine the strategic value of data critically, with one question in mind. What data do you need to execute your new business processes going forward?
4. What you can't see won't hurt you
Quite frequently, data visibility is low in most migration projects and samples of really good data are checked throughout the process. It's critical to have high visibility of all data in flight to ensure absolute confidence in successful data migration. You should also reconcile data at every step and store the reconciliation results for each load iteration to be used as part of the success criteria.
As part of the migration test process, run batch processes (jobs) against migrated data on the new target system. Do this in the migration test environment pre-functional testing and this is likely to highlight any data deficiencies not previously anticipated.
5. Effective loading of migrated data
In an ideal scenario, you should always try to load data using an API that will trigger the business logic. Even if it runs slower, this significantly mitigates the risks rather than trying to 'insert data' directly in to the database. Avoid direct loads into database at all cost, unless there's a very good reason. If you do need to do direct loads, then all the business logic will need to be replicated into the migration code. Certain business logic may also need to be switched off during the load. This should be documented and agreed during the mapping stage.
Develop a collaborative approach to data migration
MacDougall continues: "As mentioned earlier, 50% of migration projects will fail or overrun, leading to increased costs and disruption to your business. Sadly, the finger of blame points to the development or data team. In an outsourced scenario, vendors are often penalised with heavy penalty clauses for being late."
He advises that the ideal approach should not be geared only around punitive measures, but should rather be collaborative and the engagement should be structured around a success and reward mechanism.
"You need to create a culture early on that everyone is in it together. Learning from these shared lessons will help you to achieve an end result of a successful and accurate migration."
Share