Lessons learnt regarding data migration

Introduction

I just recently completed an ‘application data’ migration for a client and I’ve put down my notes regarding this. The data migration took data from a legacy system and placed it in Dynamics 365. The data migration was performed using Dynamics 365’s ‘Data Management’ component, with the data to be migrated, stored in spreadsheets

Note: This was a relatively small migration. For a larger migration, I would have considered setting up staging tables rather than using spreadsheets. These tables would mirror the structure of the Dynamics 365 entities. The data would be placed in the staging tables first and then onto Dynamics 365

My Notes

Creating the spreadsheets

To create the spreadsheets, existing data was exported from Dynamics 365. The data was then cleared and the columns copied to the spreadsheets shown in the next section

Populating the spreadsheets & linking entities

The newly created spreadsheets were then populated with the legacy data (after performing the steps listed in https://dustinminer.com/2021/03/25/preparing-data-for-migration-into-dynamics-365).

Note: To link entities such as Address and Contact, reference column was added to the spreadsheet i.e. RefAddressId

The Contact spreadsheet could then lookup the Address spreadsheet via the column ‘Primary Address Lookup’

The Address spreadsheet was imported first which meant that the Contacts spreadsheet could was then imported (and reference the addresses)

One Drive

The spreadsheets were placed on OneDrive because they could be shared securely and have their version history maintained

I needed to set the spreadsheet on One Drive to Region – Australia to resolve issues I was having with dates. That is, some date values where showing in Australian format whilst others were shown in US format.

Spreadsheets containing valid values

I needed to make sure that the Option set columns, Owner columns … contained values that were already defined in Dynamics 365. Otherwise the import of that particular spreadsheet would fail. It’s obvious but somewhat easy to forget

Spreadsheet tools

Tools within Microsoft Excel were used to manipulate the data within the spreadsheets. Functionality such as ‘Duplicate checking’ and VLookups.

Updating Business Process Flows (BPFs)

Migrated entities that contain a business process flow may need that flow updated after the migration. Referring to the screenshot below, the BPF will be placed in the ‘complete’ state.

To place all the relevant BPFs in the complete state, follow these steps

  • Use the XrmToolBox BPF plugin to perform a bulk move from the stage ‘Preparation’ to ‘Send For Approval’
  • In the ‘Form’ entity, perform a build set of the field ‘Approval Requested To’
  • For the BPF entity, update
    • ‘Completed On’ to current date
    • ‘Status Reason’ to finished
    • ‘Status’ to inactive

Importing the Spreadsheet data

A plugin or workflow may stop data being imported. An example of this could be trying to import invoices when the corresponding mandatory orders don’t yet exist. That is, when a new invoice is created, the plugin/workflow tries to retrieve the corresponding order to perform some logic