Diary Of A Data Migration Nerd

17 February 2022
Job Vacancy Image

The Role of the Oracle Autonomous Database in the Data Migration Journey

By Dermot Murray, VP of Technology at Inoapps

One of the fastest growing practices at Inoapps is our Data Migration practice. As we have seen an acceleration of our clients moving to an Oracle Cloud Software as a Service (SaaS) platform, so too have we seen an ever-increasing demand for support and assistance for the challenging workstream of data migration. 

Of course, when it comes to data migration, organizations have many choices as to the approach that they take. Many of our customers take a very pragmatic approach and migrate the bare minimum, leaving their legacy data in an archived environment (with several using the Inoapps Legacy Data Archiving solution for Oracle E-Business Suite to assist with this), while simply moving their legacy data into a data warehouse or lake. Some of our clients have more significant compliance or governance requirements and need to migrate a much greater volume and complexity of data across multiple business processes, geographies and departments. Many have a seven-year liability for tax and compliance reporting, and many of our HCM and Payroll customers need to retain all historic data relating to past employes. 

Newsflash! Data Migration is Hard.
No matter how complex or simple the migration scope, the data migration workstream will always prove to be one of the most challenging aspects of any project, representing technical, functional, transformational and change management issues. Some of the most challenging and often overlooked aspects of the migration workstream are the security and data governance implications. Regardless of how tightly controlled the security profiles and access restrictions in your source and target systems are, it is inevitable that data "in transit" represents a potential weakness and security risk. This  risk could have devastating consequences for your business, as we have seen at TSB in the United Kingdom and more recently the Dallas Police Department in the United States.

Much of the security risk is inherent in the methodology used for the migration, and all too often we see organizations using outdated and insecure platforms to stage their data during the migration process. This can lead to: 

  • Data extract files being generated in unencrypted formats
  • Data files being sent over email, chat channels or other insecure communication platforms
  • The use of unsecured channels on SharePoint, Teams and other collaboration platforms
  • Data being held locally on physical devices, often without disk encryption
  • Files being left as “forgotten artifacts” after the project is completed, leaving a long tail of risk 

Compounding these problems are the unique challenges of data migration itself. Data volumes can be huge; far larger than CSV or Flat files are designed for. We recently engaged with a customer that needed to migrate 174 million journals to meet their business requirements. Try delivering that in an Excel CSV file! And of course, the mapping, transformation and verification of data requires capabilities far beyond that which can be delivered using traditional tools. 
What risks do these issues represent when the cost of a breach can be so high? 
Surely there must be a better way? 
Data associated with a migration exercise may only be “in transit” and may exist for only a brief period. But if the business needs to store sensitive data in a safe, secure and controlled platform, with governance, audit and control, then is there a better answer than to utilize Oracle database tools to achieve that requirement? And if the solution is to use an Oracle database, why not deploy that on the most secure, scalable and lowest cost of ownership option, Oracle Autonomous Cloud (ATP). 
Inoapps has now adopted a default model for all its data migration projects to utilize an Oracle ATP database to orchestrate every aspect of the data migration journey. All extracted data is loaded into the staging cloud database at initial extract, immediately allowing the data to be secured and locked down. The power of the Oracle database can then be used to orchestrate the transformation and mapping of the data, before final preparation of data for loading. Finally, the relevant FBDI and DAT files can be generated direct from the ATP database and orchestrated straight into the Oracle SaaS data platform for onward processing. 
Immediately this framework removes the feed for staging posts of data, eliminates the restrictions of data processing by using the power of the database, and enables complex mapping and transformation to take place within the database, delivered with governance and control. When the project is complete and all data is reconciled, the database platform can be deleted and purged, removing the long tail of legacy data files that could be exploited and abused at a future date. GDPR and data compliance rules can be asserted with confidence and authority, and the end-to-end life cycle of data can be proven for reconciliation and audit. 
Go easy on yourself and your team

Data quality issues, the need for cleansing, the mapping and transformation of the data, and the need to reconcile and validate the outcome are all challenging topics.  But why not utilize the very best tool for the job and leverage the Oracle Autonomous Cloud as the perfect platform for orchestrating your data migration exercise? 
If you need help with your data migration journey, contact the experts and Ask Inoapps 

Share this