Migrating very large-scale ecommerce platform to AWS
We recently migrated our whole platform to AWS. It was relatively long journey as not all apps were cloud ready. In some cases, it was as simple as moving VMs to AWS from on premise while in many cases it was more complex than that. Moving all the applications in organisation in one go to public cloud was not easy. Almost all applications were very critical for business so limited downtime was expected. There was interdependency of applications, APIs, data flows and integrations. Not everything was following microservices pattern. There was no appetite of data loss during migration. Here are few learnings from this migration exercise:
Distributed File system — If there is huge dependency on NFS (Network File System), as it was in our case, make sure that right option in AWS is opted. IOPS, throughput and total data size in NFS plays very important role. Amazon Elastic File System provides scalable/elastic file system. There are other options such as NetApp cloud volumes on AWS marketplace to meet storage and throughput needs. IOPS, throughput and data size have to be measured in source and compared with the options available on AWS. This needs to be tested with actual values running on source otherwise this could become bottleneck while migration.
Another important variable in file system is number of files, directory structure and its depth. If you have millions of files in complex directory structure, solution could be different from if you have limited number of huge files. Data synchronisation from source is also very important criteria as sync has to be real time with source and also complete in given timeframe. If full sync with source can’t happen within minutes during cutover, your option may not be right one.
In short, Amazon EFS or NetApp CVO or any other store solution may not fit all scenarios. You need to do serious evaluation based on your needs and also test before migration.
Database Migration — When you are migrating whole platform, data becomes most important part of migration. There are many options for data migration and AWS Data Migration Services (DMS) is one of the options provided by AWS. It supports all major database migrations such as Oracle, SQL servers, postgresql. It also supports homogeneous and heterogeneous migrations. In our case, it was like to like so it was homogeneous migration.
Data sync again has to be near real time so that all data from source can be in sync during cutover within minutes. Data sync depends on lots of factors such as on-premise infrastructure, network infrastructure and type of data. All major database solutions also provide their own way of replicating data in real time. e.g. data guard, active data guard and Goldengate in case of Oracle and SQL server replication in case of MS SQL
Data replication from on-premise to AWS also depends on amount of changes on the source side. Is data sync able to cope up with changes happening at the source? Can your Data migration solution deal with all types of objects in your database? Can it handle big tables with millions of records? What is guarantee of data integrity between source and AWS? There has to be mechanism of validating data sync and validation process has to be tested before actual cutover. Reverse replication from AWS to on-premise should also be kept in mind to plan for any rollback in case of failures.
If it is homogeneous migration, native replication solutions provided by database products may be better choice.
Applications — All applications may not be cloud ready. Try using features provided by AWS for autoscaling to achieve elasticity. Breaking monolith into small micro services should not be part of cloud migration but it should be planned well in advance or in parallel. Two step approach can be followed to expedite the migration when as-is migration can be done as first step and then re-architecture. Continuous integration and continuous delivery should also be planned well in advance or in parallel and should not be clubbed with migration.