enGenome

The Challenge

• Replace the existing monolithic AWS infrastructure with a more agile, high available, and scalable one, capable of managing all the conceivable workloads.
• Enforce data protection and backup policies in order to keep all the sensitive data safe from corruption and unintended modification.
• Leverage the AWS services in order to make the HPC flows resilient to the failure of one or more steps.

The Solution

• A stateless architecture based on multiple EC2 instances automatically configured using AWS CloudFormation and AWS Systems Manager. The HPC grid engine cluster was made scalable using an advanced configuration of AWS Auto Scaling, leveraging auto scaling lifecycle hooks, custom AWS Cloudwatch metrics, AWS Lambda Functions, and AWS Systems Manager.
• Implementation of a data lake using Amazon S3 as the data storage in order to take advantage of its data protection and replication features. Moreover, an automatic backup and DR system was implemented in order to save data also in different AWS regions.
• Implementation of an advanced HPC workflow using Amazon SQS queues, AWS Lambda Functions and SSM to deliver jobs to the HPC cluster.

The Benefits

Scalability

Advanced auto scaling rules made it possible to launch and configure Spot EC2 instances fleets to support any workload, significantly improving genomics analysis processing speed.

Efficiency

Thanks to the switch from a monolithic, stateful architecture to a stateless infrastructure based on on-demand and spot EC2 instances, a managed RDS database, and S3 for scalable data storage, eVai could benefit from great performance improvement, doubling jobs completed per day. 

Data and security

enGenome took advantage of Amazon S3 durability, availability and redundancy. Moreover it also heavily leveraged Amazon S3 features like Lifecycle Hook to move older data to a different class of storage (Amazon Glacier) and File Versioning to keep track of all the data changes. An automatic backup and DR system was also implemented in order to save data in different AWS regions, too.

About beSharp and AWS

beSharp has worked with enGenome to develop a custom, highly scalable and highly available platform for the analysis of large amounts of bioinformatics and genomic data in line with DevOps best practices and in full compliance with the AWS Well-Architected Framework. beSharp’s Cloud Experts supported the enGenome DevOps team through a training-on-the-job approach to help technicians to develop skills within the company and to provide them with all the tools they need to act in total autonomy, taking full advantage of the new solution.