AWS DevOps Competency Case Study: Cloud Modernization

Our client engaged with Levio’s Cloud & DevOps services to modernize their IT infrastructure to deliver greater value to the business. Their business stakeholders were asking for access to meaningful data to enable them to make data driven decisions in real time, as opposed to waiting for overnight batch ETL processing to complete. Their existing batch ETL processing jobs ran nightly, were error prone, and required manual monitoring and intervention well past midnight to address errors to have data available for the next business day. Further to their desire to provide better access to data, the client wanted to modernize their IT infrastructure to provide self-service capabilities and move towards a platform ops model and leverage AWS Cloud managed services to innovate faster.

The overall technical objectives were to re-architect legacy applications to strategically leverage cloud services, to reduce operational burden, and improve developer experience. The client wanted to enable innovation to deliver increased business value with greater velocity and shift left security and governance by adopting a culture of collaboration and ‘everything as code.’

To kick-off, Levio performed an assessment of their DevOps capabilities, application tech stack, and data. From the assessments, Levio provided a review of the current state and delivered a roadmap of work that would allow achievement of their objectives. This work included a prioritized list of DevOps capability improvements. The top DevOps capabilities needing improvement were Infrastructure as Code (IaC), automated testing and test data, and continuous delivery. From the assessments and working with the client, we selected the scope for an initial AWS MVP / migration project.  

The MVP project started by introducing IaC practices via best-practice workshops on Github and Terraform. We also evaluated their AWS Landing Zone and delivered a custom workshop on AWS Landing Zone best practices and updated as required. From here, Levio built out an IaC pipeline using Terraform scripts in Github and Terraform Cloud.

From the application perspective, Levio modernized the client’s applications and containerized them into Docker containers. For container images, we pipelines in their existing CI tool Azure DevOps to build base container images, version and push images to AWS ECR. Once the applications were modernized into containers, build image pipelines then pushed the images to AWS ECR.

Levio extended existing Azure DevOps CI build pipelines to build application containers, execute unit, functional and integration test suites. Prior to this project, all testing was manually. However, we added test orchestration and automation to achieve greater delivery and continuous deployments. To meet velocity and quality objectives, we created automated test suites for application functional and integration tests using Cypress, Jest (Frontend) and XUnit (API).

Levio then redesigned existing CD release pipelines to AWS environments (Dev, QA, UAT, PROD), and updated the artifact promotion pipeline to automate release to UAT and PROD environments. As change management governance was in place to promote to UAT and PROD environments, we needed to integrate into an existing ITSM (ServiceNow) tool to comply with existing SOX standards and move to automated releases wherever possible.

To begin, Levio implemented an Infrastructure as Code (IaC) practice and pipelines using Terraform to provision all cloud infrastructure and services. All infrastructure was defined as code using HashiCorp Terraform. Furthermore, IaC pipelines written in Terraform Cloud were then used to deploy the infrastructure to all four environments (Dev, UAT, Cert, Prod). Manual provisioning of infrastructure was not allowed. As such, any change to infrastructure code has review and approval pipelines.

CI/CD pipelines in GitLab on AWS orchestrate the build and testing stages of the 12 microservices built into containers. Build artifacts are stored in AWS ECR after Security scans (Fortify, Sonatype and WebInspect) run on the build. Once a build passes, AWS CodePipeline is triggered to deploy to AWS ECS Fargate through AWS CodeDeploy.  A Blue/Green deployment strategy is used in every deployment. Once the microservice is deployed, it is opened to test traffic. Using AWS Lambdas, the AWS CodeDeploy Lifecycle hooks verify the microservice is in steady state before the route is switched to the new target group. AWS CodePipeline deployments are done in this manner into four environments (Dev, UAT, Cert, PROD) and in two regions in parallel for active-active redundant configuration. This supports the solutions’ high resilience requirements for Disaster Recovery (DR) multi-region.

In addition to the IaC and CI/CD pipelines, a configuration management lifecycle was implemented using GitLab, SOPS, and AWS Parameter store.

Integrations were required from AWS to on-prem data center for data encryption (Voltage), security (Sonatype, WebInspect, Wiz.io) and logging (Splunk).  These communications were also driven by the CI/CD Pipelines via an AWS Transit Gateway and AWS Direct Connect Router connecting the AWS Cloud to the customer’s Data Center.

Change management was required and implemented for any changes to upper environments (Cert, PROD) and was implemented using AWS CodePipeline. The change request form for deployments into upper environments was implemented in the CD pipeline. As such, promotions from lower environments (Dev and UAT) to upper environments (Cert and Prod) are controlled via manual confirmation through an approval gate in AWS CodePipeline.

Once in production, monitoring of the running microservices was done using existing monitoring tool stack, with Dynatrace and Splunk. Alerting on any error were configured to comply with Customer requirements. Wiz.io is used to detect any vulnerabilities on the deployed infrastructure. Levio also provided a postproduction team to support, maintain, and enhance applications and pipelines, including SREs, Technical Support Engineers and Cloud Developers.

AWS Public Cloud was selected as the cloud provider for this project. As such, the following AWS Services were used to achieve a Cloud-native, resilient, secure and easy to operate solution:

• AWS used for application infrastructure run time
  • Container registry ECR
  • Container based run time ECS
• API Gateway used to manage APIs
• AWS Cognito to manage authentication and authorization
• AWS CloudWatch
• AWS CloudTrail
• Load balancer - Application and Network
• Secrets Manager
• System Manager - Parameter Store
• S3
• AWS Route53

• Azure DevOps
• Azure Pipelines
• Azure Repos
• GitHub
• HashiCorp Terraform
• HashiCorp Vault
• XUnit
• Cypress
• Docker
• Localstack
• MongoDB Atlas

Levio implemented an Infrastructure as Code (IAC) practice, from source code repository in Github to Terrafrom modules, registry and pipeline for updates.  This achieved self service via fully automated infrastructure provisioning in AWS Cloud. With everything defined as code and versioned, infrastructure changes are auditable and controlled, with rigorous pipeline in place.

The result was greater velocity and decreased time to market with automated infrastructure provisioning which allowed for SOX controls to be maintained, thus reducing manual burden of Change Approval Boards. In addition, Levio standardized the creation and maintenance of container build images which further increased velocity and security. Using ECS Fargate for runtime compute, costs associated with managing VMs and operating systems decreased and freed the I&O team to move to more strategic tasks. The usage of ECS Fargate allowed developers to make changes, test changes, and deploy in isolation from other parts of the code, which greater improves overall velocity. 

Test automation and orchestration as part of the CI/CD pipelines was also introduced which greatly improved overall velocity. Increased quality with test automation and orchestration in CI/CD pipelines: Fully automated functional and integration test suites validating code prior to merging and deploying.

Increased security posture was also achieved as the applications were modernized to use centralized authentication and authorization in AWS Cognito and AWS Secrets Manager.

The project was delivered on time and on budget which resulted in follow up project for an additional migration.

For automation of testing, code coverage greatly increased from 0% to 100% for the applications that were modernized. End to End deployment lead time from new functionality being available for customers was decreased.

Further metrics will be gathered once the client implements metrics gathering.