10 tips for starting out with AWS & DevOps

Building scalable cloud projects utilizing DevOps can seem daunting at first, especially when browsing through the plethora of 160+ available service options in AWS. Knowing which specific set of AWS services runs quickly and enables scalability takes a balance of patience, persistence, and willingness to learn. Managing commonly used cloud best-practices--such as server patching and timed serverless operations--can be effortlessly accomplished through DevOps. But it’s knowing which services to use and how to use them efficiently that will ultimately set one’s team and her career up for success.

When starting the DevOps journey in the Cloud, one will first need to learn how to plan, code, build, and test their application before focusing on new operational releases/deployments with detailed monitoring. Currently, there are three AWS services--Lambda functions, CloudFormation, and CloudWatch--that are extremely useful for building out the operational side of DevOps functionality and can be used to focus on the release and deployment aspects of the application lifecycle. My goal with this blog is to ensure that one can understand the differences between these AWS services with a high level overview of the services, a deep dive into using the services, and a focus into the operational (Ops) aspects of DevOps. This should give one the confidence to bring this knowledge back to their team; thus, becoming a DevOps and Cloud warrior!

After a couple years of working with AWS at Capital One, I had built and destroyed many applications via the AWS console and Command Line Interface (CLI), but I needed to completely automate this process by using common DevOps principles, such as plan, code, build, test, release, deploy, operate, and monitor to release software into a quick and repeatable process. I took charge of my career and devoted the next six months to solidifying DevOps principles throughout my team's AWS applications. Building DevOps automation into my existing projects empowered me to spend more time focusing on new high priority applications.

It has now been over three years since I started my DevOps journey and since then I have been able to deploy full stack automations using Jenkins with zero intervention from a human. Looking back, this experience has led me to create a list of 10 pieces of advice that every DevOps engineer using AWS should know.

One benefit of the Public Cloud--like AWS-- is that it allows engineers to take a more hands-off approach from daily server management so they can focus their time on building, testing, and deploying their applications. More information on the different Cloud Computing Deployment models can be found in AWS’s documentation.

However, one will still need to patch on immutable servers in the Public Cloud since applications are automatically scaled up and down using load balancers and auto-scaling groups. There are several solutions out there to account for this inconvenience, but the solution that has worked the best for me is Rehydration. Rehydration is the process of spinning up new servers with the latest patches already installed on them and decommissioning/destroying old servers that do not. It is possible to rehydrate tens, hundreds, and even thousands of servers at the same time in the Public Cloud.

Engineers can scale multiple rehydrations simultaneously using DevOps principles. DevOps allows for shorter dev cycles with continuous integration and a very short wait time to deploy code. It also provides solutions to common engineering challenges such as maintaining uptime in production environments, needing better tools to automate infrastructure maintenance, and monitoring a large set of servers. Amongst the plethora of benefits that DevOps enables, virtualizing environments, executing configured systems, and performing continuous monitoring are all important practices that gave me more time to dedicate to building new applications. 

There are many different services out there, but I choose to work with three services in particular: Lambda, CloudFormation and CloudWatch. Please note that there are several DevOps tools and AWS services available that offer full application lifecycle regulation, such as AWS CodePipeline, that can automate the build, test, and deploy phases of releases during code changes. For simplicity, I am just focusing on the three AWS services that I most frequently use.

1. Lambda allows engineers to execute code without server(s) by handling the computing resources. This service is commonly referred to as the “serverless computing platform” and many companies use it because it is much more cost effective to only pay for the resources that code uses.
2. CloudFormation allows for AWS infrastructure deployments to be repeatable. With CloudFormation, one can create template files, called CloudFormation Templates (CFTs), to build and destroy collections of resources together as a single unit (“stack”).
3. CloudWatch collects operational data like logs, metrics, and events and in return sounds alarms, visualizes logs, and triggers automated actions. CloudWatch can also trigger action events, such as Lambda functions, to run.

Lambdas functions can be built one of three ways:

1. Create code from scratch: One can create their own functions by using one of the many supported languages for Lambda: .NET, Go, Java, Node.js, Python, and Ruby.
2. Use a blueprint: There are already example blueprint configurations of Lambda functions with set languages to choose from. Blueprints help offer common Lambda solutions and can be filtered by runtime, language, or specific use case. There are also Blank Function examples to configure a Lambda function from scratch, but with a new added bonus of using a wizard to trigger and deploy the code. When I was first starting, this was helpful when I was nervous about jumping straight into writing Lambda code.
3. Browse the serverless app repository: Search and discover Lambda functions that are ready to be configured and deployed from AWS’s Lambda repository. The repository allows one to configure preset Lambda function values and seamlessly deploy and manage the functions from the AWS console.

Scenario: A basic application contains a load balancer such as ELB (Elastic Load Balancer) with an attached server EC2 (Elastic Cloud Computing) instance that contains a root volume with a secondary, larger, attached volume. In this example case, the secondary volume holds all of the application’s data. A load balancer and auto scaling group will handle the scalability of spinning servers up and down as the number of EC2s increases and decreases with demand.

A Lambda script can replicate/clone a current application’s EC2 onto a new EC2 instance by breaking out the build in five simple steps. (Please use the image below to see those steps in motion.)

1. Build a new EC2 instance using the newest available AMI (Amazon Machine Image) that has been thoroughly tested by Quality Assurance (QA) and released by your organization’s engineering team.
2. Create a snapshot of the secondary volume.
3. Transform the snapshot into a usable volume.
4. Attach the newly created volume to the new EC2 Instance and then locally mount the drive using the user_data section of the EC2.
5. Remove the old EC2 Instance.
   1. Remove the EC2 from the ELB.
   2. Remove/unmount the secondary volume from the EC2.
   3. Delete the old EC2 and its associated old volume.

Individual Lambdas should be configured to create snapshots on a scheduled cadence as a CloudWatch event rule. Before there is too much load on an application, load balancers will need to be configured and auto-scaling groups (ASGs) will need to be created and connected together with the application’s EC2. During a rehydration, Lambda functions can be initiated to go through the five step process listed above. Rehydrations can be set to automatically run by enabling a CloudWatch event pattern rule to trigger a Lambda function on a set scheduled job.

In this same scenario, a CFT (CloudFormation Template) can be built ASG and ELB into it’s stack during the application’s creation to balance an application’s load by setting the auto-scaling limits of the system. The CFT build acts as a single stack and during rehydration one can delete individual EC2s by hand (to test a new application build on a single server), update the entire stack using a single button by selecting the “Update” stack button, or create new stacks as a blue-green deployment during each rehydration to set a new primary test environment before deleting the secondary old resources.

Information on serverless application architectures can be found in AWS’s Lambda documentation.

Given my experience with AWS and DevOps, here is some useful advice for working with Lambdas and CloudFormation for first time users.

For Lambda:

• When deleting an EC2 with non-root attached volumes, one must first unmount the volume from the instance before deleting the EC2 instance.

For CloudFormation:

• When updating a volume that modifies IOPS, Size, or VolumeType, there is a “cooldown period” before another operation can occur. During this time, the process will be stuck in a “waiting period” before it can make another change inside of that timed window.
• A stack is a single unit of AWS resources; when a stack is deleted, ALL of the stack’s related resources get deleted together.
• Many properties are not listed as required, but many properties are conditional. One example is SecurityGroupIds, which are only required when one chooses to specify the virtual private cloud (VPC) security groups.

The fastest way to build Lambda functions from scratch is to use a blueprint Blank Function and the fastest way to build a ready to use function would be to browse the Lambda app repository. CloudFormation has many sample region specific templates available to download from the AWS resource website.

In the AWS Console, one can trigger a Lambda function multiple ways. One of the most common ways to trigger a function is to configure a test event within a selected Lambda function and subsequently select the “Test” button. A second approach to trigger a Lambda function is to use an event pattern, such as CloudTrail logs, or a fixed schedule as a CloudWatch event. In the console navigate to the CloudWatch service and select Events > Rules.

1. Complete the target event source section.
2. Set “Lambda function” as the target.
3. Select the given Lambda function name.
4. Configure the details of the rule name.
5. Select the status of the rule to be available.

At any time one can view, edit, delete or disable, CloudWatch rules.

To avoid the pitfall of hard-coding variables in functions, Lambda offers environment variables. Environment variables allow one to avoid hard coding variables by creating variables set outside of the code block to be referenced when executing the Lambda function. Environment variables also allow the code to remain the same while variables such as size, name, region and security groups are updated. It is much easier to update environment variables than it is to constantly hard code variables for specific use cases. (Example of code shown below for  referencing a specific region that was set in the environment variable section using the boto3 library.)

As a good rule of thumb, ALWAYS specify the AWS region(s) that you are using. When AWS regions are missing, issues will occur depending on the AWS service that you are using. Sometimes builds will automatically error out when the region is not specified, such as Lambda functions. Other times regions will be automatically assigned to a CFT if a region is not assigned. When this happens there is a strong chance that it’s not going to be in the region that one would have hoped for. It can also be difficult to locate applications in the console when their location region is unknown.

The best way to troubleshoot through your problems is to use the AWS Documentation and look for specific errors on StackOverflow. There are also many different resources available through blogs and online tutorials that will help you along the way. I also recommend looking at the entirety of Capital One’s cloud content on this blog and on Medium. We share quite a bit of our cloud expertise across both these tech blogs.

It takes time, practice, and patience to undertake the DevOps journey. Determination is also important while building your first application using DevOps. As you start out, feel confident knowing that there are several template examples, blueprints and application repositories that have already been designed and are ready for use. Keep in mind that AWS regions need to be specified in code, Lambda environment variables remove any hard-coded variables, and that CFTs build stacks that will be either created or destroyed together as one single unit.

Don’t worry, certain pain points with CFTs--like cool-down periods--will all become a part of your normalized routine once you become an AWS DevOps Expert. Also, remember that while the three AWS services we covered - Lambda, CloudFormation and CloudWatch - are all services that I commonly use for DevOps, there are so many other AWS services out there with more and more services becoming available. It’s always important to keep learning new concepts and keep an open mind while trying out new services.