Best practices and top trends for serverless at scale

Capital One’s journey to serverless architecture taught our teams a lot. The following are what we consider to be best practices, so others can leverage the potential of serverless computing. This blog will also discuss lessons learned, top trends in serverless, as well as predictions. 

Capital One operates within AWS Lambda at a massive scale. We run thousands of accounts, and within those are tens of thousands of Lambda functions. Also, Capital One is a bank and is highly regulated, so we have to create compliance controls around how we operate all these accounts. What do you do when you need to manage thousands of functions? You create standards. 

Setting development and compliance standards is imperative. That’s why we launched our Serverless Center of Excellence (COE) to support our cloud adoption and modernization and to drive our serverless strategy, enablement, and support. We believe our COE defines the best practice and standard for leveraging serverless architecture at scale.

At Capital One, we have 10,000 engineers across multiple lines of business. While I work in the Retail Bank line of business there are other engineers in the Credit Card line of business. There are still additional engineers in Cyber. If I don’t engage the other lines of business, I risk making siloed decisions that could impact the whole enterprise. Decisions that impact the entire enterprise should be collaborative. Our Serverless COE team includes representatives from each line of business who collaborate to establish best practices for serverless at scale. 
We not only get rid of siloed decisions, but our COE reduces tech debt. Our team meets regularly to discuss and create guidance on all kinds of enterprise issues, such as Lambda default settings, observability, runtime deprecation process, and training. 

Needless to say, we have learned a lot from our serverless journey. Here are a few key takeaways:

Leverage industry-standard tools, particularly those designed for serverless, and empower developers to be productive and iterate quickly. Tools like AWS Serverless Application Model (or SAM) help streamline development processes. AWS SAM has a command line interface that allows you to build and iterate locally without needing to deploy to the cloud. 

Shift the unit of scale away from Requests per Second to Concurrency and make sure your developers understand this new paradigm. This might be a new concept to your team, so you’ll want to make sure that your developers understand how to calculate concurrency. Concurrency equals the average requests per second times the average request duration in seconds. Everything is measured in seconds. Let’s look at an example, say you have a Lambda-backed API with 100 RPS. Let's assume it runs your function for an average of 500 milliseconds - half a second. 100 times point 5 equals 50. It’s going to consume 50 units of concurrency. How about one more example? RPS starts at 100, but your durations are now higher - to two seconds. That’s 100 times 2, so now you need 200 units of concurrency to service this use case. Always keep in mind that, if you’re working in Lambda, the goal is to drive the invocation times down and minimize the amount of concurrency consumed. Keep in mind there's a limit to the amount of concurrency available per account per region.

Embrace a multi-account strategy where each team, or each team and environment, has their own AWS account. Opt for many small accounts over a few large ones to limit the blast radius and allow teams to scale usage with fewer constraints. Why is having a multiple of smaller accounts better than a few large accounts? Because this strategy allows you to limit blast radius and provide each team with their own accounts. AWS Organizations makes governance and management of these small accounts a lot easier.

Set development standards. Establish clear standards like the use of aliases and versions to ensure consistency across functions and be sure to enforce these standards. For example, we require a (LIVE_TRAFFIC) alias to be the target for all Lambda functions. This ensures that we understand how all Lambda functions are executed. We also require all functions to be deployed using versions to ensure consistency and prevent accidental changes. 

Through our COE, we’ve identified a list of recommended practices that come from many trials and much testing. Here are the top three:

• Optimize memory. Memory is the one setting that has a big effect on function performance and cost so getting it right is important. Utilize tools like Lambda Power Tuner to optimize memory allocation.
• Optimize for Lambda’s cold/warm lifecycle. This includes not only code but also packaging, and configuration optimizations.
• Understand observability signals. The need to observe your Lambda applications doesn't change but the tools and techniques might. Keep an eye on key metrics to ensure optimal performance and resource utilization.
• Optimize for cost and performance. Use metrics to understand how to allocate resources. This understanding will result in lower cloud costs and more performant serverless applications.

Here are also the top trends I see in serverless technology:

• Multi-cloud approach. There is a growing interest in serverless computing to use a multi-cloud approach. The benefits of multi-cloud deployment for serverless applications is that there is not vendor lock in and it increases availability. 
• Use of AI. AI models can be deployed via serverless architecture. This can prove efficient for scaling machine learning tasks.
• Securing the software supply chain. Third-party dependencies form an attack vector for serverless applications. Securing the software supply chain is even more significant. 
• Durable execution. Serverless developers are looking for ways to ensure code runs to completion. This becomes vital as businesses move critical business processes to serverless.

I predict the interest in serverless architectures will continue to increase and large enterprises will continue to realize the benefits of lower Total Cost of Ownership (TCO), allowing developers to focus on delivering value and delighting customers. 

If you are interested in more of our learnings, check out my team’s serverless blogs. Feel free to borrow from these best practices if you and your team are also seeking to leverage the potential of serverless computing within a massive-scale environment. 

We’re all in on the cloud, serverless and open source, driven by our mission to change banking for good.

• Learn more about serverless at scale at Capital One.
• Explore tech careers at Capital One.