Improving developer productivity: Investing in governance

Governance is the balance between speed to market and risk avoidance. Capital One teams strive to innovate daily on behalf of our customers, delighting them with new technology and exceptional customer experiences. To do so, we must evolve existing systems and innovate new ones through mechanisms that allow engineers to deliver systems within guardrails that meet compliance requirements. Balancing these two goals is challenging, but developer productivity is unleashed when the right combination of proactive and detective controls are employed.

When implementing governance controls, it's important to consider both proactive and detective controls. Proactive controls validate governance policies before resources are deployed to an environment, preventing the introduction of noncompliant resources. Detective controls detect, log, and alert on resource changes that may violate governance policies for deployed resources. Both proactive and detective controls are necessary guardrails in well-governed architectures.

Let's look at an example to see how proactive and detective controls can contribute to balanced governance.

Serverless applications often include multiple AWS services that are combined to provide a larger capability to users. For example, an event-driven serverless application that processes data published to an S3 bucket may be represented by the following architecture diagram.

In this solution, an Amazon EventBridge rule filters S3 object creation events, triggering an AWS Lambda function invocation. The Lambda function includes business logic to process the S3 object, updating an Amazon DynamoDB table. Updates to the DynamoDB table are then streamed to a second Lambda function for processing and downstream notification.

This application includes business logic in two Lambda functions, which is coded and unit tested on an engineer's local workstation, using a combination of an integrated development environment (IDE), application frameworks, and test data to complete the task effectively. Local development and unit testing provide fast feedback loops to engineers as they write code to meet business requirements.

The application also includes code that describes the AWS infrastructure configurations, like the S3 bucket name in the EventBridge event rule, the Lambda function memory size, and the DynamoDB table schema. For serverless applications, engineers can use a single code repository for all the components by using AWS CloudFormation or AWS Cloud Development Kit (CDK) to combine the function logic with resource configuration in a versioned artifact that can be deployed through an automated pipeline to multiple environments for testing and release.

To ensure deployed AWS resources comply with policies, detective controls execute through event-based or scheduled triggers. Governance rules engines, like Open Policy Agent or AWS Config Rules, are used to identify noncompliant resources, provide notification, and trigger corrective actions.

To deploy the application, an engineer commits the code changes (application and infrastructure) to GitHub, which triggers an automated deployment pipeline. The pipeline is responsible for proactively checking that the application's AWS resources configurations comply with cloud control guardrails.

This is an effective mechanism to avoid risk, however it increases the time for developer feedback between application changes and errors resulting from noncompliant configurations. An increase in feedback time leads to a decrease in productivity.

By incorporating command line interfaces (CLI) and IDE tooling, the same proactive enforcement used by the pipeline can also be executed on the engineer's workstation, as illustrated in the diagram below:

IDE and CLI tools allow engineers to scan their Infrastructure-as-Code (IaC) files locally to identify cloud governance compliance violations. These tools follow the same established techniques used for the local development and testing of business logic. They provide early feedback on noncompliant resource configurations and increase confidence in successful pipeline executions. By decreasing developer feedback time, productivity is increased.

Our commitment to innovation within the framework of effective governance underscores the significance of proactive and detective controls. As organizations navigate the evolving landscape of innovation, the insights from Capital One provide valuable guidance for achieving success in the delicate equilibrium between speed, control, and sustained productivity.

If you’d like to learn more, please watch the recording of SVS309 from re:Invent 2023, where I co-present with our AWS Principal Specialist SA for Serverless, Heeki Park. Heeki dives deep into AWS services used to enforce proactice and detective controls, and I speak about how we’ve applied similar practices at Capital One.