Responsible data governance using Capital One Slingshot

With the exponential growth of data in the cloud, the need to properly manage and govern data has become a critical part of an effective data management strategy, as well as for the overall success of a business. 

At Capital One, we recognize the full power of data cannot be realized without good governance. Companies need a way to balance responsible governance with the enablement of their team members. Let’s take a closer look at how we’ve been able to achieve this while solving common data governance challenges.

Balancing good governance and enabling teams is a challenge because you want to ensure data meets security and quality requirements, while at the same time not keeping business teams from maximizing the value of data.

At Capital One, we were able to find this balance by first dispersing data responsibilities to our lines of business, which removed bottlenecks and freed our data team from a backlog of data requests. By further federating data management to our lines of business in a self-service model, with centralized tooling and central policies, we enabled data owners to manage it and oversee governance. 

Next, we needed to ensure these self-service capabilities were well-managed. We identified the following data management challenges that proper governance could address:

• Poorly written queries: Poor queries cost businesses on performance and money. Proactive monitoring and alerting can help companies identify and fix bad queries early, as well as optimize query performance.

• Overprovisioning compute: Infrastructure overprovisioning can lead to wasted resources. Governance policies specifically for infrastructure can reduce waste. 

• Creeping storage costs: Storing too much data can lead to increased costs and poor data hygiene. Bringing full visibility across all accounts with built-in retention enforcement can ensure data is retained and stored for the appropriate time.

• Budgeting and consumption-based billing: Operating in the cloud means companies must adjust to consumption-based costs versus the fixed costs of on-premises systems. Custom and granular chargeback reports can help control costs while promoting transparency and accountability.

With our teams enabled through self-service capabilities, we enabled federated data governance and self-service capabilities for our teams by building Capital One Slingshot. Slingshot was designed to optimize Snowflake investments, enhance efficiency and reduce cloud costs at scale. This helped us streamline governance for our Snowflake data cloud so we could scale responsibly. 

Slingshot enabled federated governance for our teams by: 

Teams were able to enforce consistent standards and guardrails for warehouse provisioning by offering pre-configured templates for each use case. If you wanted to limit the size of a warehouse that an individual can create, for example, this could be accomplished through customizable templates.

We also gave our teams mechanisms to manage their portfolio of warehouses and make informed decisions based on key data points and data-driven recommendations.

We built approval workflows that allowed users to create warehouses with the appropriate oversight in place. 

Cost insights dashboards with tagging capabilities gave our team members greater transparency into costs and understanding of their usage trends to make informed decisions. These dashboards detected cost spikes, made spending trends visible and helped with forecasting. They also allowed for viewing costs and usage by month, business organization, warehouse, environment or account.

Taking these steps to apply the right governance to our data led to many benefits for Capital One, including hundreds of new Snowflake use cases, the ability to run billions of queries each year, reduced Snowflake compute costs and thousands of data engineer hours saved per year.

As companies push forward in their data journeys, it will be important to scale responsibly. By empowering teams with the right strategies and tooling, businesses can realize the full power of their data while limiting risk, optimizing costs and reducing waste and inefficiencies.