Managing Snowflake warehouses with Capital One Slingshot

Data warehouses in the cloud are foundational for data-backed businesses that manage and analyze massive amounts of data from any number of sources. Many organizations moving their data workloads to the cloud have turned to Snowflake, a flexible, scalable data cloud platform, for their data warehousing needs. However, effectively managing costs and use of resources at scale is important to minimize data inefficiencies and balance Snowflake cost and performance.

Capital One’s Slingshot helps businesses looking to scale and manage their data workloads with Snowflake to achieve that balance. Slingshot’s warehouse scheduling and provisioning feature allows businesses to set dynamic schedules and automate provisioning processes to reduce manual oversight and minimize data spillage, without sacrificing performance. By managing Snowflake warehouses with Slingshot, organizations can optimize costs, improve performance and maximize their use of Snowflake to meet their business goals.

Due to its unique architecture, a Snowflake data warehouse allows organizations to run complex, resource-intensive analytics at scale on large volumes of data without affecting performance. The compute costs of running warehouses make up the bulk of Snowflake cloud costs for businesses, so optimizing Snowflake data warehouses is key to making the most of an organization’s investment in the platform. However, there are several things to consider to effectively manage the costs and resources of warehouses.

• Scalability: As organizations grow, their data warehouses must handle growing volumes of data and greater complexity, requiring more compute power and leading to greater costs. Many organizations face difficulties as they scale in identifying inefficiencies and the right times to reduce or increase resources to optimize their warehouses.

• Cost management: With Snowflake, customers only pay for what they use. But with the consumption-based pricing model, costs fluctuate based on need such as data volume and query complexity. It is important to closely manage costs, particularly at scale, to allocate resources efficiently and proactively address cost spikes to minimize unnecessary expenses. Provisioning warehouses based on need and usage can help businesses minimize costs and bring predictability to their cloud management.

• Warehouse provisioning: In Snowflake, warehouses are managed and optimized manually, which is increasingly time consuming as a business scales. It can also introduce inefficiencies such as over- or under-provisioning warehouses. At the organizational level, manual provisioning and optimization of warehouses affects employee productivity, taking time away from more strategic tasks.

• Warehouse management: Failing to properly provision a data warehouse leads to issues with speed and performance, such as sluggishness in running queries. Additionally, warehouse requirements fluctuate. For example, demands can change throughout a day on peak and off peak hours, and the warehouse size should change to accommodate the compute power required.

Slingshot significantly reduces the complexity of managing data warehouses in Snowflake, providing automation and standardization to warehouse optimization and cost efficiency. Provisioning a warehouse involves configuring the right amount of resources, like computing power, to process and analyze data. As organizations grow data workloads and expand their use of Snowflake, provisioning warehouses in a way that balances cost and performance across various stakeholders and teams becomes time consuming and complex.

Snowflake allows customers to provision warehouses with parameters like warehouse size (compute power), minimum and maximum number of clusters and auto-suspend time. However, these parameters remain standard once set for a warehouse. For example, a medium size warehouse remains that size whether running a few or many queries, which means the organization incurs the same cost regardless of workload. While parameters can be changed, they must be done manually, which is time consuming and difficult to scale as the organization grows.

Slingshot builds on Snowflake to help increase resource efficiency and optimize warehouses with dynamic scheduling. In Slingshot, Snowflake customers can create a schedule that tunes warehouse size and scaling policies (standard or economy) based on need, like usage, environment, day or time.

Through dynamic schedules, warehouses can scale automatically in size at different times of the day to accommodate the peaks and valleys of computing power, such as the difference in query demands in normal working hours versus during off hours. Slingshot dynamically changes the parameters of the warehouse during those set times to achieve maximum efficiencies of cost and performance in Snowflake. Multiple schedules can also be set, which provides granular control over warehouses scaling up or down depending on usage. Slingshot also provides recommendations to right-size warehouses based on historical usage patterns to continue optimizing schedules into the future. By tailoring Snowflake data use in these ways, Slingshot helps organizations reduce data spillage and pay only for what’s needed.

Once set, schedules and policies are automatically applied to new warehouses, reducing the time and effort needed to manually provision warehouses and freeing teams to work on other tasks. Additionally, warehouse templates with preconfigured parameters allow for the quick provisioning and creation of warehouses with built-in governance controls. 

For the times when a warehouse is inactive, dynamic scheduling with Slingshot allows for the managing of auto-resume, auto-suspend and set off-hours scheduling, enabling companies to only incur costs that are necessary.

Capital One Slingshot helps businesses optimize their Snowflake investments to strike the right balance between cost and performance. Through data warehouse scheduling and provisioning, companies can use Slingshot to simplify data warehouse management in Snowflake, with the flexibility to set predictable changes to warehouse parameters while controlling costs.