Cloud FinOps: a guide to cloud financial forecasti...

Riya2311 · 08-19-2024 04:32 PM

Authors: Bruce Warner, Eric Lam, Andre Ellis, Riya Shrivastava

Overview

Your cloud bills have been coming in quite consistently, varying no more than 2-3% per month. Then the unthinkable happens - this month's bill surges 20% higher than your peak billing this year. Your email and instant messaging light up as the head of IT Finance and CIO urgently ping you. What's going on? Did engineering deploy a new service that had no guardrails in place? Did the analytics team run a query that hit a much larger data set than anticipated? Or did last month's seasonal sale lead to extra customer demand. Is this a good news story in disguise?

As companies increase their investments in advanced cloud technologies such as Generative AI and Machine Learning and create further variability to their monthly spend, their ability to identify and forecast granular cloud costs becomes essential to establishing a well controlled cloud environment. Forecasting, or the practice of making future predictions based on past and current data, has long been in use, with particular focus on portions of the P&L that can fluctuate significantly, such as contract pipeline, revenue, or component and labor costs.

As organizations transform from a data center driven model and adopt more agile infrastructure, the techniques and practices for forecasting in the cloud have evolved as well.

Since companies largely (but not entirely) account for cloud consumption as an operating expense, the way in which Finance manages planning and forecasting processes needs to accommodate variable consumption and related costs.

In order to be successful in this change, organizations need to adopt new methods to effectively and accurately forecast cloud consumption.

What is cloud financial forecasting?

A process to predict cloud spend using a combination of historical (trend-based) and business or demand-driven (driver-based) methods
Cloud Forecasting predicts future cloud consumption and spend for short to long-term periods and typically leverages a variety of statistical and predictive analytics-driven techniques.
Future cloud infrastructure and application lifecycle changes may impact current budgets and influence fiscal year planning and future cloud investment decisions. A robust Cloud Forecasting method and process must address these dynamics.

Key challenges

Finance, Planning and Analysis (FP&A) teams who have historically managed relatively fixed technology costs at a vendor level face multiple challenges to manage dynamic cloud costs

Lack of visibility to real-time spend insights: Limited access to granular cloud consumption and actionable cost data
Antiquated tools for financial planning in the cloud: Lack of modern and advanced technology to analyze and forecast spend
Dynamic consumption & new investments: Challenges to segregate costs associated with new workloads and anticipate drivers for scaling workloads
Shifting from Fixed to Variable cost model: Transition from fixed data center costs to variable cloud consumption

Unfortunately, no one forecasting method addresses all situations. The variable nature of cloud spend is inherently difficult to predict. Engineers can launch workloads at any time typically without having to go through a procurement process. For this reason, the ability to accurately forecast cloud spend depends on other cloud FinOps capabilities, such as detailed and consistent tagging & labeling standards and implementation of a cloud cost allocation method.

Types of cloud financial forecasting

As organizations deploy a wide range of applications on cloud infrastructure, from core systems such as ERP applications to customer facing systems and advanced machine learning and generative AI systems, Finance and FinOps teams can choose from a variety of forecasting methods to reflect the unique characteristics of associated cost drivers. It is important to understand that the best forecasting method for you and your organization will depend on your specific needs, and many organizations choose to use a combination of methods.

Business value-based: Forecast consumption for new cloud workloads and pipeline (both migrated and new cloud native applications)
Trend / historical data: Leverage 6+ months of data to determine key spend pattern behavior, identify growth rates and adjust your forecast accordingly
Driver-based: Identify external demand drivers, factor in impact of new customer products and services, account for seasonality, macro economic impacts and change in business needs

Implementing cloud financial forecasting

Understand the level of cost granularity needed to accurately forecast cloud costs

Forecasting can be performed at different levels of granularity, depending on the specific needs of the organization. It is important to make this decision before beginning the forecasting process, as changing the level of granularity later can be time-consuming and complex.

Forecasting at a less granular level (such as vendor, Business Unit, or cloud service) requires less effort whereas at a more granular level (such as subscription, application, or project) provides more flexibility and allows detailed analysis, but it also requires the most effort.

Those companies who accurately forecast cloud spend are typically forecasting at an application or project level of granularity, as this allows for insights of how the top cost drivers perform. In particular, the ability to understand the patterns and behaviors of dynamic workloads is usually dependent on establishing an application or project level understanding of costs, especially for the top 20% of projects that drive a majority of costs.

Define labels, tags, and folder structure

To facilitate allocation of costs at the desired level of granularity, organizations should define a comprehensive labeling, tagging, and folder structure for all cloud resources.

In Google Cloud, labels and tags can be used to allocate costs at the project, application, billing account, folder, organization, or resource level. Folders can also be used to organize resources for cost allocation purposes. The design of the tagging, labeling, and folder structure should not be left to IT alone. Given the dependency on forecasting, cost allocation, chargeback and advanced metrics such as unit costs, Finance and business leadership should also be involved in defining these standards.

Define forecasting method

At the granular cost level, define the forecasting method that will be used for each cost bucket (trend-based, business value-based, or driver-based). Analyze cost trends and consider seasonality and shifts in demand.

Establish forecasting frequency and feedback loop

Organizations should choose a forecasting frequency that aligns with their business needs and forecasting objectives. Many companies have moved to monthly forecasting of cloud costs, and some have even adopted a weekly/biweekly forecast in situations where cloud costs represent a significant portion of costs of goods sold. Every forecasting process should include a feedback loop to review variances on a timely basis with respective product owners or lines of business. As part of this cycle, forecasts are compared to actuals to identify any discrepancies. Those variances that exceed a given threshold, such as 10% and exceed a certain $ amount, should be investigated by the respective application team to understand the cause of the variance. Variances will be due to either resource inefficiency, differences in how costs are modeled versus how they perform, or demand variation, and those learnings can be used to improve future forecasts.

Leverage anomaly detection

As companies forecast at a more granular level, it can be difficult to discern between ‘noise’ and trends. Use of anomaly detection capabilities can help businesses automate the identification of cost aberrations to quickly address true cost discrepancies as well as new demand trends, both of which can strengthen forecasting models and improve forecasting accuracy across the organization.

Detecting changes in the underlying trends and patterns in the data can be used to update the forecasting model and improve the ability to anticipate consumption changes.

Examples of financial forecasting

As a way to analyze our methodology, let’s use an example to showcase this concept in practice. In this example, a large enterprise technology company which has been one of the leading pioneers in developing cloud-based solutions, is seeking ways to enhance its business strategies leveraging cloud FinOps capabilities (namely cloud forecasting). The finance and accounting team explored the construct of forecasting future expenditures to ensure optimal resource allocation and cost-effectiveness.

Trend/historical based

The team decided to start their forecasting journey by leveraging the historical data of the past 6+ months. The team meticulously examined the spending patterns, identified key growth rates, and adjusted their forecasts accordingly. During the historical analysis, they discovered that cloud usage tended to spike during the holiday season as businesses scaled up their operations. Armed with this knowledge, the company strategically allocated additional resources and support during peak times, ensuring seamless service delivery and customer satisfaction. Moving forward the team now has the ability to leverage Vertex AI to enhance its pattern recognition capabilities by querying years of historical data which will increase its forecasting accuracy.

Business value based

As a way to quantify the value, the team aligned forecasting with the strategic goals of the business. One goal of the team was to launch a new service targeted at e-commerce businesses. The team projected the resource consumption for this new workload, factoring in the expected influx of clients and the specific requirements of the cloud-native application. This allowed the team to allocate resources appropriately, ensuring a seamless and tailored onboarding experience for its new clients.

In conclusion, the finance and accounting team’s strategic use of forecasting methods ensured that the company was not only prepared for current demands, but also poised to adapt to the dynamic nature of the cloud services market. The combination of historical analysis and business value-based forecasting allowed the team to navigate the complexities of the industry with agility and foresight.

Conclusion

In the era of cloud computing, cloud financial forecasting has emerged as a critical practice for organizations to effectively manage their cloud expenditures and make informed decisions about their cloud infrastructure and application lifecycle. By embracing a combination of historical trend-based approaches and forward-looking business-driven techniques, cloud financial forecasting empowers organizations to:

Predict Future Cloud Costs: Gain proactive insights into projected cloud spend for short to long-term periods, enabling organizations to plan and allocate resources effectively.

Understand Business Drivers: Identify the impact of changes in business demand, application usage, and infrastructure requirements on cloud costs, allowing organizations to anticipate and adapt to future spending patterns.

Optimize Cloud Investments: Make informed decisions about cloud resource allocation, scaling, and procurement strategies to maximize value and minimize waste, ensuring that cloud investments align with business objectives.

Empower Data-Driven Decision-Making: Translate cloud cost insights into actionable strategies for improving cost efficiency and aligning cloud spend with business goals.

Organizations should carefully consider their specific needs, data availability, and technical expertise when implementing a forecasting method and process. By adopting a data-driven approach and continuously refining forecasting techniques, organizations can harness the power of cloud computing without compromising their financial goals.

You can learn more about FinOps on Google Cloud by visiting the following resources: