Backstage and Cost Insights: shifting cloud costs left

Backstage and Cost Insights: shifting cloud costs left

Cloud computing - the double-edged sword

Managing cloud costs is fast becoming a strategic priority. While cloud service providers (CSPs) like Google Cloud, Microsoft Azure, and Amazon Web Services offer incredible flexibility and scalability, they can quickly lead to ballooning costs if left unchecked. We’ve all seen the cloud billing horror stories, but even for companies that have a good grip on their cloud costs, it’s easy to get into a situation where cost growth begins to outstrip associated revenue growth.

Such a position can undermine profitability, growth, and operational flexibility, ultimately putting the organization on an unsustainable trajectory. Beyond simple expense, runaway costs can mask inefficiencies in architecture, resource allocation, and service usage, creating a vicious cycle of wasted consumption. Left unmanaged, this can undermine even the most successful products, ultimately affecting their long term viability.

A notable case in point is Spotify, who encountered this scenario early on which forced the company to rethink its approach to managing cloud costs. Spotify realized that to get a handle on this, they needed to empower their engineers - the people closest to the actual cloud usage - to own the costs and be responsible for optimizing them. Rather than a top-down approach mandating a cost reduction, Spotify opted to make the engineers themselves responsible for their own cloud costs, an excellent example of the phenomenon of shifting cloud costs left.

Shifting cost awareness left: a cultural change

Shifting cost awareness left is a concept in effective cloud cost management that is gaining traction. Essentially a form of embedding cost considerations earlier in the engineering process, not as an afterthought for finance teams, that’s exactly what Spotify did, according to James Governor at RedMonk: “Spotify engineering teams are used to having a lot of autonomy, so the company couldn’t just introduce new cost guardrails as a top down concern. Therefore the Cost team tried to foster a culture where optimization would be fun.”

Shifting cost awareness left isn’t just about giving engineers tools, it’s a cultural shift where engineers take ownership of the financial impact of their work and are invested in cost optimization. When engineers are empowered with real-time cost insights, they become agents of change. Instead of waiting for an end-of-month cloud bill to identify costly inefficiencies, engineers can make informed decisions in real-time. This shifts cost management left from a reactive process driven by finance to a proactive, engineering-led initiative.

There’s a philosophical value here that goes beyond dollars and cents. By making cost awareness a core part of the development lifecycle, companies can drive a sense of ownership and even healthy competition among teams. Engineers begin to actively look for ways to optimize their cloud usage, often competing with each other to drive down costs. This culture of cost ownership not only saves money but also leads to better architecture and more efficient systems overall.

According to Janisa Anandamohan, Spotify Senior Product Manager, Cost Engineering:

We know engineers are natural optimizers when it comes to reliability, security, performance, et cetera. And now we’re telling them, hey, add costs into the mix. And they were super excited about that. We had many teams that were just able to tweak their services and data pipelines and to make them more efficient. And we know efficiency doesn’t just help cost, but helps reliability and performance as well. So we were getting double, triple wins along the way.

Spotify’s solution: Cost Insights

Spotify’s approach to shifting managing cloud costs left took the form of a cost optimization and visualization plugin called Cost Insights, on their own internal version of Backstage. The concept is simple: provide engineers with the tools to visualize and manage the costs associated with the services they build, all within the internal platform they are already using - their internal developer portal. By tying cloud costs to specific entities in the Backstage catalog, teams are empowered to take control of their spending and optimize resources more effectively.

This approach worked well for Spotify, partly due to their internal engineering culture, but mostly as a result of the significant engineering effort invested into getting Cost Insights integrated into their developer platform. While they have since open-sourced a pared-down version of the Cost Insights plugin back to the community, most organizations attempting to replicate their success will find it challenging, even when using their plugin. This is largely because of the technical complexity required, and because the current Spotify Cost Insights plugin, like many of the other aspects of the Backstage framework, is far from an out-of-the-box solution.

Just how much work would it take an engineering team, even working with the open-sourced Cost Insights plugin to implement a working solution? It’s a significant lift; here’s what they’d have to do:

1. Cloud billing integration: Set up a mechanism such as an API to pull cost data. This involves not only access configuration but also understanding the data schema of your cloud provider, which can be complex and time-consuming.
2. Backend development: Develop a backend to fetch, process, and store the billing data. This step requires designing a database schema that can handle potentially large amounts of data efficiently and setting up a server to run this service.
3. Cost Insights API implementation: Implement the API endpoints required by the Cost Insights plugin. This includes endpoints for fetching cost data, projecting future costs based on current trends, and breaking down costs by services, projects, or departments.
4. Frontend integration: Integrate the Cost Insights plugin into your Backstage instance. This may involve customizing the plugin to fit into your organization’s Backstage environment and ensuring it interacts correctly with your newly developed backend.
5. Testing and validation: Thoroughly test the plugin with real data to ensure accuracy. Validate the cost projections and insights with your finance team to ensure they align with actual expenditures.
6. Maintenance and updates: Continuously update the backend and frontend as cloud providers change their APIs or pricing models, and as new features or fixes become available in the Cost Insights plugin.

For a small team of three to four engineers, this could take anything from several months to a year to fully implement. This is a big lift for most organizations, which means that for many, this complexity keeps the solution remains out of reach. While the concept of cost transparency and shifting cost awareness left resonates, the time and effort required to implement and maintain a working Cost Insights tool deters widespread adoption.

As if it wasn’t hard enough, all of the work above assumes an organization is using only a single CSP in their stack. In reality, any organization that is using multiple CSPs (say, AWS and GCP) faces a thorny additional hurdle: the lack of data homogenization and normalization from CSPs. Each CSP often presents cost data in a different format, making it extremely challenging for organizations to consolidate and interpret this data in a unified manner if they’re using multiple CSPs.

Fortunately, the recent introduction of the FOCUS (FinOps Open Cost and Usage Specification) standard has changed the landscape for the better. Developed as a collaborative effort by the FinOps Foundation, the FOCUS standard aims to normalize the cost and usage data across different CSPs, providing a common format that makes it easier for organizations to integrate and analyze their cloud spending. This standardization is a crucial enabler, simplifying the data integration process and reducing the overhead associated with translating disparate data formats.

Introducing: Roadie’s Cost Insights Plugin

The adoption of the FOCUS standard significantly simplifies the integration of cost data across various cloud platforms. However, the complexity and effort of setting up Cost Insights is still high - which is where we at Roadie saw an opportunity to help. We recognized that the idea of empowering engineers to manage cloud costs was spot on, but the solution needed to be simpler, more accessible, and ready to use out of the box. As such, we’re in the process of refining and enhancing the existing Cost Insights plugin, making it significantly easier to deploy and use right out of the box.

Our enhanced Cost Insights plugin builds on Spotify’s version, but aims to reduces the setup friction, allowing engineering teams to access powerful cloud cost insights immediately - no custom backend development required. This streamlined approach means engineers can spend more time optimizing their services and less time managing the infrastructure for cost tracking.

With Roadie’s plugin, teams can quickly see which services are driving up their cloud costs, how those costs have trended over time, and what actions can be taken to reduce unnecessary spending. The key here is not just providing data but making it accessible and actionable, so engineers can immediately use it to make decisions.

Roadie’s Cost Insights plugin takes full advantage of the FOCUS standard, ensuring that the data we present is both accurate and comparable across providers. By eliminating discrepancies in how cost data is reported, the plugin enables engineers to gain a clear understanding of their cloud usage without needing to reconcile different data formats.

The ability to assign costs to specific entities within a Backstage catalog is what we’re most excited by. In traditional cloud billing tools, costs are often presented at a very high level, making it hard to drill down and see what’s truly driving the expenses. By associating costs with individual services and the teams responsible for them, the plugin makes cloud costs real and relatable for developers. When a team sees exactly how much their service is costing, they can no longer ignore it or avoid responsibility - it becomes part of their job to optimize those costs.

The Roadie Cost Insights dashboard displaying cloud cost by product

How it works

Our Cost Insights plugin, currently in internal beta, integrates seamlessly with a Roadie-managed Backstage instance to provide teams with an out-of-the-box solution for tracking and managing cloud costs. Key features include:

• Easy setup: No complicated setup required - simply connect to your cloud service provider to Roadie Cost Insights via your cloud administration settings or via a secure broker, and allow the Roadie Cost Insights compatibility layer to take care of all the data modeling and translation.
• Project and group-based cost tracking: Track cloud costs at the entity, team, or product level, and drill down into specific projects or groups for more detailed insights.
• Trend visualization: View cloud cost trends over time, breaking down data by dimensions like products, services, and regions to identify patterns and anomalies.
• FOCUS standard integration: The plugin leverages customer-provided FOCUS data, ensuring consistency and like-for-like comparability across cloud providers.
• Actionable insights: Dashboards help engineers take ownership of their services, and immediate action by optimizing those services or reducing over-provisioned resources.

For example, a team might notice that the costs of a particular product are rising faster than expected. With Roadie’s plugin, they can drill down into the cost breakdowns, identify high-cost services, and take corrective action - such as optimizing resource allocations or reducing usage.

Roadie Cost Insights also supports multiple regions, meaning a DevOps team managing multiple cloud services in different regions could use Roadie’s Cost Insights to track cloud costs by region. For instance, they could spot that their compute resources in one region are significantly more expensive compared to others. Using the plugin’s trend visualization and breakdown capabilities, the team can identify the services or instances driving up the cost and adjust their architecture to either reduce or move resources to more cost-effective regions.

Roadie Cost Insights Architecture - note that the user configuration overhead is limited to installing the Cost Insights client on the cloud service provider infrastructure (or the use of a broker client) while Roadie takes care of all the backend setup.

The future of Roadie’s Cost Insights Plugin

We believe that by simplifying cost management and putting it directly in the hands of engineers, companies can not only save money but also drive innovation, efficiency, and alignment across their teams.

Cloud cost management is no longer just a financial issue—it’s an engineering one. By providing real-time insights into cloud spending and shifting cost awareness left, Roadie’s Cost Insights plugin helps companies empower their engineering teams to take ownership of their cloud usage. This leads to smarter decisions, lower costs, and more efficient systems.

While Cost Insights is still currently in an internal beta, if you’re interested in learning more or becoming an early design partner, get in touch with us today. We’d love to work together to bring the future of cloud cost management to your team.

Title image by Brian Penny from Pixabay