Addy Osmani published The Agent Stack Bet a little while ago and it's getting the attention it deserves. He names four infrastructure bets that teams building production agents need to place: dedicated agent identity, universal context integration, persistent durable execution, and purpose-built platform primitives over DIY plumbing. The framing is right, and his list is almost complete.

Almost. What Osmani describes is the infrastructure that lets agents operate. He's largely silent on the infrastructure that makes operating them safe. The gap shows in month three, not the first sprint - when someone has to account for what the agent did, not just whether it's running.

The fifth bet is runtime governance. The Cloud Security Alliance found that only 16% of enterprises currently govern AI agent access to core business systems effectively.

What the four bets actually buy you

The four bets are real and the industry is under-invested in all of them. Agent identity matters because agents operating on shared credentials are impossible to audit and trivially compromised. Context integration matters because an agent reasoning from thin or stale information is worse than useless - it's confidently wrong. Persistent durable execution matters because multi-step workflows that can't survive a restart or a credential rotation can't do real work. Building on platform primitives rather than hand-rolling infrastructure is sound engineering at any scale.

But notice what those four bets describe: the agent as a machine. A machine with an identity, access to data, an ability to run for a long time, and a well-built chassis. They don't describe who operates that machine, what it's allowed to do under different conditions, how you inspect what it did, or who is responsible when it acts outside its intended scope.

Osmani's piece is about infrastructure for building agents. When teams move from "this works in staging" to "this is running in production on 40 workflows", they discover that infrastructure is necessary but not sufficient. The gap is operational.

What governance debt actually looks like

Osmani calls this "governance debt" - his phrase for the silent accumulation of security and audit risk that eventually forces a full rewrite, usually right after the first incident that reaches the CISO. The frame is right.

An agent with the four bets in place can run cleanly for weeks across dozens of production workflows. Then it takes an action that shouldn't have happened. Maybe it escalated a ticket to an external partner using a template that was out of date. Maybe it triggered a deployment to a production environment during a freeze window because it didn't have visibility into the freeze state. Maybe it queried a data source that had recently been reclassified as sensitive.

The incident review happens. The question is simple: why did it do that?

Agents do produce decision traces. The model usually surfaces what it reasoned about, what it called, and what it tried. The problem at production scale isn't the absence of traces - it's that raw model traces aren't structured for accountability. Without an audit trail that captures what context the agent saw at runtime, what policy it operated under, and what decision pathway led to that specific action, you can't answer the question that actually matters: why was it allowed to do that?

That's governance debt coming due. It's a showstopper. Engineering leadership, legal, compliance - they don't care how impressive the efficiency ratio is. They care whether you can account for what the system did. When it arrives, the failure tends to follow a recognisable shape: the agent ran cleanly for weeks, then took one action nobody had authorised, and the question of who was responsible stalled the rollout regardless of how the infrastructure had performed.

The three things governance actually is

Runtime governance covers three distinct functions, and they have to work together.

Policy enforcement

An agent with a valid identity and access to correct context can still take actions outside its intended scope. The distinction matters: identity establishes who the agent is, policy establishes what it can do right now. Those are different questions with different infrastructure answers. Osmani correctly argues that policy should be enforced at the platform level, not in application middleware. But that principle needs to be cashed out operationally. Runtime governance means a policy layer that evaluates each agent action against current rules before executing it, not after. Not a system prompt saying "don't touch production". An infrastructure-level enforcement point that determines what the agent can do before it does it.

The policy needs to be dynamic, too. A deployment agent that has write access during normal operating hours should not have the same access during an active incident, or during a code freeze, or when the target service is in a degraded state. Static permission grants don't handle this. Runtime policy enforcement does.

Context quality standards

This is the one that surprises most teams. You've built the context layer. You've integrated your sources. The agent has what it needs.

Context has a quality dimension that's separate from its existence. A deployment record from three weeks ago tells you less than one from three hours ago. An ownership record created before a reorg may point to a team that no longer owns the service. A runbook never validated against current infrastructure may be accurate, or may be subtly wrong in ways that only show up in edge cases.

Without provenance tracking - where did this fact come from, when was it last verified, how should conflicts between sources be handled - the agent consumes data of unknown reliability. At small scale that's manageable. At production scale, with agents acting on context across hundreds of services simultaneously, an untracked staleness problem propagates into dozens of decisions before anyone notices. Governance includes the standards that keep context trustworthy, not just the pipeline for ingesting it.

The governance question here is accountability: not just who built the pipeline, but who signs off that the context an agent is about to act on is trustworthy enough for the action it's about to take. That accountability has to be explicit. If it isn't, it defaults to nobody, which means the agent is operating without a quality floor.

Designed human oversight

Osmani mentions human-in-the-loop approval gates as part of his persistent execution bet. Right call, but the framing can be tightened. Human-in-the-loop should be a governance design pattern, not a recovery mechanism you activate when something goes wrong.

The difference is architecture. Recovery-mode oversight says: pause the agent when it's about to do something catastrophic. For that to work, you need to know in advance what "catastrophic" looks like, and you need to have defined the triggers correctly. In production, you won't always know. The novel failure modes - the ones that damage trust are usually the ones nobody anticipated.

Designed oversight says: at these specific points in the workflow, a human reviews the agent's proposed action before it runs. Not because you expect failure, but because the workflow has high enough stakes that human judgment belongs in the loop by design.

When the ratio of agent actions to human decisions reaches production scale, the humans aren't reviewing everything - and they shouldn't be. The whole point is to get humans out of the routine path. Governance determines what the humans do review: the decision points where errors compound, where actions are irreversible, where the agent is operating at the edge of its validated context. You have to design those checkpoints in advance, not discover the need for them afterwards.

Why the IDP is the natural governance layer

The hard parts of governance infrastructure are largely already built - for humans.

A mature internal developer portal already governs what developers can do. It controls which scaffolding templates are available. It enforces which deployment targets a team can push to. It gates access to production systems. It tracks ownership, so every service has a named team accountable for it. It records the relationship between teams, services, APIs, and dependencies.

Extending that governance to agents is not starting from scratch. The portal already knows the ownership graph. It already has the policy model for what different teams can access and change. It already maintains the service topology that tells you what an agent is allowed to touch on behalf of which team.

The audit trail question - who ran this, from what state, and when? - is the same question the portal already answers for human actions. The infrastructure for answering it is the same infrastructure runtime governance for agents needs.

I've argued before that the biggest mistake platform teams make is treating agent deployment as a technical problem when it's an organisational one. You can't measure deployment frequency across your organisation until you agree on what a deployment is. No tool can solve that alignment problem for you. The same logic applies to governance. You can't enforce what agents are allowed to do until you've agreed on what they should be allowed to do - and that agreement has to exist at the team level, the service level, and the environment level simultaneously. The portal is where those agreements already live, because it's where platform teams have spent years capturing them.

Platform teams are positioned to own the governance layer because they already own the hard parts. They understand what "context quality" means in their environment because they've spent years keeping the catalogue accurate for humans. The policy model already exists because they've spent years managing what developers are allowed to do. Runtime governance for agents extends that practice.

The fifth bet

Osmani asks what happens to teams that don't place the four bets. They stay trapped at the demo stage - agents that impress in staging and fail in production.

The governance gap creates a different but equally costly trap. Teams place the four bets correctly. They build something that genuinely works. They scale it to production. Then they get shut down after the first serious accountability failure - not because the infrastructure was wrong, but because there was no governance layer to make it auditable, policy-constrained, and safe to operate at the scale they'd reached. Gartner projects that over 40% of agentic AI projects will be cancelled by 2027 due to inadequate risk controls.

Build it early. Policy enforcement, context quality standards, and designed human oversight are much cheaper to add before an agent is running 40 production workflows than after you're trying to reconstruct why one of them did something wrong.

The fifth bet doesn't generate the conference talks. It generates the confidence to keep the programme running past month three.

The vocabulary around AI agents is a mess right now. "Context" means different things depending on which tool or blog post you're reading. "MCP" gets used for the protocol and the server in the same sentence. "Agent" covers everything from a one-shot LLM call to a fully autonomous system that writes code and merges PRs.

If you're building on a platform team and trying to hold a coherent conversation about how AI fits into your stack, shared definitions help. These are the terms we use at Roadie. They cluster into three groups: agents, context, and MCP. For each term below: a definition, a concrete example from platform engineering, and a note on how it connects to the others. If you want the full argument for why context architecture matters before diving into definitions, Smart Agents Need Smart Context covers that ground.

Agents

An agent is an LLM-driven system that can plan and execute multi-step work by combining reasoning with tool use. In practice, an agent wraps a model with a goal or task specification, access to tools and data sources, memory or state, and guardrails covering permissions, policies, and evaluation hooks.

What separates an agent from a plain prompt-and-response is that the model decides what steps to take, uses tools to get information or take actions, and checks its work against the original goal. An agent investigating a production incident queries logs, checks recent deployments, correlates alerts, and produces a structured finding. It can open a ticket or page a team.

Agents need context to do useful work. That context comes from the context layer and is accessed via tools. MCP is the protocol that makes tool access standardised across different systems.

Harness

A harness is the surrounding runtime and scaffolding that makes an agent reliable and testable.

If the agent is the reasoning engine, the harness is everything around it: the system instructions that set the agent's behaviour, the adapters that connect it to tools and handle retries, the state management that tracks intermediate work across a multi-step task, and the logging and evaluation hooks that let you see what the agent did and whether it did it correctly.

When a team says "we've built an agent for X", they usually mean they've built a harness around a model for X. The model is often off-the-shelf. The harness is the engineering work - and it's where most of the investment sits. Swapping the underlying model in a well-built harness takes days. Rebuilding the harness from scratch takes months.

In the context of this glossary, the harness is what connects an agent to its context sources and MCP tools. It handles auth, retries, and observability so the agent logic doesn't have to.

Skill

A skill is a reusable, named capability that packages decision logic, procedural knowledge, and optionally tool use into a unit an agent can invoke for a specific class of task. Where a tool is a discrete function - fetching data or taking an action - a skill is the logic that defines how to approach a task: which steps to take, which tools to call, what conventions to apply, and how to structure the result. Some skills are purely instructional, encoding domain knowledge or output standards with no tool calls. Others orchestrate sequences of tool calls. Most non-trivial ones combine both.

An agent handling an on-call alert might invoke a "triage-deployment-failure" skill, which encodes the steps to follow, calls the deployment history tool, queries active alerts, checks the owning team's runbook, and returns a structured finding. The calling agent gets a result without reconstructing that logic each time.

For platform teams, skills are the unit of reuse in a multi-agent system. A blast-radius-assessment skill, once defined, can be called by any agent in the stack - an incident-responder, a change-management agent, a developer-facing assistant. They're also a natural testing boundary: you can verify that a skill produces the right output for a given set of tool responses without testing the full agent reasoning loop.

Skills are distinct from a Harness, which wraps a specific agent and manages its runtime. A skill is portable logic that any harness exposing the right tools can invoke.

Context

Context is the information an agent needs to do useful work for a specific situation.

This is distinct from the model's general training. A model is trained on broad data. Context is what you inject at runtime to make that general capability specific to the task at hand. When an agent reviews a pull request, the relevant context isn't everything the model knows about code review - it's this PR, this repo's conventions, this team's recent deployment history, and the services this change touches.

Good context is relevant to the task, scoped so the model isn't processing noise, and trustworthy enough to act on. Bad context - stale, incomplete, or inaccurate - doesn't just fail to help. It causes the agent to act on wrong information, which is often worse than no information at all. The sub-concepts below describe different ways of structuring and thinking about context when you build infrastructure to support it.

Context Plane / Context Layer

The context plane - also called the context layer - is what an agent has access to at runtime: entities, relationships, temporal state, rules, provenance, and standard operating procedures compiled into a queryable store. Usually this is a graph.

In a platform engineering context, services are nodes, ownership and dependency relationships are edges, and metadata attaches to each node. An agent assessing the blast radius of a proposed change queries this graph and gets a typed, traversable result - not a text description. The context plane is what turns a service catalog from a documentation tool into a data source agents can actually use at runtime.

The quality of the context plane depends on the quality and freshness of its underlying sources - catalog data, deployment records, observability outputs, runbooks. A graph built on stale or partial data produces stale or partial answers.

Context Lake

A context lake is the broader collection of raw context sources available to a platform: service catalog data, repository metadata, observability signals, incident history, ticketing systems, runbooks, deployment records, and anything else an agent might eventually draw from.

The context lake is everything that could feed the context layer. It's distinct from the context layer in that not all of it is structured, current, or relevant to any given task. An agent investigating an incident needs recent deployment history and active alerts, not every runbook in the organisation. The context layer selects and structures what's relevant for each task; the context lake is where those raw inputs live.

The term is in wider market use. What Roadie means by it specifically: the full set of raw data sources that platform teams manage and that the context layer compiles from - not the compiled, queryable result, but the underlying inputs.

Business Context Layer

The business context layer is the set of organisational and operational facts that sit above raw technical data: team ownership, cost attribution, service criticality, SLOs, compliance requirements, and incident accountability.

Technical context tells an agent what a service does and how it's connected. The business context layer tells it who owns it, what it costs to run, and what the consequences of a failure are. An agent that can traverse a dependency graph but doesn't know which team owns a downstream service - or whether that service is customer-facing - doesn't have enough context to make reliable decisions about escalation or rollback.

For most platform teams, the business context layer is the hardest part of the context problem. Not because the information doesn't exist, but because it's distributed across ticketing systems, cost dashboards, spreadsheets, and institutional memory.

Minimum Viable Context

Minimum Viable Context (MVC) is the smallest set of context that enables an agent to complete a specific task reliably. More context is not always better. A larger context window costs more, takes longer to process, and can distract the model from what it actually needs to do. Too little context produces hallucinations and errors. Minimum Viable Context is the engineering discipline of finding the right set - the context that's necessary, and no more.

Determining the MVC for a task means asking: what does this agent actually need to do this job without making critical errors? For an agent investigating a deployment failure, the MVC might be the deployment record, the last three alerts, and the owning team's runbook. It probably doesn't need the full incident history for every service in the organisation.

Getting MVC right is one of the main levers for improving agent reliability and cost. It's also a harness design problem: a well-built harness assembles only the context the task requires, rather than passing everything available into the model's context window.

MCP

MCP, or Model Context Protocol, is a lightweight protocol that standardises how an application or agent provides tools and context to an LLM. MCP defines a common interface for discovering tools, calling them with structured inputs, and returning structured outputs - so models can interact with external systems without custom integration code for each one.

The practical benefit is that MCP solves an integration problem at the protocol level. Before MCP, connecting a model to an external tool meant writing bespoke code for that tool, and repeating that work for every new tool. MCP replaces that with a shared specification that tools and models both implement once.

MCP Server

An MCP Server is a service that implements the MCP specification and exposes capabilities to models. It publishes a catalog of available tools, validates inputs, enforces auth and permissions, executes tool calls or forwards them to the underlying service, and returns results in a consistent, machine-readable format.

In a platform engineering context, a service catalog backed by an MCP Server means any agent can call "which teams own services in the payments domain?" and get a typed, structured answer - not a blob of markdown from a docs search. The MCP Server handles the translation between the model's tool call and the catalog's underlying data model.

A catalog-backed MCP Server tends to be the highest-value starting point for platform teams building agent tooling, because service ownership and dependency data is already maintained and is high signal for most agent tasks.

MCP Gateway

An MCP Gateway is a routing and policy layer that sits in front of one or more MCP Servers.

As you add more MCP Servers - one for the catalog, one for your observability platform, one for your ticketing system - you end up with scattered auth configurations, inconsistent rate limits, and no single point for setting policies about what agents can do. The gateway centralises all of that: authentication and tenant isolation, tool allowlists and safety policies, request routing and load balancing, observability across all tool calls, and version compatibility between clients and servers.

Most platform teams don't need a gateway on day one. In our experience, it becomes worth introducing somewhere between three and ten MCP Servers. Before that point, direct connections work fine. After it, the overhead of managing each server's auth and policies separately makes a gateway the cheaper option overall.

Tools

Tools are the discrete, callable functions exposed via MCP that let an LLM take actions or fetch data. Each tool has three parts: a name and description so the model knows the tool exists and when to use it, a strict input schema so calls are valid and typed, and a structured output so results are usable without parsing. Examples: "get service entity", "query deployment history", "search runbooks", "create incident".

Tools are what make agents useful rather than just knowledgeable. A model without tools can reason about a problem. A model with the right tools can act on it. In a platform engineering context, the quality of the tools - specifically how precisely their schemas match real workflows - determines most of the difference between agents that work and agents that look good in demos.

At BackstageCon Europe on March 23, 2026, Roadie's Head of Product Sam Nixon shared a number that got people's attention: the agent-to-human interaction ratio on Roadie's platform has reached 100:1 on certain days. One hundred automated actions for every human decision. In Roadie's own usage, the bulk of support requests and on-call alerts are now handled without engineer involvement - though Sam was candid that this is partly a function of Roadie operating its own system end-to-end.

Most enterprise AI deployments aren't producing results like that. Teams have capable models. They've written careful prompts. They've shipped workflows that run perfectly in demos. And then in production - on a real incident at 2am on a Monday - the agent gives them an answer that's technically coherent and completely wrong. The gap is the context, not the model.

What agents are actually reasoning from

When an agent fails in an engineering workflow, the first instinct is to diagnose the model. Swap to a smarter one, improve the prompt, adjust parameters. Sometimes that helps. More often the failure is upstream: the agent was reasoning from thin, stale, or structurally ambiguous input.

A context window full of files is not the same as a context window full of facts.

Most of what I see teams doing with context right now just doesn't work. And it fails in a specific way: they've connected their tools to the agent but haven't built the layer between them. The agent has access to information. It doesn't have authoritative, structured context.

In March 2026, Andy Chen - an engineer at Abnormal Security - published a detailed account of building an enterprise context layer from scratch. The piece is worth reading because it makes a distinction that most vendor messaging elides: retrieval and synthesis are different problems. A retrieval system finds the best-matching document. Synthesis produces the judgment call - which source to trust when three docs contradict each other, whether this service is safe to deploy right now, when to escalate to a human. Current tool stacks conflate the two. They give agents access to documents and hope reasoning handles the gap.

The token budget compounds this. Apideck published benchmarks showing that connecting three standard developer tool servers - GitHub, Slack, and Sentry - consumes 143,000 of Claude's token context window before the agent has processed a single message. 14% of the budget, gone, on tool definitions, assuming you're using the 1M version of Opus. Think about that: you haven't asked a question yet, and you've already spent an eighth of your reasoning budget on overhead. Teams running at 100:1 aren't working around this - they've built a different architecture.

The four motions

On Roadie's platform, the context layer is four operations that work in sequence - what we call the four motions. Each one solves a distinct part of the problem, and skipping any of them shows up in production.

Pull in data

The first motion is integration: repos, deployments, incidents, ownership records, documentation, infrastructure state. Most teams start here and assume the hard work is done. They've connected the sources. The agent has access.

Connecting sources is the easy part. The question is whether the data is fresh enough and trustworthy enough to reason from. An agent querying a context store with deployment data from three weeks ago, or ownership records that haven't been updated since the last reorg, will produce results that look authoritative and are wrong. The context layer needs to know the provenance of each fact: where it came from, when it was last verified, and how to handle it when it conflicts with a different source.

Andy Chen's piece describes this as a source-reconciliation problem. His agent swarm surfaced five principles on its own: architecture claims and status claims belong in different places; there's no universal source of truth; documentation describes the ideal state, not the current state; facts that appear in three independent sources can be trusted; and conflicting information should be documented as a conflict rather than resolved arbitrarily. Those principles hold for any context layer meant to be the substrate agents reason from, regardless of implementation.

Build relationships

This motion is what separates a context layer from a document index. You can have accurate data in separate systems - a service catalog with team ownership, an incident tracker with affected services, a deployment log with what changed - and still be unable to answer the questions that matter under pressure.

At 2am you need to know which team owns the failing service, what changed in the past 24 hours across its dependencies, and who is on call for that component.

Those answers live at the junctions between datasets. Getting there requires a graph, not a catalogue of documents. The relationships - service to team, API to consumer, runbook to incident type, deployment to downstream dependency - have to be explicit, typed, and traversable.

This is the part most early context layer attempts skip. They pull in data correctly and then assume the model can infer relationships from raw text. It can, sometimes. Under time pressure, with contradictory signals, inference is the weakest link. If the relationship isn't in the graph, you're relying on the model to guess - and guesses that present as confident answers are the most expensive kind.

Assemble bundles

This is where the actual engineering happens. An agent doesn't need your entire service graph for every query. It needs the right slice: the topology of affected services, the current ownership chain, the deployment history for the past few hours, the runbooks tagged to this incident type.

Assembling that slice on demand - scoped to the question, progressive in disclosure - is what keeps the token budget sane and the answer accurate. The Apideck benchmarks are a symptom of context that hasn't been assembled. When you surface the full tool manifest upfront, you pay for definitions you won't use. Tiered access - categories first, detail on request - gets you the same information at a fraction of the cost. Apideck's own analysis puts the gap at $3.20 per month for a well-scoped CLI workflow versus $55.20 for naive MCP integration.

Bundle assembly is also where governance lives. Not every agent should have access to every slice of context. Security posture data, compliance records, and personnel information need different access controls than service topology. This is an architecture decision you have to make up front, not a compliance checkbox you add later. Build access controls in from the start, or you'll retrofit them when it's much more expensive to do so.

Agents consume and contribute

This is the motion most deployments haven't reached yet. It's also where the compounding value shows up.

An agent that successfully runs a runbook, investigates an alert, or assesses a deployment has produced new context: decisions made, state at the time, actions taken, what worked. The trail is evidence. If the context layer captures what agents do, the next agent in the workflow starts from a richer position. If it doesn't, every invocation starts cold.

The temptation is to add that feedback loop once the basic flows are working. That's reasonable. But the teams at 100:1 got there partly because they built it early. The context layer improves with every agent run. The graph gets richer. The bundles get more accurate. Agents that contributed to the graph last week make agents this week faster and more reliable.

Sam Nixon laid out at BackstageCon what an agent-ready context layer actually requires: a comprehensive, fresh graph of your software topology; that graph enhanced with relationships and additional context outside the catalog, in a format agents can consume; and the actual tools to act on that information. The four motions are the operational shape of those three requirements. By the time agents are contributing back to the graph, all three are in play - and the system compounds with every run.

This is a platform engineering problem

The phrase "context engineering" has arrived as a job title. There's real work here - the kind that doesn't happen without someone owning it.

But the teams positioned to do this well aren't starting from scratch. The platform engineering team that built the service catalog, scored compliance, made deployments observable, and kept ownership records current already owns most of this substrate. They know which sources to trust, which relationships exist between systems, and what "current state" means in their environment. The hard part of the context layer is the organisational knowledge that feeds it, not the technology.

The homegrown version - a thousand lines of Python, a GitHub monorepo of markdown, an agent swarm crawling internal sources - can get you to a proof of concept quickly. Chen's piece is a genuinely useful account of how far that approach can go. But the version that survives contact with compliance requirements, multi-tenant access controls, and production scale looks different. Access control, auditability, multi-tenancy, reading from production systems without causing incidents: these are what make a context layer something an organisation can actually operate. They're also exactly what gets hand-waved in practitioner write-ups and bites you at scale.

Andy Chen's framing from his ECL piece applies here: the enterprise context layer is "closer to DevOps than to Salesforce." A practice, not a purchase. You build the discipline - the ingestion pipelines, the relationship mappings, the bundle definitions, the access controls - and then you maintain it. The four motions are the shape of that maintenance.

Roadie's platform implements this architecture. The context store holds your service graph, your operational data, and the relationships between them. The MCP interface handles progressive disclosure so agents get scoped context rather than a full dump. Agent contributions feed back into the graph. Access controls are first-class from day one, not a retrofit.

The 100:1 ratio comes from better context infrastructure, not better models. At a hundred agent actions for every human decision, the quality of those actions is determined almost entirely by what's in the context window when the agent starts reasoning.

The teams still debugging agent failures are usually debugging the wrong thing. A context layer that provides authoritative, structured knowledge - ownership, relationships, provenance, agent history - is what separates 100:1 from teams still tuning prompts. Power them with facts, not guesses. If you want to see how Roadie builds this for your engineering team, request a demo or start a free trial.

Consider a practical scenario: an engineer asks their assistant to refactor a synchronous downstream call in payments-service into retry logic with exponential backoff. The suggestion looks clean. The tests pass. The PR gets approved. But the assistant couldn't see that payments-service is already running at 99.91% against a 99.9% SLO, its p99 latency sits at 450ms against a 500ms budget, and the Fintech squad placed a code freeze on it 48 hours ago ahead of a critical release. The retry logic hammers a downstream service that's already degraded, pushes latency past the budget, and burns the remaining error budget in under an hour.

The model produced syntactically correct, architecturally reasonable code. Every piece of information that would have changed its output lived in the service catalog, the observability platform, and the on-call tooling, none of which it could reach. The problem is a missing operational data layer between your engineering metadata and the AI's context window.

The Four Categories of Missing Platform Context

The industry conversation about context for AI coding assistants has focused almost entirely on codebase coverage: how many files can the retriever pull, how good is the semantic index, and does it understand cross-repo dependencies? Augment Code's treatment of the context delta and Sourcegraph's breakdown of keyword, embedding, and graph-based retrievers describe this codebase-level layer well. While this is important, it's actually the easier half of the problem. The missing context that actually costs engineering teams time and trust falls into four categories.

Service ownership

Pete Hodgson's Constraint-Context matrix analysis highlights that AI assistants carry no awareness of the constraints that only humans in your org hold. If payments-service is owned by the Fintech squad, and they have a code freeze in effect, any PR against it requires their explicit review and sign-off. None of this exists in the repository. It lives in spec.owner in your service catalog, team topology tooling, or PR process documentation. The assistant treats the code as ownerless.

SLOs and reliability targets

Adding a synchronous downstream call to a service running a 99.95% availability SLO introduces a tail-latency risk that the model has zero signal for. It can read your retry configuration, but it can't read the Datadog SLO tracking data that tells you you've consumed 78% of this month's error budget. That signal lives in your observability platform and is only surfaced as structured metadata if you've built the bridge from observability to catalog.

Deployment history and cadence

A major refactor on a service deployed three times a day carries entirely different risk from the same refactor on a quarterly-release service. The EntityArgoCDHistoryCard exposes per-entity deployment history (such as revisions, timestamps, commit info, and sync status) at the catalog level. An AI assistant working from the file buffer has no access to whether this service last deployed four hours ago or four months ago, and that cadence signal is material to any reasonable risk assessment.

Incident history

If payments-service generated three P1 incidents in the last 60 days because of connection pool exhaustion under load, and the assistant suggests a pattern that increases connection pressure (spawning additional database clients per request, for instance), that's a regression. Post-mortem context is organizational memory. The model can't infer it from code because it lives in incident reports and runbooks.

Why Individual Workarounds Don't Scale

Teams are already responding to this missing context with rational, locally effective solutions. Google Cloud's AI coding assistant best practices guide recommends GEMINI.md files that give the assistant project-specific context, including what framework you're using, what conventions to follow, and what patterns to avoid. That's a sensible individual workaround for a team of five.

The problem is the maintenance surface. A GEMINI.md file written this morning doesn't reflect the deployment that happened this afternoon or the SLO breach that opened an hour ago. It's a static document maintained by humans in a system that changes continuously. If you have 50 engineers, 200 services, and 10 teams, accurate context file maintenance becomes a full-time job, and there’s little incentive to keep any individual file current because the cost of staleness is invisible until something breaks.

The queryability problem is separate. An AI retrieval pipeline can't ask a GEMINI.md "which services owned by the Payments team have open P1s?" and get a structured answer it can reason over. The file format is designed for human reading, so structured cross-service queries require a different architecture entirely. Because each file lives with its service, the assistant working on Service A has no access to Service B's SLO status even when Service A calls Service B synchronously.

With one hundred engineers and five hundred services, these files are perpetually stale and inconsistently structured, giving teams a false sense of coverage. The context that the AI receives reflects the state of a service as understood by whoever last edited the file, which is rarely the person who owns the service today.

Platform Context Engineering: The Architectural Shift

Engineering organizations need to identify the authoritative, structured, machine-readable data layer that represents operational reality, and confirm if it has an API surface that AI retrieval pipelines can query. That reframe shifts the problem from prompt engineering (improving how the AI is instructed) to context infrastructure (improving what structured data the AI can retrieve).

Platform-level context engineering requires entities (services, teams, APIs, infrastructure components), relationships (ownership, dependency, responsibility), and operational signals (SLO status, deployment cadence, incident frequency). All of this information needs to be kept current through automation and exposed through a queryable API. Machine-readable, automatically updated data is the baseline requirement. The goal is a structured entity graph that works for both human engineers browsing the catalog and AI agents querying it programmatically.

This is an important distinction for AI retrieval specifically because a pipeline can only return what's structured and indexed. An LLM can ingest an unstructured text file, but it cannot reliably extract "which of the services that I depend on are currently breaching their error budget" from a collection of Confluence pages. Structured entity data with relationships and operational signals enables precise retrieval, which can return specific details like "Payments team, code freeze active, SLO at 99.91%, last deployed 4 hours ago" as a structured response that the retrieval pipeline can return directly.

Closing the Delta: From Entity Graph to AI Context

Roadie's engineering context platform implements this architecture as a production-ready retrieval layer. The catalog reflects the actual ownership state of the repository without manual maintenance, because each entity carries structured ownership via spec.owner, which is populated automatically through the CODEOWNERS integration. Tech Insights surfaces SLO data as structured scorecard metadata: add a datadoghq.com/slo_tag annotation to your catalog-info.yaml, point the built-in Datadog Data Source at your Datadog instance, and SLO compliance data becomes a queryable fact on the entity, accessible from the same pipeline that serves the AI. Each of these steps still asks service owners to maintain one annotation. That's unavoidable, but it's a big improvement over GEMINI.md files. The right long-term approach is to ingest this association directly from the source systems, so the annotation burden eventually disappears.

The data flow from operational signals to the AI context runs through this architecture:

Roadie's RAG AI Plugin makes this entity graph AI-consumable. It indexes catalog entities, TechDocs, OpenAPI specs, and Tech Insights scorecard data as embeddings, stored in PostgreSQL with the pgVector extension (enabled via CREATE EXTENSION IF NOT EXISTS vector). The LLM backend supports AWS Bedrock and OpenAI. The retrieval pipeline is explicitly extensible: teams can add incident post-mortem data, on-call schedules, or architecture decision records as additional indexed sources.

An assistant pulling from this pipeline can receive: "payments-service, owner: fintech-squad, SLO target: 99.95%, SLO status: 99.91% this month (78% error budget consumed), last deployed: 4 hours ago via Argo CD revision abc123f, P1 incidents last 60 days: 3 (connection pool exhaustion)." The assistant now has access to the same judgment inputs that a careful engineer would consult.

For Roadie's cloud-hosted platform, the AI Assistant is configured via Administration > Settings > AI Assistant. For self-hosted Backstage instances, the plugin is available as the @roadiehq/rag-ai frontend package with the corresponding backend plugin.

Three steps to better retrieval context

You don’t need a complete overhaul to improve your retrieval context. There are three steps you can take this week to give your AI retrieval pipeline ownership, reliability, and deployment recency signals.

First, audit which services in your catalog have spec.owner set. Entities missing that field are invisible to any ownership-aware retrieval. Use CODEOWNERS auto-assignment to populate ownership systematically. The catalog can derive it from your existing file ownership structure, so you're not asking teams to maintain a new artifact.

Second, if you're tracking SLOs in Datadog, enable the Tech Insights Datadog Data Source. Navigate to the Tech Insights section in Roadie, add the built-in Datadog Data Source, and annotate your catalog-info.yaml files with datadoghq.com/slo_tag. Your SLO compliance data becomes a structured, queryable fact on each entity, updated automatically as Datadog updates.

Third, if you're using Argo CD, install the Argo CD plugin and add EntityArgoCDHistoryCard to your entity page for each service. Link components to their Argo CD applications via the argocd/app-name annotation. You get per-entity deployment history, sync status, and revision data, all queryable by any retrieval pipeline you build on top.

All three steps change what AI agents receive without changing how your engineers work with those agents.

See what structured engineering context looks like for your team and your AI agents. Explore Roadie

The suggestion is wrong in three distinct ways. The intermediate layer being bypassed is the team's PCI-scope data access abstraction, required by a compliance contract with the platform team. The downstream service the new code queries belongs to a squad that accepts calls exclusively through an async queue, documented in an API contract that lives in Confluence. The logging Cursor outputs plain strings into a service that has enforced structured JSON since 2023, codified in an architectural decision record written by an engineer who left the company 18 months ago.

Cursor processed the open files, a handful of related functions, and the natural language problem description. The compliance boundary, the ownership edge, the API contract, and the logging convention don't live in source files.

This is the core failure mode of AI coding assistance in production engineering orgs. Outputs that are locally correct and globally invalid because the model operated without system-level context. Expanding the token window doesn't address this, and feeding the AI more source code doesn't address it either. What's absent is structured, queryable metadata about how services relate, who owns what, and what constraints govern each component.

Context Engineering Is Not Prompt Engineering

Prompt engineering occupies the interaction layer of a larger system. It addresses how you word an instruction to improve a model response within a single exchange. Context engineering addresses the architecture of the full information environment that the model operates in, covering retrieval mechanisms, memory systems, structured metadata, entity relationship models, state management, and output formatting constraints.

The practical distinction shows up where the work happens. Improving your prompt is a single-file edit. Building context infrastructure requires decisions about data schemas, entity relationship models, API surface areas, retrieval strategies, and update mechanisms. Prompt engineering is one layer of a three-layer stack, and context engineering designs the entire stack:

1. Structured data layer: service metadata, ownership graphs, dependency declarations, API contracts, SLO definitions, incident histories, and architectural decision records.
2. Retrieval layer: how agents and developer tools access that data at query time, via catalog API queries, entity graph traversals, or vector store lookups.
3. Interaction layer: how retrieved context gets injected into the model window at inference time, via prompt templates, system messages, few-shot examples, and output schema constraints.

Most discourse on context engineering addresses layer three. The architectural decisions that determine whether AI tools actually perform in production engineering orgs happen at layer one.

A RAG pipeline over Confluence documentation retrieves text chunks ranked by semantic similarity, and its ceiling is determined entirely by the quality, structure, and currency of the underlying documents. A typed entity graph with declared schemas delivers deterministic query results. When an AI agent asks which team owns payment-processor and what its declared dependencies are, a typed API response gives the agent something it can act on without hedging. Semantic search over unstructured documentation works well for discovery tasks. For AI agents making operational decisions at runtime (routing escalations, scoping incident blast radius, and verifying SLO compliance before a deployment), the two retrieval mechanisms belong to categorically different reliability classes. Build your data layer to support both, with explicit clarity about which class of problem each handles.

What Structured Engineering Context Actually Looks Like

The entities that constitute useful engineering context at the operational layer are services and their declared owners, dependencies with SLO targets, deployment environment definitions, API contracts, past incidents and their resolutions, and architectural decision records. That information exists in most engineering orgs today, but it’s distributed across YAML configs, Confluence pages, Google Docs, Slack channels, and tribal knowledge.

The Backstage entity descriptor format provides a practical starting schema for the structured data layer. A well-formed Component entity gives an AI agent something it can query deterministically:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-processor
  description: Handles charge authorization and settlement for the checkout flow
  annotations:
    roadie.io/oncall-runbook: "https://wiki.internal/runbooks/payment-processor"
    roadie.io/slo-target: "p99 < 200ms, availability > 99.95%"
    pagerduty.com/integration-key: "abc123xyz"
spec:
  type: service
  lifecycle: production
  owner: group:payments-team
  system: checkout
  dependsOn:
    - component:ledger-service
    - component:fraud-detection
    - resource:payments-db
  providesApis:
    - payment-processor-api

An AI agent triaging a latency incident on payment-processor can resolve the owner to payments-team in one API call, pull the SLO target to assess whether the current p99 violates it, identify ledger-service and fraud-detection as declared dependencies to check, and surface the runbook URL. A RAG pipeline over documentation pages might surface similar information after several retrieval rounds, but a typed entity query retrieves it deterministically in milliseconds, with no ambiguity about which document version applies.

The architectural significance extends beyond any single entity. When ledger-service also has a well-formed descriptor, you can traverse the dependency graph to find which teams own the services that payment-processor depends on, which of those are currently below their SLO targets, and which dependencies changed in the last 24 hours. Those traversals require typed entity relationships with declared schemas, where each edge carries semantic meaning, and each node exposes a queryable API.

How Context Infrastructure Reduces Context Switching in Practice

Incidents are where the cost of manual context assembly becomes most visible. The engineer receiving a PagerDuty alert at 2am has no buffer for multi-tool lookups.

Without context infrastructure, the engineer opens the alerting service in their IDE, switches to Confluence for the runbook, searches Slack for recent incident discussions, and then checks GitHub blame for the last deployer. At that point, they ask Copilot for help. Copilot has access to the open files and the natural language question. It answers without any of the operational context that the engineer spent 25 minutes assembling.

With a queryable context layer, the same engineer issues one API call against a structured entity graph. The response returns the service owner, the last deployment timestamp, the linked runbook, the current SLO status, and the full dependency list. An AI agent receives that same response and uses it to constrain its output. It can flag that a proposed change affects fraud-detection, which is owned by a separate team and is currently operating at 99.96% availability against a 99.95% target. The suggestion is system-aware because the agent's context is system-aware.

The design decision embedded here matters. If the AI agent fetches the context and routes it silently into the model window while the developer sees only the final suggestion, you've solved the AI's information problem but preserved the developer's cognitive debt problem. The right implementation surfaces the resolved metadata as part of the engineer's workflow, so the structured context is visible to both the person and the agent drawing on it. Engineers who can see what their AI agent knows maintain comprehension, and that understanding deepens over time.

Context infrastructure also eliminates the manual assembly step that causes context switching in the first place. The developer stops acting as the integration layer between their tools, stitching together information from runbooks, Slack, and GitHub before they can form a useful question.

Building Context Infrastructure: What Engineering Teams Should Prioritize

The sequence is important because each layer depends on the one below it. Teams that ship AI features on top of incomplete metadata then wonder why the productivity gains don't materialize are building the wrong way around.

Ship in this order, and don't advance to the next step until each one is complete:

1. Declare entity ownership across your entire service catalog. Every service, library, data pipeline, and API must have a declared owner field pointing to a team or group entity. This is the minimum viable signal for routing, escalation, and impact scoping. AI agents querying your catalog with no ownership data will produce unreliable escalation paths and incorrect blast radius assessments.

2. Attach structured operational metadata to those entities. SLO targets, runbook URLs, deployment environment declarations, and dependency lists all belong in schema-defined fields. Free-text description fields are retrieval fodder for RAG pipelines. Structured annotation fields are what AI agents query programmatically. You need both, and they serve different purposes.

3. Expose your catalog via an API that agents can call. A service catalog that only renders in a browser UI is inaccessible to AI agents and automation tooling. A REST or GraphQL catalog API is the connective tissue between your structured metadata layer and any AI capability you want to build on top of it. Without programmatic access, you have a portal, and portals don't compose with agent workflows.

4. Run a coverage audit before adding AI features. If more than 30% of your catalog entities are missing owner, system, dependsOn, or a custom operational annotation (runbook URL or SLO target), context-aware AI features will underdeliver.

Roadie's engineering context platform implements this architecture as a SaaS product built on open standards. Its entity graph exposes typed, relationship-aware service metadata via REST API, queryable by both human engineers and AI agents against the same underlying data layer. Catalog v2 is explicitly designed for this pattern, providing the coverage and programmatic access that make AI-layer features reliable at scale. Teams running custom Backstage installations get the UI, while teams running Roadie get the queryable context infrastructure that those AI features actually require.

Start Here: Run a Metadata Completeness Audit on Your Service Catalog Today

The single most useful action you can take before evaluating any AI developer tooling is a metadata completeness audit on your existing service catalog. Pull every Component entity from your catalog and check four fields: owner, system, dependsOn, and at least one custom operational annotation (runbook URL or SLO target).

For teams running a Backstage-compatible catalog API, the query is straightforward:

GET /api/catalog/entities?filter=kind=Component

Parse the response for entities where spec.owner is empty, spec.dependsOn is an empty array or absent, spec.system is unset, or metadata.annotations contains none of your defined operational fields. Flag any entity missing two or more of those fields as a coverage priority.

If more than 30% of your catalog entities fail that check, fixing metadata coverage will deliver more AI productivity gain than any retrieval or prompt optimization effort. A missing owner field alone means AI agents have no reliable signal for routing, escalation, or impact scoping. A service with no dependsOn declarations means dependency traversal returns an empty graph, making blast radius assessment impossible. If your catalog has above 70% coverage on all four fields, you have the foundation to build context-aware AI features that will actually perform.

Metadata coverage is an engineering problem with an engineering solution: schema enforcement, catalog-as-code with required fields, and ownership reviews in your service onboarding checklist. Ship those fixes, then revisit your AI tooling configuration with a complete data layer underneath it.

Give your engineering team and AI agents the structured context they need to actually ship faster. See how Roadie's engineering context platform works and request a demo.

Atlan's 2026 analysis of enterprise AI failures found that context failures (missing, stale, or conflicting information) are one of the leading causes of production AI breakdowns. Treating retrieval augmented generation as a complete context engineering system is the architectural decision that produces these failures.

Context engineering involves intentionally designing every slot in an LLM's context window. RAG is one retrieval primitive within that system, well-suited to semantic document lookup. An agent that draws its entire context from a vector store has wired up one slot out of six, and the remaining five require source systems with entirely different query mechanisms.

What is RAG?

RAG has a precise mechanical definition. At query time, you embed the user's input, run an approximate nearest-neighbor search over a vector index (Pinecone, Weaviate, or pgvector), retrieve the top-k chunks, and inject them into the prompt. The model generates a response grounded in those retrieved passages.

The pattern works well for unstructured document retrieval at scale. If an engineer asks "what does our authentication service's token expiry policy say?", an embedding over documentation chunks surfaces the relevant passage reliably. Production RAG implementations often add considerable sophistication on top of this core pattern. BM25 sparse retrieval runs alongside dense vectors, reciprocal rank fusion merges ranked result sets, and multi-stage rerankers score passages for relevance before final injection. These techniques improve retrieval precision, but retrieval precision only matters if retrieval is the right tool.

This retrieval primitive isn’t suitable for ownership data. "Which team owns payment-service?" is a lookup against a structured entity record that links a service node to a team node in a typed graph. Embedding the query and running cosine similarity over chunked service documentation may surface a passage that mentions the team, if someone wrote that team name into a document at some point. A structured entity graph returns the authoritative, current record by traversing typed relationships directly.

What is Context Engineering?

Treating context engineering as a synonym for RAG collapses a complex architecture into a single retrieval pattern. Context engineering is the discipline of intentionally deciding what occupies every slot in an LLM's context window: what enters, in what format, from what source, updated on what cadence, and subject to what size constraints. A production context payload is assembled from six distinct source categories, each with its own query mechanism and staleness profile:

1. System prompt: behavioral constraints, output format requirements, tool definitions, and role instructions. Static for a given deployment, updated by engineers.
2. Structured entity metadata: service ownership, dependency graphs, API configurations, on-call schedules, and tier assignments. Authoritative and current, queried from an entity graph.
3. Conversation history: prior turns in the current session, typically compressed with selective summarization to fit within token budgets.
4. Retrieved documents via RAG: runbooks, architecture docs, policy references, and knowledge base content. Semantically retrieved from a vector index.
5. Tool call outputs: live data fetched at inference time, including API responses, database query results, alert feeds, and CI/CD records.
6. Agent working memory: accumulated reasoning state across turns, including tested hypotheses, ruled-out causes, and timeline observations.

Each source category produces outputs with different reliability characteristics. A system prompt updated by a senior engineer carries different authority than a documentation chunk that may reflect last year's architecture. An entity graph record updated when a team transfers service ownership is more authoritative for ownership questions than any passage in a runbook.

The architectural decision in context engineering is matching each question type to the right source, then assembling the payload efficiently within the model's token budget. SingleStore's context engineering analysis describes this as the difference between a system that retrieves documents and one that curates, structures, and evolves the full information payload.

The Taxonomy: Where RAG Sits in the Context Engineering Stack

In a production context payload, RAG occupies a single slot among six, each with a different retrieval mechanism. For example, an incident triage agent assembles its payload from entity_metadata queried from an entity graph, retrieved_docs fetched from a Pinecone index over the runbook repository, tool_outputs pulled from live alert APIs, and working_memory maintained across turns in LangChain state. RAG accounts for the retrieved_docs field:

{
  "system_prompt": "You are an incident triage assistant. Treat entity_metadata as authoritative for ownership and dependency data...",
  "entity_metadata": {
    "service": "payment-service",
    "owner": "team-payments-infra",
    "tier": 1,
    "dependencies": ["auth-service", "postgres-primary", "redis-cache"],
    "on_call": "alice@company.com",
    "last_deploy": "2025-01-14T09:22:00Z"
  },
  "conversation_history": [
    {"role": "user", "content": "payment-service is throwing 500s on checkout"},
    {"role": "assistant", "content": "Alert confirms elevated error rate since 09:41 UTC..."}
  ],
  "retrieved_docs": [
    {"source": "runbook-payment-service-v3.md", "chunk": "For elevated 5xx rates, check connection pool exhaustion first..."}
  ],
  "tool_outputs": {
    "current_alerts": {"metric": "error_rate", "value": 0.34, "threshold": 0.05},
    "recent_deploys": [{"sha": "a3f2c1", "deployed_at": "2025-01-14T09:22:00Z"}]
  },
  "working_memory": {
    "hypotheses_tested": ["connection_pool", "auth_service_upstream"],
    "timeline": "Error rate spiked at 09:41, 19 minutes after the a3f2c1 deploy"
  }
}

A 2025 survey on LLM-based agents from researchers at NUS, Renmin University, and Fudan University formally positions RAG as one component within the broader agent memory and context engineering landscape, intersecting with agent memory, LLM internal memory, and the full context engineering system as a whole.

What Breaks When You Treat RAG as Context Engineering

When a context pipeline routes all queries through a vector store, four failure patterns appear consistently in production.

Stale structured data

A RAG pipeline retrieves documentation that reflects what was true when it was written, not what is operationally true now. Reranking operates on retrieval precision over the indexed content, and stale source data produces wrong answers regardless of ranking quality. Ownership data requires a queryable source that updates when ownership changes, and documentation repositories are generally not kept in sync at that granularity.

Entity relationship traversal

“Which services depend on this database?” routed through a vector store returns passages where an engineer happened to write about the database's consumers. That result set is incomplete if documentation coverage is patchy, outdated if it wasn't maintained, and missing entirely if the dependency was never documented. This question requires a traversal over structured dependency metadata, which a typed entity layer can answer deterministically by following dependency edges directly.

Cross-turn reasoning state

In multi‑step workflows like incident response, re‑retrieving documents on each turn causes previously processed context to crowd out the agent’s evolving incident timeline. Hypotheses, checks, and temporal observations must be maintained in a persistent working memory layer, managed by the agent’s state system (for example, LangChain's ConversationSummaryBufferMemory or an equivalent custom store).

Context window inflation

A naive strategy that injects the top‑20 retrieved passages can consume tens of thousands of tokens on documentation that’s irrelevant to the current reasoning step. Because cost per agent invocation scales with context size, bloated payloads add latency and expense without improving output reliability.

What a Production Context Engineering Stack Actually Looks Like

An AI agent handling incident triage puts all six context slots into production simultaneously, each drawing from a distinct source system matched to its query requirements.

The system prompt carries behavioral constraints, tool definitions, and output format requirements. These load at agent initialization in LangChain or LlamaIndex and remain static across turns.

The structured entity metadata slot is populated by querying an entity graph directly. For the incident agent, that query specifies service=payment-service and returns owner, tier, current on-call, dependencies, and last deploy time. Because this data comes from the same graph that engineers use operationally, the agent inherits its authority and update semantics. Roadie's context engineering platform provides that graph: a queryable record of services, ownership, dependency relationships, and operational metadata that engineers and agents query from the same source. The graph updates when teams update their service records, so the agent always operates on current data. In practice the graph is fed from the operational systems themselves - Git, cloud accounts, PagerDuty, Kubernetes - so ownership and dependency changes surface without a developer remembering to touch a metadata file.

The conversation history slot carries a rolling summary of the current incident session, managed by the agent's memory layer. The retrieved_docs slot runs RAG over a hybrid Pinecone or Weaviate index over the runbook repository, using dense embedding similarity for semantic matching and BM25 for exact term hits, merged via reciprocal rank fusion. The tool_outputs slot carries live data fetched at inference time from PagerDuty, Datadog, and the CI/CD system.

Audit Your Agent's Context Sources Today

Gartner predicts context engineering will appear in 80% of enterprise AI tools by 2028. To prepare, audit your agent’s context pipeline. For each context slot populated on a turn, answer four questions: what is its authoritative source, such as documentation, an entity graph, a live API, or in memory state? How stale can it be before causing errors? Is the data structured or unstructured? Is it retrieved fresh each turn or carried across turns?

Agents that rely solely on vector databases can accurately answer “what does the documentation say,” but fail on questions about current service ownership, dependencies, or multi-step reasoning state. Those require explicit wiring of additional slots, including entity metadata, tool outputs, and working memory, sourced from systems designed for authority and freshness.

The highest value addition is usually structured entity metadata, including service ownership, dependency graphs, on-call assignments, and API configurations. These change faster than documentation can keep up. Roadie exposes this metadata via a queryable API, making it straightforward to integrate into LangChain or LlamaIndex agents as an additional tool.

Production-grade agents treat context assembly as infrastructure rather than retrieval, matching each question type to a source with the right authority, freshness, and query semantics. Retrieval precision alone is not enough.

Explore Roadie's context engineering platform.

The model is operating without grounding in the organization’s actual engineering reality. It doesn’t know which team owns which service, which APIs are live, or which runbooks are current. In that environment, outputs default to general training patterns rather than verified organizational state, and the most likely outcome is a hallucination. You can address this using context engineering, which many enterprise AI initiatives skip entirely.

What Context Engineering Actually Is

Context engineering is the practice of curating, structuring, governing, and delivering domain-specific information to an LLM across the full request lifecycle. It covers data collection and normalization, semantic modeling and entity resolution, retrieval strategy design, freshness guarantees, and output validation. Each of those layers determines whether the model has accurate, current, and scoped information to work with at inference time.

Context engineering is closely related to, but distinct from, the two other disciplines that shape LLM behavior. Prompt engineering manages the instructions and format directives inside a request. Fine-tuning adjusts model weights using training examples to shift a model's behavioral defaults. Context engineering constructs and delivers the factual content that fills the context window. All three operate at different points in the system, and OpenAI's prompt engineering guidance makes clear that in-context instruction can only do so much when the information being processed is itself inaccurate or absent. A production LLM feature requires all three to be addressed as separate, independent problems.

In practice, teams will often try fine-tuning to resolve an issue when context engineering is the right tool, and the substitution carries real operational cost. Fine-tuning encodes knowledge into weights at training time, so retraining is required every time your organizational data changes, which could be as often as daily. Context engineering delivers that information on demand, moving through four architectural stages, from raw data to validated output:

1. Data collection and normalization across source systems
2. Semantic modeling and entity resolution (resolving "payments-api", "payments_api", and "Payments API" to a single canonical entity)
3. Retrieval and delivery strategy (hybrid search, re-ranking, query rewriting, and knowledge graph traversal)
4. Context validation and eval loop (scoring outputs against known-good answers before and after any retrieval change)

The Context Debt You're Already Accumulating

A 2015 NeurIPS paper found that in production ML systems, the non-model work (e.g., data pipelines, feature engineering, and monitoring) tends to dominate total engineering cost. Context engineering handles the equivalent complexity for AI applications. Leave it unaddressed, and you accrue context debt.

Three failure modes tend to compound as LLM features evolve. Context is often embedded directly into system prompts, where it is rarely versioned or auditable. Retrieval introduces another source of error when vector search returns content that is semantically similar to a query while being incorrect for the specific service, team, or incident involved, since similarity scoring and factual correctness are independent properties and optimization for one does not guarantee the other. Outdated grounding data adds further divergence, as documentation that reflects earlier architectures continues to be retrieved and treated as current.

Each LLM feature built on an unstructured context layer adds another load-bearing workaround to an increasingly brittle foundation. Over time, unresolved context issues increase the cost of every change to retrieval, evaluation, or schema design. Ad‑hoc fixes create undocumented dependencies between prompts, data sources, and retrievers, and resolving conflicts between them often requires replacing the context layer across multiple features at once.

Why Enterprise Data Is the Hardest Part of This Problem

Every retrieval-augmented generation (RAG) tutorial assumes a clean, queryable dataset, but enterprise data rarely is. A mid-sized engineering organization typically has service metadata distributed across GitHub (repository structure, CI config), Jira (ownership assigned inconsistently by team), PagerDuty (on-call rotation with no canonical service identifier), Confluence (runbooks at varying levels of staleness), and one or more custom CMDBs that were accurate two platform migrations ago. These systems are unlikely to have a consistent schema. Ownership modeling defaults to ad hoc assignments with no cross-tool canonical identity. Lineage tracking (who last verified this record and when) exists in almost no tooling by default.

The gap between "how do I build a retriever?" and "do I have anything worth retrieving?" is where most enterprise AI initiatives stall. You’ll answer the first question during the first sprint, but the second question won’t surface until the first production failure.

The retrieval problem gets more complicated when you factor in data type variance. Service ownership and dependency metadata are structured, entity-resolved, and queryable if your catalog enforces a schema. Incident postmortems are semi-structured, time-sensitive, and specific to a point-in-time system state that may no longer apply. Runbooks are unstructured prose, version-critical, and frequently orphaned when the system they document changes. A single RAG pipeline optimized for one of these types will return misleading results for the other two.

RAG Is One Layer in the Context Engineering Stack

RAG handles retrieval, and context engineering determines what it has to work with and how trustworthy that material is. The full context engineering stack for an engineering AI tool has four layers. Schema-enforced software catalogs provide a structured, canonical, machine-queryable foundation, with service records that carry verified owners, API contracts, dependency relationships, and linked runbooks. Knowledge graphs (Neo4j, Apache Jena) extend that foundation with relationship-aware traversal, enabling multi-hop reasoning across entities that flat vector search cannot replicate. Agentic context gathering adds dynamic retrieval triggered by user intent rather than static query patterns. Interaction history provides user-scoped persistence for stateful context across sessions.

Engineers who've moved to models with 1M-token context windows sometimes assume retrieval precision becomes a solved problem at that scale. Research has shown that models consistently miss or deprioritize information buried in the middle of a large input, a phenomenon known as the lost in the middle problem. Flooding a context window with loosely relevant chunks degrades output quality regardless of window size, so retrieval precision matters at any scale.

Context Infrastructure Is a Prerequisite

Context infrastructure requires design decisions that feed directly into your data model, your retrieval architecture, your eval criteria, and your schema enforcement strategy. Those decisions can't be made as patches after an LLM feature is already running in production. The context layer has to be designed, built, and validated before the first LLM call reaches a user. Retrofitting a context layer after an LLM feature is live requires changing data models, retrieval behavior, and evaluation logic while users are already depending on the system’s outputs.

A well-maintained software catalog that enforces ownership metadata, keeps documentation versioned and linked against specific services, and tracks dependency relationships and API contracts already forms the critical foundation layer for a context infrastructure layer for engineering AI tools. When the catalog is built to the right standard, it and the context infrastructure for AI are the same artifact.

Roadie provides structured, production‑grade engineering context used by both human engineers and AI agents. Service records capture verified ownership, runbooks, API specifications, and dependency relationships in a schema‑enforced and machine‑queryable form. This context is often surfaced through a Backstage catalog, though the same pattern applies to any structured engineering metadata source. Queries against structured systems return resolved entities with verified ownership, while queries against unstructured wikis return the highest‑scoring text fragment with no guarantees of accuracy or freshness.

Run a Context Readiness Audit Before Your Next AI Feature Ships

When context infrastructure is weak, failures tend to show up after deployment. A readiness audit is how you surface those risks before they become production incidents. Before writing a single line of LLM integration code for the next internal AI feature, ask the following questions about every data source it will consume. If the answer to any of these is no, you’re not ready to start development.

1. Can every entity the LLM might reference (service, team, API endpoint, incident) be resolved to a single canonical, schema-enforced record? If the same service has three names across four systems, the model will invent a fifth.

2. Does every entity record carry ownership metadata with a verified-by date? If no one is accountable for keeping a record accurate, treat the record as unverified until proven otherwise.

3. Has the retrieval mechanism been tested against adversarial and edge-case queries, specifically queries where semantic similarity would surface the wrong result? Happy-path retrieval tests do not validate production behavior.

4. Is there an eval loop that scores LLM outputs against a set of known-good answers before and after any context change? LangSmith provides the tracing and evaluation infrastructure for this kind of continuous context validation. Shipping a change to your retrieval pipeline without an eval loop leaves you with no signal on whether the change improved or degraded output quality.

5. Is the context layer versioned and observable? Can you trace exactly which context payload was passed for any given model output? OpenTelemetry instrumentation on your context assembly pipeline gives you the observability data you need to debug a hallucination with precision rather than guesswork.

Five yes answers mean your context layer is ready to support an LLM feature. Fewer than five means you have unresolved architectural work to complete first. The audit takes an afternoon. The alternative is weeks of post-deployment firefighting over outputs that were structurally guaranteed to be wrong before the first user ever touched the feature.

See how Roadie's context infrastructure makes AI features in engineering workflows actually work. Book a demo

This constraint defines where prompt engineering ends and context engineering begins. Prompts control how a model behaves, and context engineering determines what information actually reaches the context window when the model is called.

What is Prompt Engineering?

Prompt engineering covers everything within the instruction itself, such as the role specification, task description, output format, and behavioral constraints. It uses several well-understood techniques, including few-shot examples to calibrate output style, chain-of-thought decomposition to externalize reasoning, JSON mode to constrain structure, and negative instructions to prevent specific failure patterns.

If you provide a stable, well-scoped problem, prompt engineering can produce precise and consistent outputs for tasks where the input is predictable, the task is bounded, and the instruction is the primary variable. Some examples include classifying a support ticket, extracting fields from a contract, or generating a unit test from a function signature. These techniques assume the required information is already present in the context window.

However, the limitations of this approach become clear quickly in practice. A service dependency lookup agent can have a carefully crafted system prompt that is role‑specified, chain‑of‑thought enabled, and output schema constrained, but still return incorrect results when it runs without the relevant entity metadata:

User: Which services does payments-service depend on?
Agent: The payments-service typically depends on fraud-detection,
       transaction-gateway, and ledger-store. [hallucinated]

If you inject the actual entity metadata from your catalog, an identical prompt with the relevant context produces a correct, grounded answer instead of a hallucinated one:

User: Which services does payments-service depend on?
Agent: Based on the catalog data, payments-service depends on
       fraud-detection-service (v2.1), ledger-api (v1.4).
       [correct, sourced from injected context]

Nothing about the prompt changed. The improvement came entirely from what was loaded into the context window, which is exactly what context engineering controls.

What is Context Engineering?

Context engineering is about designing, populating, and optimizing everything that enters the context window before each LLM call. It’s a runtime system that assembles context dynamically for each invocation.

LangChain frames the scope across four operations.

1. “Write” defines what state to persist across turns, such as conversation summaries, tool outputs, and agent memory.
2. “Select” determines what to retrieve for a given call, pulling only the relevant subset of the knowledge base.
3. “Compress” reduces token volume without discarding signal, using techniques like summarization, LLMLingua-style compression, and chunk deduplication.
4. “Isolate” decides what to offload to sub-agents rather than placing everything in a single context window, which matters for multi-step tasks that would otherwise exhaust the budget.

Scope is where prompt engineering and context engineering diverge most clearly. A prompt is a static artifact that a developer writes, reviews, and commits. A context pipeline is infrastructure made up of retrieval chains, reranking models, summarization steps, schema injection, token budget management, and assembly logic. These components execute at runtime before each model call. Prompt engineering is typically owned by the person who writes the system prompt. Context engineering is owned by platform teams and lives alongside API servers, data pipelines, and observability tooling.

Diagnosing a context pipeline that misbehaves in production involves a different debugging surface. Rewriting the system prompt won’t resolve the issue. Instead, you need to trace retrieval results, audit the assembled context, and inspect token utilization.

Four Ways Prompt-Only Approaches Break in Production

1. Positional Attention Degrades Recall on Long Contexts

Stanford's "Lost in the Middle" research (Nelson F. Liu et al., 2023) showed that LLM recall performance degrades significantly when critical information appears in the middle of a long context. Models attend most reliably to content at the start and end of the context window. Put 30 retrieved documents into the context in arbitrary order, and the model will effectively ignore the middle 20. This is a positional attention pattern baked into how transformers process long sequences, which you can’t instruct your way around. The solution is to fix how you assemble and order the context.

2. Context Rot Hits Before You Hit Token Limits

Many teams observe accuracy degradation as context grows, even before hitting token limits. This is often due to reduced signal density and attention dilution rather than hard limits. Research on context rot shows that even on simple tasks, performance drops as total token count grows, well before you approach 128K or 200K limits. Switching to a larger context window, for example, from GPT‑4o to Claude’s 200K window, can delay the failure, but it does not address the underlying issue. A smaller context with tightly selected, high‑signal content consistently outperforms a much larger context filled with irrelevant material.

3. Static Prompts Can't Track Dynamic Systems

A static prompt encodes knowledge at write time. It doesn't know that payments-service changed its primary upstream dependency last Tuesday, that the on-call rotation shifted, or that the runbook was updated after last quarter's incident. The model answers from training data or from whatever static text you last injected, both stale by definition in a dynamic engineering environment.

4. Non-Determinism From Uncontrolled Context Is Impossible to Debug

For pipelines that depend on parseable, stable responses, such as incident triage agents, onboarding bots, and automated code review tools, it can be very difficult to identify failures due to information changing between prompts. You can't reproduce these by running the prompt again. You need observability on what was actually in the context window for the failing call.

Anatomy of a Production Context Pipeline

A typical production context pipeline runs as a sequence: user query, intent classification, parallel retrieval (vector search plus catalog API lookup), reranking, compression, context assembly, token budget audit, and LLM call.

A Python implementation of the core assembly step can use LangChain with a Backstage catalog API and a pgvector store:

import tiktoken
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import PGVector
import requests

ENCODING = tiktoken.encoding_for_model("gpt-4o")
TOKEN_BUDGET = 128_000
SLOT_WEIGHTS = {
    "system_instructions": 0.10,  # ~12,800 tokens
    "retrieved_context":   0.55,  # ~70,400 tokens
    "tool_schemas":        0.15,  # ~19,200 tokens
    "conversation_history": 0.20, # ~25,600 tokens
}

def fetch_catalog_entities(service_name: str, backstage_url: str) -> list[dict]:
    """Query Backstage catalog API for a component and its dependency metadata."""
    resp = requests.get(
        f"{backstage_url}/api/catalog/entities",
        params={"filter": f"kind=Component,metadata.name={service_name}"},
        headers={"Authorization": "Bearer \${BACKSTAGE_TOKEN}"},
    )
    resp.raise_for_status()
    entities = resp.json()
    return [
        {
            "name":        e["metadata"]["name"],
            "owner":       e["spec"].get("owner", "unknown"),
            "depends_on":  e["spec"].get("dependsOn", []),
            "description": e["metadata"].get("description", ""),
            "annotations": e["metadata"].get("annotations", {}),
        }
        for e in entities
    ]

def assemble_context(
    query: str,
    system_instructions: str,
    tool_schemas: list[dict],
    conversation_history: list[dict],
    backstage_url: str,
    vector_store: PGVector,
    service_name: str,
) -> dict:
    # Fetch typed entity metadata from the catalog
    catalog_entities = fetch_catalog_entities(service_name, backstage_url)

    # Vector similarity search over embedded TechDocs chunks
    docs = vector_store.similarity_search(query, k=8)

    # Assemble named slots with explicit budget allocation
    context = {
        "system_instructions": system_instructions,
        "retrieved_context": {
            "catalog_entities": catalog_entities,
            "techdocs": [d.page_content for d in docs],
        },
        "tool_schemas": tool_schemas,
        "conversation_history": conversation_history,
        "user_message": query,
    }

    # Enforce token budget before the LLM call
    # This is an approximation—actual token usage depends on how the context is serialized into the model’s message format.
    total_tokens = len(ENCODING.encode(str(context)))
    assert total_tokens < TOKEN_BUDGET, (
        f"Context exceeds budget: {total_tokens} / {TOKEN_BUDGET} tokens"
    )
    return context

The slot allocation (10% system instructions, 55% retrieved context, 15% tool schemas, 20% conversation history) is a starting point, not a fixed rule. Incident triage agents typically allocate more budget to retrieved context because the reasoning depends heavily on documents. Coding assistants often shift more budget toward conversation history to preserve code state across turns. You can encode these allocations directly in code so you can see exactly how a context was constructed when a call fails.

In production, each stage of the pipeline needs to be inspectable. That usually includes the retrieved documents, reranked results, final assembled context, and token counts. Without this visibility, debugging becomes guesswork. You can address this by capturing full context snapshots per request using tools like LangSmith or custom logging pipelines.

When a task no longer fits cleanly within a single context window, you can decompose it into sub‑agents. The sub‑agent boundary determines what information each agent needs access to and what it can safely ignore.

Prompt Engineering vs Context Engineering: Scope, Ownership, and Failure Mode

In production systems, prompt engineering functions as one component within a larger context assembly pipeline, with different ownership, tooling, and failure modes.

| Dimension | Prompt Engineering | Context Engineering | |---|---|---| | Scope | Instruction design for a single call | Full context window lifecycle across all calls | | Owner | Prompt author / ML engineer | Platform engineer / infrastructure team | | Primary artifact | System prompt string | Retrieval and assembly pipeline | | Repeatability guarantee | Varies with context state | High when context assembly is deterministic | | Failure mode | Non-determinism from uncontrolled inputs | Context rot or retrieval miss | | Debugging approach | Rewrite instructions and re-test | Trace retrieval results, audit token budgets, inspect assembled context |

The Developer Portal as Context Infrastructure

For AI workflows inside an engineering organization, document retrieval on its own is rarely enough. When teams ask questions about a service, they usually need concrete facts like who owns payments-service, what it depends on upstream, what its SLO targets are, where the runbook lives, and who is on call. That information exists as structured data, not as natural language text, and treating it as just another document tends to produce unreliable results.

The Backstage catalog API exposes this data directly. A GET /api/catalog/entities?filter=kind=Component call returns JSON with fields like kind, spec.owner, spec.dependsOn, metadata.annotations, and metadata.description. These map directly to an LLM context schema with minimal transformation. The more structured the source data, the smaller the hallucination surface area. The model doesn't need to infer ownership or dependency relationships from text because the typed fields make both explicit.

The difference becomes most apparent during incident triage workflows. In practice, this usually involves an internal tool being queried as the first step of triage. For example, if an agent receives a PagerDuty alert for payments-service without catalog context, it falls back to generic advice like checking database connections, CPU utilization, or recent deployments. When catalog data is injected at call time, including the owning team, upstream dependencies, and the linked runbook, the response references the actual dependency chain and points to the correct escalation path.

Roadie provides this data layer for both human engineers and AI agents. The catalog API, TechDocs indexing, and entity relationship graph are maintained out of the box. Teams don't absorb the operational cost of self-hosting Backstage, including upgrade cycles timed with every Backstage release, plugin compatibility work, and infrastructure ownership, while also building the AI pipelines on top of it. Roadie handles the context source, while your team focuses on the retrieval and assembly pipeline.

Start Here: Wire Your Catalog Into a Context Pipeline Today

The service metadata in your catalog is already structured, authoritative, and queryable. You can wire that data into a retrieval pipeline that your agents can consume in three steps:

1. Export catalog entities: Call GET /api/catalog/entities?filter=kind=Component against your instance. The response gives you all service entities as JSON. Extract metadata.name, spec.owner, spec.dependsOn, and metadata.description for embedding.

2. Chunk and embed: Concatenate the relevant fields into a single text block per entity and generate embeddings.

```python
import openai

entity_text = (
    f"{entity['name']}: {entity['description']}. "
    f"Owner: {entity['owner']}. "
    f"Depends on: {', '.join(entity['depends_on'])}"
)
response = openai.embeddings.create(
    model="text-embedding-3-small",
    input=entity_text,
)
vector = response.data[0].embedding
# Store vector alongside raw entity JSON in pgvector
```

Store vectors alongside the raw JSON entity payload in pgvector. For higher query volume or managed infrastructure, Pinecone offers the same cosine similarity search without the Postgres operational overhead.

3. Build the context assembly function: On each agent query, run similarity search against the vector store, retrieve the top‑k entity payloads, and inject the serialized JSON into the model’s context before the LLM call.

Any team with a populated service catalog can implement this with minimal overhead and typically see an immediate reduction in hallucinated service names and incorrect ownership references. Once real service metadata occupies the context window, the model stops inventing relationships from training data.

If your team is already running Backstage or Roadie, your context source exists. The next step is building the retrieval pipeline on top of it. Book a demo to see how Roadie structures catalog data for AI-ready context retrieval.

The LLM has the capability to reason about dependency graphs in generic contexts, but it can’t tell you anything about a dependency graph it doesn’t have access to. The gap between what a modern LLM can do and what it actually knows about your specific systems is what kills AI adoption for anything beyond code generation.

The Two Categories of AI Tasks in Engineering Orgs

AI tasks that show up in engineering organizations can usually be split between generic and org-specific tasks.

Generic tasks such as writing a unit test, suggesting a refactor, or explaining a regex work well out of the box because the underlying knowledge is universal. The model has seen thousands of similar examples in its training data.

Org-specific tasks are different. "Which team owns auth-gateway?" "Did payments-service deploy anything in the last four hours?" "What's the runbook for checkout-api queue consumer failures?" These questions require private, structural knowledge about your organization that no pre-trained model has and can't hallucinate accurately. The knowledge is specific, relational, and continuously changing.

Most teams try to close this gap by dumping documentation into a vector store and calling it RAG. Confluence pages, GitHub READMEs, and runbook docs get chunked, embedded, and retrieved at query time. This might work until the docs go stale (immediately), ownership information gets siloed in a wiki nobody maintains (also immediately), or the agent retrieves a plausible document that describes the system as it existed eighteen months ago. Unstructured documentation is a poor substrate for org-specific AI tasks. It has no canonical entity IDs, no typed relations, and no consistent update cadence. You end up with confident wrong answers, which are worse than no answers.

The most structured, continuously updated source of engineering context in your org is your Internal Developer Platform. Context engineering, or deciding what data populates a model’s context window at inference time, treats the IDP as context infrastructure rather than a simple developer portal. The underlying knowledge is already there, and exposing that data involves wiring it into the model’s context.

What Your IDP Actually Knows (and Why That Data Is Rare)

A mature Backstage IDP maintains a layered graph of operational facts about every registered service. Each context layer maps to a different type of data:

1. Service catalog: Component, API, System, and Resource entity kinds carry spec.type, spec.lifecycle, and metadata.tags for tech stack metadata. spec.owner links every component to the team or group accountable for it. This alone answers a class of AI queries ("who owns this service?", "which services are in production lifecycle?") that most agents can't currently handle.

2. Ownership graph: The traversal spec.owner >Group entity >spec.members gives you a directed chain from any service name to a list of actual humans or an on-call rotation. When a PagerDuty plugin is attached, the group entity can resolve directly to an active incident responder, not just a team name.

3. Dependency map:spec.dependsOn, spec.providesApis, and spec.consumesApis form a queryable directed graph of service-to-service relationships. This is the data an AI agent needs to answer "what else does this change affect?" during change-impact analysis or incident scope assessment.

4. Deployment history:GitHub Actions, Tekton, and ArgoCD Backstage plugins surface deploy metadata as catalog annotations (backstage.io/last-deploy-timestamp, commit SHA, deploying user). An agent with access to this data can answer "did anything deploy to checkout-api in the past 6 hours?" without needing to query GitHub directly.

5. Incident data: PagerDuty and Opsgenie plugins embed open incident counts, on-call rotation names, and service health thresholds as entity annotations. This is the difference between an agent that helps triage and one that produces noise.

All of this data is machine-readable by design. Every piece of it exists as structured YAML at the source, delivered as typed JSON through the Backstage Catalog API. Contrast that with a Confluence page about checkout-api, which might contain some of this information, written in prose, last updated whenever someone remembered to do it. The IDP version is authoritative, entity-keyed, and alive.

Proprietary portals that don't offer accessing the catalog entities through a consistent data structure lack the structural backbone that makes this data tractable as an AI context source.

Context Engineering: What Goes Into the Window and Why It Matters

Unlike prompt engineering, which focuses on how you phrase a request, context engineering relies on a set of decisions to determine what the model sees before it generates anything at all. These decisions might include what data to retrieve, how to structure it, when to inject it, and how much to trim.

For an AI agent operating in a production engineering org, IDP data maps to four distinct context types, each relevant to a different query class:

• Factual context (ownership, lifecycle, tech stack) answers "who owns this" and "what kind of thing is this"
• Relational context (dependency maps) answers "what else is affected" and "what does this call"
• Historical context (deployment events, incident records) answers "what changed recently" and "has this broken before"
• Procedural context (runbooks, ADRs, TechRadar entries linked to catalog entities) answers "how do we handle this"

The architectural advantage of IDP data over unstructured docs is precision. Catalog entities have canonical identifiers and typed relations, which means retrieval can combine semantic search with structured filtering. A vector similarity search can surface the relevant entity description, while the entity name, namespace, and relations ensure the agent retrieves the correct checkout-api component, not just a document that happens to mention checkout in passing. Semantically similar isn't good enough; the context needs to resolve to the exact service entity.

Three Patterns for Wiring Your IDP to an AI Agent

The following three patterns describe different ways of consuming the Backstage Catalog API as an underlying data source. Each uses a different retrieval mechanism and requires distinct infrastructure.

Pattern A: RAG Over Catalog Entities

Embed catalog entity descriptors as structured text chunks and store them in a vector index, such as pgvector or a hosted vector service (use what you already have). Retrieve the relevant chunks at query time and inject them into the system prompt. LangChain and LlamaIndex both have straightforward document loader patterns for this.

The chunking strategy matters. Avoid embedding an entire entity as a single chunk. Instead, split entities by context type: a facts chunk (name, owner, lifecycle, description), a dependencies chunk (dependsOn, providesApis, consumesApis), and an incident/deployment chunk (annotations). This produces three embeddings per entity, all keyed to the same entity name, and enables more precise retrieval when a query targets dependencies rather than ownership.

Best for: Read-only Q&A at scale ("list all services owned by team-payments", "which services are in experimental lifecycle?").

Trade-off: Can impact index freshness. If you don't wire catalog change events to index updates, your RAG pipeline will drift from the live catalog.

Pattern B: MCP Server Wrapping the Backstage Catalog API

Run a Model Context Protocol (MCP) server that wraps the Backstage Catalog API and exposes catalog operations as agent-callable tools. Anthropic’s MCP server (released November 2024) defines a standard for exposing external systems in this way, allowing agents to fetch fresh catalog data during inference.

The MCP server translates existing Catalog API endpoints into tool definitions. For example:

| MCP Tool | Backstage Endpoint | |---|---| | get_component_by_name | GET /api/catalog/entities/by-name/component/{namespace}/{name} | | list_entities_by_owner | GET /api/catalog/entities?filter=spec.owner={team} | | get_entity_relations | GET /api/catalog/entities/by-name/{kind}/{namespace}/{name} |

When the agent receives a query, it can call these tools dynamically and retrieve fresh catalog data during reasoning.

For example, answering a question like:

Which services owned by team-payments have dependencies on checkout-api?

might involve multiple tool calls:

1. Query services owned by the team
2. Retrieve each entity's dependency relations
3. Filter those that reference checkout-api

An MCP server orchestrates those API calls while exposing them to the agent as simple tools.

Best for: Multi-step agentic workflows that need to traverse the service graph dynamically.

Trade-off: Every tool call translates into a Catalog API request, so complex queries can introduce additional latency compared to pre-indexed retrieval.

Pattern C: Direct Function-Tool Definitions

Define Catalog API endpoints as function tools directly in your OpenAI or Claude API call. No additional infrastructure. The agent calls the tool during inference, fetches entity data, and incorporates it into its response.

Here's a minimal tool definition for get_component_by_name:

{
  "name": "get_component_by_name",
  "description": "Retrieve a Backstage catalog entity for a named service component. Returns ownership, dependencies, lifecycle status, runbook URL, and deployment metadata.",
  "parameters": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string",
        "description": "The component name as registered in the Backstage catalog (e.g., 'checkout-api', 'payments-service')"
      },
      "namespace": {
        "type": "string",
        "description": "The Backstage namespace, defaults to 'default'",
        "default": "default"
      }
    },
    "required": ["name"]
  }
}

The implementation calls GET /api/catalog/entities/by-name/component/{namespace}/{name} against your Backstage instance.

Best for: Shipping a proof of concept today. No new infrastructure, real-time data, works with any LLM that supports function calling.

Trade-off: You're making an API call per query. At high volume, Pattern A's pre-indexed retrieval will be faster.

Choosing the right pattern

Start with Pattern C today. Graduate to Pattern B as your agentic workflows get more complex, particularly when they require multi‑hop traversal. Pattern A is the right call if you're building read-heavy Q&A at scale and want to minimize per-query API latency.

The table below summarizes the key trade‑offs between the three patterns.

| Pattern | Freshness | Latency | Infra Complexity | Best For | |---|---|---|---|---| | A: RAG over catalog entities | Index refresh cadence | Low (pre-indexed) | Medium (embedding pipeline + vector store) | Read-only Q&A at scale | | B: MCP server | Real-time | Higher (per-hop API call) | High (MCP server) | Multi-step agentic workflows | | C: Direct function tools | Real-time | Medium (per-query API) | Low (none) | Zero-infra proof of concept |

Building the Pipeline: From catalog-info.yaml to Context String

In the pipeline from IDP to AI agent, Backstage acts as the system of record, where services are described in catalog-info.yaml with structured relationships and metadata. The Catalog API provides a queryable interface over that data, returning entity definitions as JSON. From there, the pipeline converts those entities into smaller, purpose-built context representations — either as embeddings for retrieval or as structured responses returned through tool calls.

A typical catalog-info.yaml in a production Backstage instance can define metadata such as its name, description, owner, dependencies, and the APIs it provides.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: checkout-api
  description: "Handles payment checkout flows and queue consumer processing"
  annotations:
    pagerduty.com/service-id: "P1234XY"
    backstage.io/last-deploy-timestamp: "2025-01-14T22:31:00Z"
    runbook-url: https://runbooks.internal/checkout-api
spec:
  type: service
  lifecycle: production
  owner: team-payments
  system: payments-platform
  dependsOn:
    - component:default/payments-service
    - component:default/inventory-api
    - resource:default/checkout-queue
  providesApis:
    - checkout-api-v2
  consumesApis:
    - payments-processing-api

The Catalog API surfaces the YAML as a JSON entity at GET /api/catalog/entities/by-name/component/default/checkout-api. The fields your agent needs are in metadata, spec, and metadata.annotations. An important thing to keep in mind is that while the catalog-info.yaml file is the most common method to declare entities, the catalog can also be populated from multiple data sources, such as AWS, and in many scenarios, enriching an entity from multiple sources can provide a richer context to the AI agent.

Here's a Python function that takes the entity JSON and returns a structured context string ready to inject into a system prompt or retrieve from a vector index:

def parse_entity_ref(ref: str) -> str:
    """
    Convert Backstage entity refs like:
    'component:default/payments-service'
    into just:
    'payments-service'
    """
    try:
        return ref.split("/")[-1]
    except Exception:
        return ref
def entity_to_context_string(entity: dict) -> str:
    metadata = entity.get("metadata", {})
    spec = entity.get("spec", {})
    annotations = metadata.get("annotations", {})

    name = metadata.get("name", "unknown")
    description = metadata.get("description", "No description provided")
    owner = spec.get("owner", "UNKNOWN — ownership gap")
    system = spec.get("system", "UNKNOWN — system unassigned")
    lifecycle = spec.get("lifecycle", "unknown")

    depends_on = [parse_entity_ref(r) for r in spec.get("dependsOn", [])]
    provides_apis = [parse_entity_ref(r) for r in spec.get("providesApis", [])]
    consumes_apis = [parse_entity_ref(r) for r in spec.get("consumesApis", [])]

    runbook = annotations.get("runbook-url", "No runbook linked")
    last_deploy = annotations.get("backstage.io/last-deploy-timestamp", "No deploy data")
    pagerduty_id = annotations.get("pagerduty.com/service-id", "No PagerDuty link")

    return f"""Service: {name}
Description: {description}
Owner: {owner}
System: {system}
Lifecycle: {lifecycle}

Dependencies: {', '.join(depends_on) if depends_on else 'None recorded'}

Provides APIs: {', '.join(provides_apis) if provides_apis else 'None recorded'}
Consumes APIs: {', '.join(consumes_apis) if consumes_apis else 'None recorded'}

Last Deploy: {last_deploy}
Runbook: {runbook}
"""

Example: What the Model Actually Sees

When the agent retrieves context for a service like checkout-api, the information injected into the model’s context window is a structured block derived from the catalog entity, not the raw YAML or JSON.

A typical context injection might look like this:

Description: Handles payment checkout flows and queue consumer processing
Owner: team-payments
System: payments-platform
Lifecycle: production

Dependencies: payments-service, inventory-api, checkout-queue

Provides APIs: checkout-api-v2
Consumes APIs: auth-api

Last Deploy: 2025-01-14T22:31:00Z
Runbook: https://runbooks.internal/checkout-api

This block is small enough to fit comfortably inside an LLM context window while still giving the model the critical operational facts it needs to answer questions like:

"Who owns checkout-api?"

"What services might be affected if checkout-api fails?"

"Did anything deploy recently that could explain this incident?"

For Pattern A (RAG), different slices of this information are typically embedded separately. For example, one embedding for service facts, one for dependency relations, and one for operational history. So, the retrieval layer can return only the context relevant to the user’s question.

The model reasons over authoritative service metadata pulled directly from the IDP catalog, rather than relying on inference or approximation.

If you're running on Roadie, the Catalog API is already available at a stable, authenticated endpoint. Roadie also ships the AI Assistant RAG plugin, which implements the embedding, indexing, and retrieval layer of Pattern A out of the box. If your org is already on Roadie, the pipeline in this section is largely already running. You would only need to connect your LLM endpoint to it, instead of building the chunking infrastructure from scratch.

Operational Hygiene: Your Context Is Only as Good as Your Catalog

Incomplete catalog data creates predictable failure patterns when used as AI context. Here’s a typical example: an AI agent pages the wrong team during an incident because spec.owner was missing from a catalog entity, and the agent fell back to a default or hallucinated a plausible owner name.

To avoid this class of failure, a catalog needs to meet three completeness requirements before it’s safe to use as an AI context source:

spec.owner must be populated on every component: Unenforced ownership means the agent has no escalation path. An agent that can't answer "who owns this service?" is useless for incident triage and on-call routing, which are the two highest-value use cases for real-time IDP context.

metadata.description must be non-empty: Empty descriptions degrade embedding quality and cause false-positive retrievals in Pattern A. A query for "checkout flow services" can return inventory-api simply because its description is empty, effectively turning it into a wildcard candidate during retrieval.

System relations must be defined: Without spec.system, the dependency graph is a set of disconnected nodes. An agent trying to answer "what other services are in the same system as checkout-api?" can't traverse a graph that doesn't have system edges. This matters for change-impact analysis, which needs to understand blast radius within a system boundary.

Beyond completeness, two operational concerns apply regardless of which pattern you use:

Access control: AI agents query the catalog on behalf of users. Backstage's permission framework must be enforced at the API layer so agents can't surface catalog data that the requesting user isn't authorized to see. Don't skip this step just because the agent interface feels informal.

Catalog freshness: Catalog auto-sync must be wired to source change events such as SCM push hooks and CI completion events, not nightly batch jobs. Deployment history and incident annotations are time-sensitive. A last-deploy timestamp from eight hours ago is misleading context during an active incident. Every hour of staleness widens the hallucination window on operational queries.

That said, wiring SCM-triggered catalog refresh reliably is harder in practice than it sounds. Sync failures, webhook misconfigurations, and integration drift are recurring operational costs of self-hosted Backstage. On Roadie's SaaS platform, SCM-triggered catalog refresh and entity validation are managed infrastructure, which removes the self‑hosted costs associated with stale‑catalog problems caused by sync failures. This is a concrete build-vs-buy consideration for teams deciding where to invest engineering time.

Start Here: Audit Your Catalog for AI Context Readiness Today

To get started auditing your catalog for AI context readiness, here are three steps you can complete before end of day. No new dependencies required.

Step 1: Run a completeness audit. Hit GET /api/catalog/entities?filter=kind=component against your Backstage instance and pipe the response through this script:

import requests
from collections import defaultdict

BACKSTAGE_URL = "https://your-backstage.example.com"
TOKEN = "your-backstage-token"

def audit_catalog_completeness():
    url = f"{BACKSTAGE_URL}/api/catalog/entities?filter=kind=component"
    headers = {"Authorization": f"Bearer {TOKEN}"}

    response = requests.get(url, headers=headers)
    entities = response.json()

    gaps = defaultdict(list)

    for entity in entities:
        name = entity["metadata"]["name"]
        spec = entity.get("spec", {})
        metadata = entity.get("metadata", {})

        if not spec.get("owner"):
            gaps["missing_owner"].append(name)
        if not spec.get("system"):
            gaps["missing_system"].append(name)
        if not metadata.get("description"):
            gaps["missing_description"].append(name)

    total = len(entities)
    print(f"\nCatalog Completeness Audit — {total} components\n")

    for gap_type, names in gaps.items():
        pct = len(names) / total * 100
        print(f"{gap_type}: {len(names)} entities ({pct:.1f}%)")
        for name in names[:5]:
            print(f"  - {name}")
        if len(names) > 5:
            print(f"  ... and {len(names) - 5} more")

audit_catalog_completeness()

This tells you exactly how much context debt you're sitting on. If 30% of your components are missing spec.owner, that's 30% of the queries an AI agent handles about ownership that will produce wrong or empty answers.

Step 2: Fix the gaps on your top 10 most critical services first. Define "most critical" as the highest deploy frequency, most upstream dependents, or most incident-prone, whichever your team can quantify. These are the services an AI agent will be asked about most often. A complete catalog entry for checkout-api is worth more than partial entries for 50 internal tools nobody queries.

Step 3: Write and test one function-tool definition. Take the JSON tool definition from Pattern C above, attach it to a Claude or OpenAI Playground session, and ask a real question such as "Who is on call for payments-service?" or "What does checkout-api depend on?" If the catalog entry is complete, the answer will be correct. If the answer is wrong or empty, the output will point to the gap in the catalog data, such as a missing spec.owner, an empty spec.dependsOn, or a PagerDuty annotation that hasn’t been set.

The catalog completeness audit is the forcing function here. A RAG pipeline built on top of an incomplete catalog cannot reliably produce accurate answers. Getting the data right comes first. The context pipeline is the easy part.

Frequently Asked Questions

What is context engineering for AI agents?

Context engineering is the discipline of deciding what data populates a model's context window at inference time: what to retrieve, how to structure it, when to inject it, and how much to trim. Unlike prompt engineering, which focuses on how you phrase a request, context engineering controls what the model sees before generating a response. This means wiring structured IDP data, including ownership graphs, dependency maps, and deployment history, directly into agent queries.

Why is an IDP better than Confluence for AI context?

An Internal Developer Platform like Backstage stores engineering knowledge as structured, entity-keyed, machine-readable YAML. Every component has a canonical ID, typed relations (spec.dependsOn, spec.owner), and is updated continuously via SCM and CI integrations. Confluence pages go stale immediately, have no canonical entity IDs, and contain no typed relations. Structured IDP data produces grounded, accurate answers.

Which pattern should I start with: RAG, MCP, or function tools?

Pattern C (direct function-tool definitions) is usually the best place to start. It requires no new infrastructure, delivers real-time Catalog API data, and works with any LLM that supports function calling. You can ship a working proof of concept today. Graduate to Pattern B (MCP server) when your agentic workflows need multi-hop catalog traversal. Pattern A (RAG over catalog entities) is a better fit when you need low-latency, read-only Q&A at scale.

Final Thoughts

Every engineering org with a functioning IDP already has the context to make org-specific AI tasks tractable. Ownership graphs, dependency maps, deployment history, and incident annotations are already present as structured, entity-keyed data that updates continuously. Bridging the gap between "we have an IDP" and "our AI agent knows our system" is largely a matter of wiring.

The three patterns above provide a concrete path from the Backstage Catalog API to a grounded AI agent, whether you want to ship something quickly (Pattern C), build a scalable retrieval pipeline (Pattern A), or support multi-hop agentic workflows (Pattern B). The catalog completeness requirements define the data quality bar that makes any of these patterns reliable in production.

Your IDP's catalog is already the most accurate, continuously-updated map of your engineering org, making it a practical foundation for AI agents’ context infrastructure.

If you're running Backstage and want the Catalog API, entity sync, and AI Assistant RAG pipeline without the self-hosted maintenance overhead, Roadie's managed platform ships all three. See how Roadie turns your IDP into a context engine.

Today, there are three distinct paths:

1. Self-hosted Backstage
2. Proprietary portals
3. SaaS portal built on an open standard

Backstage became the Kubernetes of developer portals, not because it's easy to run, but because it's the open standard that solves the portability problem. When you build on Backstage, you're choosing an ecosystem where your data model isn't locked to a single vendor. But this flexibility comes with real operational costs that many teams underestimate.

This guide examines Backstage's technical architecture, specifically its plugin system, entity model, and extension points, and explains where it fits in the modern platform engineering stack. You'll understand the trade-offs between framework flexibility and product convenience, and why SaaS portal built on an open standard represents an architectural evolution rather than just another hosting option.

Deconstructing Backstage: How it Actually Works

Backstage isn't a product you install. It's a collection of TypeScript libraries that you assemble into your own developer portal. This distinction matters because it changes how you think about customization, upgrades, and ownership.

Assembly Required: Backstage as a Framework

When you start with Backstage, you're creating a new Node.js application. You import Backstage packages as dependencies, configure them through code, and deploy the resulting application to your infrastructure. This is fundamentally different from signing into a SaaS portal where the vendor controls the codebase. Backstage is an extensible application framework that you build and tailor to your organization’s needs.

The core Backstage repository provides the foundational packages: @backstage/core-components for React UI elements, @backstage/backend-defaults for the new backend system, and @backstage/catalog-model for the type system. You compose these into your instance by modifying the packages/app and packages/backend directories in your Backstage monorepo.

This architecture gives you the ability to treat your portal as an internal platform product and something you can evolve deeply to match your organization. If you want to adjust navigation patterns, you can modify the React components directly. If your authentication model is complex, you can implement it in your backend package. If your organization has domain concepts that don’t fit predefined schemas, you can extend the entity model to reflect them.

The portal becomes an application you continuously evolve over time. Want to change how the sidebar renders? You edit the React components directly. Need custom authentication logic? You implement it in your backend package.

In proprietary portals, customization typically happens through predefined settings and extension points, which is faster to adopt but structurally more constrained.

The Plugin Architecture: Backstage’s Extensibility Model

In Backstage, even core capabilities like the catalog are implemented as plugins. This isn't just organizational; it's the technical mechanism that makes Backstage an ecosystem rather than a single product.

Plugins are NPM packages that implement specific interfaces. Frontend plugins use createFrontendPlugin from @backstage/frontend-plugin-api and contribute UI through extension points such as pages and entity components (for example via blueprints like PageBlueprint or EntityCardBlueprint). Backend plugins use createBackendPlugin from @backstage/backend-plugin-api with a microservice architecture where plugins operate in complete isolation, communicating only through network calls.

The plugin system is implemented through Backstage's extension API. Installing a plugin is a build-time composition step: you import the NPM package, register it in the app, and Backstage wires it into the portal through defined extension points. Backstage core provides shared platform services like routing, authentication context, configuration, and consistent UI primitives, while plugins focus on domain-specific integration logic.

For example, the Kubernetes plugin provides a frontend component that displays cluster resources witha backend integration that queries Kubernetes APIs. The plugin reuses shared concerns like authentication context, configured proxy settings, and secrets handling rather than re-implementing them.

This plugin-first architecture is what enables the Backstage ecosystem to scale, with over 250 community plugins. Instead of requiring changes to core, integrations can be delivered as modular plugins - connecting to systems like PagerDuty, Datadog, or your CI/CD system - while remaining aligned to a shared extension model.

The Entity Model: The System of Record

Backstage's other foundational concept isthe Software Catalog. At its core, this catalog is a graph of entities that represent your software ecosystem. Entities are commonly defined in catalog-info.yaml files that live alongside your source code.

An entity can represent: a service, a library, a team, an API, or a custom resource type you define. Each entity has a kind (Component, API, Resource, System, Domain, Group, User, Template, or Location), metadata (name, description, annotations), and relationships to other entities.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  description: Handles payment processing
  annotations:
    github.com/project-slug: company/payment-service
spec:
  type: service
  lifecycle: production
  owner: payments-team
  system: checkout
  providesApis:
    - payment-api
  consumesApis:
    - fraud-detection-api

The model is declarative and versioned. Your entity definitions live in git, versioned alongside your code. Over time, the catalog becomes a structured system of record for: ownership, lifecycle state, dependencies, API relationships and the context required to power templates, scorecards, and automation.

The catalog backend continuously processes these YAML files through entity providers, plugins that discover and ingest entities from different sources. For example, the GitHub provider scans your repositories for catalog-info.yaml files and the Kubernetes provider can generate entities from cluster resources. This extensibility means you control what gets cataloged and how.

The result is that Backstage is more than a developer portal UI. It is a structured model of your engineering organization, which the portal then uses as context for workflows, integrations, and governance.

The Landscape: Three Ways to Adopt a Developer Portal

Once you understand Backstage’s architecture, the market becomes easier to reason about.

Organizations evaluating developer portals are typically choosing between three distinct models:

1. Self-hosted Backstage (framework ownership)
2. Proprietary SaaS portals (closed product ownership)
3. SaaS portals built on the Backstage standard (productized standard)

The differences are not primarily about features. They’re about ownership, extensibility, and where your system of record lives.

Self-Hosted Backstage: You Own the Framework

In this model, you adopt the Backstage standard directly and operate it yourself. You control the React frontend, the Node backend, the plugin surface area, the entity model and schema evolution as well as the upgrade and migration lifecycle.

This maximizes architectural control, but you also comes with responsibility for maintaining a rapidly evolving framework. The portal becomes an internally operated software product. For organizations with strong platform engineering capacity, this can be a deliberate and strategic choice.

Proprietary Portals: The Vendor Owns the Model

In a closed SaaS model, the vendor owns the application, the schema and the extension model.

These tools are typically optimized for rapid setup. You connect your Source Control and your CI system, and a catalog is generated. You configure scorecards and workflows through predefined interfaces. This speed comes from removing choices. However, the catalog and workflow layer usually live inside a vendor-defined schema. Even when APIs are available, the data model is product-defined. Over time, your service metadata and automation logic become coupled to that model.

Extensibility typically happens through predefined extension points: custom fields, webhooks, workflow builders, or API integrations. That can be sufficient for many teams. But if your organization’s domain model diverges significantly from the product’s assumptions, deeper customization often requires vendor support or roadmap alignment.

The trade-off is clear: faster time-to-value, but less control over how the system evolves.

SaaS portal built on an open standard: Productizing the Architecture

In this hybrid approach, the vendor owns the application, operational lifecycle (upgrades, migrations, infrastructure), and schema evolution. However, the architecture remains aligned to Backstage principles and ecosystem patterns.

The key distinction is extensibility. Rather than limiting integrations to a fixed set of vendor-defined connectors, the system is built around a plugin-oriented model. Integrations are modular, scoped, and composable. Teams can extend the portal to integrate internal systems, domain-specific tooling, or custom workflows without modifying core application code.

The difference from closed SaaS portals is architectural. Instead of limiting extensibility to configuration, webhooks, or predefined workflow builders, this model treats extensibility as a first-class concern. New capabilities can be introduced as modular integrations, rather than requiring changes to the core product.

This approach exists because many organizations want the architectural advantages of the Backstage standard like modular integrations, catalog-driven context, and workflow extensibility, without taking on the responsibility of operating and upgrading a framework deployment themselves.

The Data Model Trade-off

Over time, a developer portal becomes less about UI and more about context: ownership, dependencies, lifecycle state, maturity standards, and the metadata needed to power workflows. Where that context lives, and who controls how it evolves, has long-term implications.

In a self-hosted Backstage deployment, the context model is defined in code and declarative descriptors (often versioned in Git). The organization controls how the model evolves and how relationships are represented.

In a closed SaaS portal, the context model typically lives inside a vendor-managed database with a product-defined schema. Even when APIs exist, the shape of the model and its evolution are governed by the vendor.

In a SaaS portal built on open standard, the vendor owns the application and the evolution of the model, but the portal is still built around catalog-driven context and modular extensibility rather than a fixed set of vendor-only connectors.

The trade-off isn’t theoretical. Once ownership, automation, and governance are encoded into a portal, migrating between models is rarely just a data export. It involves re-mapping relationships, workflows, and assumptions embedded in how that context is represented.

The Operational Layer: Running Backstage Is Real Engineering Work

Backstage’s power comes from the fact that it’s an open source framework, but that also means you are running a real software system. A self-hosted Backstage deployment is not a static tool. It’s:

• A Node.js backend with multiple plugins and integrations
• A React frontend composed from independently versioned packages
• A PostgreSQL database backing the catalog and metadata
• Authentication, authorization, and secrets management and the lifecycle of ongoing dependency and framework upgrades

This is infrastructure-level responsibility.

Backstage releases happen monthly, on the Tuesday before the third Wednesday of each month. Individual packages follow semantic versioning, but Backstage releases explicitly don't adhere to semver. Breaking changes are documented with **BREAKING**: prefix in changelogs - but the practical reality is that staying current requires consistent engineering attention.

The TypeScript tax is real. Plugins evolve quickly, and compatibility issues do happen: plugin updates can introduce breaking changes, and deep customizations can conflict with upstream changes. Running Backstage well requires engineers who understand the Backstage framework architecture, can debug the plugin composition model, and can keep upgrades from turning into periodic migrations.

Security is your responsibility. You configure authentication, implement authorization rules, and manage secrets. RBAC in Backstage requires implementing permission policies as code through the permission framework, which means you need someone who actually understands the permission model to secure your instance correctly as the portal grows

Operationa complexity compounds at scale. You need to monitor the backend, tune database queries, and keep frontend bundle sizes from drifting. When the portal is down, you handle the incident response. And because the portal becomes a daily surface for developer workflows, reliability expectations tend to climb quickly.

Many teams understaff Backstage operations. They assume that because it's "just a developer portal," it doesn't need dedicated ownership. Then they struggle with upgrades, accumulate technical debt, and lose confidence in the platform.

The complexity isn’t a flaw, it's the cost of adopting a highly extensible standard. When ownership is vague, upgrades get deferred, incompatibilities pile up, and confidence erodes. When it’s staffed like a core platform service, it behaves like one. That pattern shows up in Backstage’s own community reporting too: organizations with larger, dedicated portal teams consistently report modestly higher satisfaction.

Roadie: SaaS Delivery Without Closed-System Tradeoffs

Backstage established the architectural standard for developer portals: a plugin-first model, a structured software catalog, and an ecosystem of integrations. But consuming a standard doesn’t require operating the framework directly.

Roadie is a SaaS developer portal built on the open standard and inspired by its ecosystem. It enables the architectural advantages that made Backstage the default framework like a plugin-based architecture, a catalog-driven context and an ecosystem integration model, then elevates them into a product designed for long-term context stability, governance, and scale.

This matters because most organizations don’t want to become framework maintainers, but they also don’t want their portal to become a closed system with a vendor-owned schema. Roadie offers this alternative: the architectural upside of the standard, with a SaaS product delivery.

Architecture: SaaS Built on the Open Standard (Not a Hosted Distribution)

Roadie is a SaaS developer portal built on the open standard, but it isn’t simply a hosted version of Backstage.

Roadie delivers a SaaS portal experience with hosting, databases, monitoring, security controls, and upgrades, while retaining the Backstage ecosystem approach that made extensibility and integration portability possible in the first place. Roadie can absorb ecosystem change, improve workflow UX, and provide enterprise operational guarantees, including SOC 2 Type 2, without requiring customers to become framework maintainers.

What Roadie Optimizes (and Why It’s a Distinct Third Model)

Roadie is designed for teams who want the architectural upside of the Backstage standard without turning the portal itself into an internally operated framework.

From the Backstage ecosystem, Roadie inherits the most valuable properties: an extensible integration model and a large body of community and partner plugins. Teams can adopt integrations incrementally and extend the portal over time, instead of being limited to a fixed set of vendor-owned connectors.

Importantly, extensibility isn’t restricted to pre-approved marketplace integrations. Teams can build their own plugins and integrations using the same plugin-oriented model to connect internal systems, domain-specific tooling, and organization-specific workflows. Extensibility also applies to automation: platform teams can also extend self-service workflows through Scaffolder templates, including custom actions that integrate directly with internal tooling and automation systems. This means the portal can evolve alongside your architecture and operating model, rather than being constrained by a vendor-defined capabilities.

Roadie productizes areas where a framework deployment typically requires code-first implementation. For example, access control and governance can be configured through product workflows rather than requiring teams to implement RBAC policy logic directly in code. Secrets and integration configuration can be managed through dedicated interfaces instead of being pushed into environment variables and bespoke deployment pipelines.

From an operational perspective, Roadie owns the lifecycle that comes with self-hosting Backstage: framework upgrades, compatibility testing, security patching, infrastructure operations, and production monitoring. In a self-hosted model, each of these becomes ongoing internal engineering work - work that competes directly with building new developer workflows, improving golden paths, or investing in higher-leverage platform initiatives.

This goes beyond “hosting”. Roadie tracks changes across the Backstage ecosystem, including core framework releases and plugin-level updates, which are evaluated and incorporated into the Roadie product lifecycle. Platform teams don’t need to plan and execute backend migrations, reconcile plugin API changes, or coordinate dependency upgrades across their portal deployment as the ecosystem evolves.

In practical terms: self-hosted Backstage makes your team responsible for maintaining the framework itself. Roadie shifts that responsibility boundary: your team focuses on extending and using the portal, not operating it.

The result is a SaaS-delivered portal that remains aligned with Backstage’s extensibility model and ecosystem approach, without requiring customers to maintain the framework itself.

Technical Detail: Addressing Backend System Migration and Upgrade Burden

Backstage's architecture evolved significantly with the introduction of the new backend system, which reached stable 1.0 in 2024. For self-hosted teams, this wasn’t a minor version bump. It was a migration-class change that touched plugin wiring, backend dependencies, and integration compatibility.

Plugin registration changed from imperative setup to declarative backend.add() calls. The legacy @backstage/backend-common package was deprecated in favor of @backstage/backend-defaults. Service-to-service communication patterns shifted to dependency injection. Configuration structure changed. In practice, this meant reviewing custom plugins, updating imports, reconciling breaking changes, and validating that integrations continued to function correctly.

For teams running Backstage themselves, migrations like this become internal engineering projects. They require planning, testing, staged rollouts, and often temporary freezes on other platform work. This aligns with findings from the 2025 State of Backstage report: "Teams running Backstage themselves describe a very different day to day reality compared to those using managed platforms, especially around stability and upgrades."

Roadie evaluates changes in the core Backstage framework and plugin ecosystem, tests them against supported integrations and configurations, and implements the necessary adjustments within the platform itself.

In this case, Roadie absorbed this migration into the product lifecycle. Changes in the core Backstage framework and plugin ecosystem were evaluated, tested against supported integrations and real customer plugin combinations, and the necessary compatibility work was implemented within the platform itself. Customers did not need to refactor backend code, reconcile plugin API changes, or coordinate dependency upgrades across their portal deployment. Upgraded instances were delivered without requiring each team to manage the migration work.

This approach scales. As Backstage evolves, SaaS providers can absorb the upgrade complexity without customers modifying their configurations or code.

Conclusion: Choosing Your Architecture

The choice between frameworks and products isn't about features. It's about ownership and long-term flexibility.

Proprietary portals offer speed: you get a working portal quickly andavoid operational complexity.But that speed usually comes from a vendor-owned model: the schema, workflow layer, and extensibility boundaries are defined by the product. If your organization fits those assumptions, this can be a great trade.

Self-hosted Backstage offers portability and control: your data model isn't locked to a vendor and you can integrate with any tool through the plugin system. But you also take on the full cost of ownership: upgrades, migrations, security, scaling, plugin compatibility, and the ongoing work of maintaining a rapidly changing framework.The operational burden is real. For many teams, it becomes the limiting factor. Not because Backstage is flawed, but because running it well requires sustained platform engineering investment.

Roadie represents a third model: a SaaS portal experience aligned with Backstage’s architectural principles, without requiring customers to operate the framework. Roadie carries forward the architectural lineage of Backstage, including plugin-based extensibility, ecosystem familiarity, a catalog-driven foundation, and workflow flexibility, while taking responsibility for upgrades, migrations, infrastructure, and long-term lifecycle management. It’s a different responsibility boundary: teams keep the architectural upside of the standard, while offloading the operational and upgrade burden that makes self-hosting expensive over time.

The portal you choose becomes part of your platform architecture, and the decisions you make about extensibility, portability, and ownership will shape your strategy for years. Choose a model aligned with how you want your engineering organization to operate.

The industry spent the last two years obsessed with prompt engineering. Teams refined instructions, added chain-of-thought reasoning, and built elaborate system prompts. None of that addresses the real issue: your AI doesn't know your company. It knows the world, but it doesn't know your world. Thankfully, context engineering can help, and it's going to define the next wave of platform engineering work.

What Is Context Engineering

The working definition of context engineering is the systematic practice of curating, structuring, and retrieving information to ground AI models in specific domain knowledge.

The roots of context engineering go back to context-aware computing from the early 1990s, when researchers like Bill Schilit started building systems that could adapt their behavior based on location, user, and environment. The insight was the same then as now: the value of any intelligent system scales with the quality of context it can access. RAG architectures, popularized by the 2020 Lewis et al. paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," brought this concept into the LLM era. But RAG is just the retrieval mechanism. Context engineering is the discipline of deciding what to retrieve, how to structure it, and which sources to trust.

This difference matters because the industry is moving from model-centric thinking ("we need a better model") to context-centric thinking ("we need better data retrieval"). GPT-4 Turbo and Claude 3.5 Sonnet are good enough for most coding tasks. The bottleneck has shifted from model intelligence to grounding.

Context Engineering vs. Prompt Engineering

For the past two years, teams have focused on prompt engineering, refining instructions so models behave correctly. That works when the problem is ambiguity or reasoning quality. It fails when the model simply lacks the right information.

No prompt can reliably answer:

Who owns checkout-service?

What lifecycle state is legacy-auth-api in?

What security constraints apply to payment-service?

If the data isn't accessible through a retrieval layer, the model will infer, and sometimes hallucinate.

Prompt engineering optimizes how you ask. Context engineering optimizes what the model knows.

The Three Layers of Context

When your developers use GitHub Copilot or Cursor, those tools are solving a layered context problem, and they're only solving part of it.

Layer 1 is local context: the file you have open, the function you're editing, the variables in scope. This is what language servers (LSP) have always done, and what Copilot does well. The model sees your current cursor position and the surrounding tokens. For greenfield code written against public libraries, this is often enough.

Layer 2 is repository context: the patterns, structures, and dependencies in the current codebase. Cursor's codebase indexing handles much of this, using vector embeddings to make local code semantically searchable. You can ask "how does the authentication middleware work?" and get a coherent answer drawn from files across the repo.

Layer 3 is organizational context, and this is where every current AI coding tool struggles. Organizational context is the knowledge that lives outside any single repository:

• Who owns payment-service and what's its current lifecycle status?
• What's the approved pattern for creating a new S3 bucket with FIPS-compliant encryption?
• Where's the API spec for the internal event bus, and what events does order-service emit?
• What's the SLA tier for fraud-detection-service and what are its production constraints?

No amount of clever prompting retrieves this information. It doesn't exist in any repo. It lives in your organization's institutional knowledge, and without a structured home, it's functionally invisible to any AI system.

The most natural system to solve Layer 3 is the Internal Developer Portal (IDP).

Why the IDP Is the Natural Context Engine

Your IDP already contains the two things an AI needs most: metadata and semantics.

Metadata lives in catalog-info.yaml, the structured backbone of every Backstage service entry. Each record carries owner, tier, lifecycle, dependencies, and tags in a machine-readable, version-controlled format. That's exactly the information an AI needs to answer "what are the production constraints on payment-service?" without guessing. The answer isn't buried in a README or a Slack thread. It's in a schema-enforced file your platform team already maintains as part of normal operations.

Semantics live in TechDocs: your architectural decision records, how-to guides, runbooks, and onboarding documentation. When a developer asks Cursor, "What's the right approach for adding distributed tracing to a new Go service?", the correct answer exists in your TechDocs, not on Stack Overflow. TechDocs is already your single source of truth for how things work here. It needs to be your AI's source of truth too.

Spotify proved this architecture works at scale. Their internal "AiKA" (AI Knowledge Assistant) and "Honk" background coding agents both integrate deeply with Backstage's catalog and metadata graph. Spotify didn't need to build a separate AI knowledge layer; the IDP was already taking care of that. The IDP's role as a context engine follows directly from its architecture: it's the only system in your organization that maintains structured, governed, and continuously updated metadata about every service you run.

Because Roadie is managed Backstage, this architectural pattern is available to you without the overhead of building and maintaining the context infrastructure yourself. The Spotify engineering team spent significant effort wiring Backstage into AiKA and Honk. Roadie ships that foundation.

Connecting AI Tools via the Model Context Protocol

Knowing that the IDP holds the right data is one thing; getting AI tools to query it reliably is another. That's where the Model Context Protocol (MCP) comes in.

MCP is USB-C for AI. Before USB-C, every device had its own connector. You needed a drawer full of adapters for every combination. Before MCP, every AI tool had its own proprietary method for connecting to external data sources, if it connected at all. MCP is the open standard that lets any compliant AI client (Cursor, Claude Desktop, a custom agentic pipeline) connect to any compliant MCP Server and query its data through a consistent interface.

Roadie acts as an MCP Server, exposing your Backstage catalog and TechDocs as queryable endpoints. Here's what that looks like in practice:

1. A developer opens Cursor and asks: "Generate a Terraform config for payment-service that meets its production SLA requirements."
2. Cursor's MCP client queries Roadie's MCP server for the payment-service catalog entry.
3. Roadie returns structured metadata: tier-1 service, multi-region deployment, PCI-compliant, owner: group:payments-team, depends on fraud-detection-service and event-bus.
4. Cursor incorporates this context into the generation, auto-applying the correct instance types, encryption configuration, and cross-region failover settings.

Without MCP and a populated catalog, the developer gets a generic Terraform template that needs manual adjustment and a second round-trip to whoever actually knows the service requirements. With it, the AI generates something correct for this service, at this organization. That's the difference between an AI assistant and an AI that actually knows where it works.

The Problem with "Just Dump It in a Vector DB"

Here's an approach I've seen teams try when they get serious about AI grounding: export everything (Confluence pages, Jira tickets, Slack threads, internal wikis), embed it all in Pinecone or Weaviate, and call it a "context lake."

This doesn't work. Vector databases are great tools. The issue is that unstructured, uncurated data produces what I'd call "context poisoning".

Your Confluence instance has 2,000 pages. Maybe 300 are accurate. Another 400 are outdated, written before the infrastructure migration, the reorg, or the service's deprecation. The remaining 1,300 are drafts, duplicates, or meeting notes that were never meant to be authoritative. When your AI retrieves from this pool, it has no way to distinguish the canonical architectural decision record from the three-year-old wiki page that contradicts it. The model doesn't know that legacy-payment-gateway-setup.md was superseded by a new doc eighteen months ago. So it pulls from both, fuses them, and produces output that's partly right and partly dangerously wrong.

This is the risk with some proprietary "context lake" approaches: you can't engineer good context from bad source data, no matter how sophisticated your retrieval layer is. And Cortex's scorecard-based "AI readiness checks" are a step forward, enforcing that services have owners and documentation, but scorecards tell you whether context exists, not whether it's accurate or semantically coherent.

The catalog-info.yaml schema enforces structure at the source. Every service entry has a defined owner (not "the payments team" but group:payments), a machine-readable lifecycle status (production, deprecated, experimental), and explicit dependency links. A deprecated service can't masquerade as current because its lifecycle field says otherwise. An unmaintained entry gets surfaced by catalog health checks before it becomes a source of bad context. The data is searchable and governed.

That said, this does require your catalog to be accurate. A Backstage instance with stale catalog-info.yaml files is its own form of context poisoning. The discipline of context engineering starts with the discipline of catalog maintenance.

Final Thoughts: Building the Map for Your Digital Workforce

The agentic AI era is arriving faster than most platform teams are ready for. Chatbot-style AI assistants are forgiving. A slightly wrong answer gets ignored and corrected. Agents that take action aren't forgiving. An agent that auto-generates a pull request, provisions cloud infrastructure, or routes a production incident based on service ownership data needs that data to be correct. The cost of a context error scales directly with the autonomy of the system making decisions from it.

The teams preparing well are focusing on reliable organizational context, not model sophistication or GPU budgets. The catalog entries you keep accurate, the TechDocs you maintain, and the ownership mappings you get right form your AI workforce's institutional memory. That's what your AI tools actually run on.

Context engineering is platform engineering's next frontier. The good news is that if you've been running Backstage, you're further along than you think. You have the schema, the governance model, and the data. You just need to connect it.

Your AI needs a brain. Start building your organization's memory today with Roadie's managed Backstage platform. Book a demo to see how your Service Catalog can become your context engine.

This glossary defines every key term in the context engineering stack through the specific lens of platform engineering — Service Catalogs, TechDocs, golden paths, and on-call data — not abstract data science. Bookmark it, share it with your team, and use it as a reference before your next architecture decision.

This glossary focuses on context supply, not model training, fine-tuning, or prompt copywriting. Context engineering does not make models smarter — it determines what the model is allowed to know, when, and why. If an answer is wrong, the first place to look is rarely the model; it’s the data pipeline feeding it.

Section 1: Context Fundamentals

What Is Context Engineering?

Context engineering is the practice of curating, structuring, and retrieving the right infrastructure data so that an LLM can answer domain-specific questions accurately. The word choice matters: it's engineering, not prompting. Where prompt engineering focuses on the wording of individual queries, context engineering focuses on the entire information supply chain that feeds the model before it generates a word.

For platform teams, context engineering means deciding which fields from your catalog-info.yaml get indexed, how your TechDocs chunks get sized and tagged, and what real-time operational signals get injected at query time. A developer asking "Is the payments service production-ready?" gets a useful answer only if the lifecycle field from the Service Catalog was curated, indexed, and retrieved correctly. The LLM itself contributes maybe 20% of that answer's quality; context engineering accounts for the rest.

An LLM is a powerful reasoning engine with no institutional memory. Context engineering is how you give it one. In other words, context engineering is not about improving reasoning quality — it’s about constraining the information surface the model can reason over.

What Is a Context Window?

The context window is the total number of tokens an LLM can process in a single request, covering the system prompt, retrieved documents, conversation history, and the generated response combined. GPT-4o supports up to 128,000 tokens, and Gemini 3.5 Pro pushes to 1 million tokens. These numbers sound large until you picture a Service Catalog with 800 registered components, each with full metadata and linked TechDocs pages.

Stuffing everything into the context window is not a strategy. Irrelevant data degrades output quality, increases cost per query (models like Claude charge per input token), and slows response time. The engineering discipline is in selecting the right 2,000 tokens out of 2,000,000 available, pulling only the service metadata relevant to the specific query, not the entire catalog.

Efficient context selection is where retrieval architecture pays for itself.

What Is Grounding in LLMs?

Grounding anchors an LLM's response in verified, authoritative data sources rather than the model's pre-trained weights. Without grounding, a model answering "Who is the on-call engineer for the checkout service?" will either hallucinate a plausible name or admit it doesn't know. With grounding, the response comes from the live PagerDuty schedule injected at query time.

In a platform engineering context, your Service Catalog is the primary grounding layer. When every answer the AI gives traces back to a specific entity in the catalog, with a citable source, you've achieved grounded output. Ungrounded AI assistants erode trust fast: one invented service name or wrong runbook link and developers stop using the tool. RAG is an architectural mechanism; grounding is the result. You can implement RAG without achieving grounding if the retrieved data isn’t authoritative or current.

Section 2: Architecture and Retrieval Terms

RAG (Retrieval-Augmented Generation)

Retrieval-Augmented Generation (RAG) is the architectural pattern where a system retrieves relevant documents from an external knowledge base before passing them to an LLM for response generation. The model doesn't rely on what it learned during training; it reads what you give it at runtime.

The flow for an IDP-backed assistant looks like this:

A developer asks: "How do I rotate credentials for the auth service?" The system encodes that query, searches TechDocs for credential rotation guides tagged to the auth service, pulls the service owner from the catalog, and injects both into the prompt. The LLM generates a specific, sourced answer, not a generic "here's how credential rotation works" response scraped from its training data.

RAG is the foundational pattern for any AI assistant built on top of an IDP. Every other term in this glossary relates to how well your RAG implementation performs.

What Are Vector Embeddings?

A vector embedding is a numerical representation of text, typically a list of 768 to 3,072 floating-point numbers, that captures semantic meaning rather than just the words themselves. Two sentences that mean the same thing will have similar embeddings even if they share no words. "Service is deprecated" and "component has reached end-of-life" end up close together in embedding space; "deploy to production" and "YAML syntax error" end up far apart.

To build RAG for your IDP, every TechDocs page, every catalog entity description, and every relevant metadata field needs to be converted into an embedding and stored. When a developer submits a query, the query also gets embedded, and the system retrieves the stored documents whose embeddings are most similar. That's semantic search.

Generating and managing these embeddings is non-trivial. You need to pick an embedding model (OpenAI's text-embedding-3-large or a self-hosted Sentence Transformers variant), decide chunk sizes, handle incremental updates when docs change, and keep embeddings in sync with the underlying catalog. Roadie handles this entire pipeline automatically for TechDocs on your managed Backstage instance. You don't maintain a separate embedding job or manage model versions.

What Is a Vector Database?

A vector database is a storage engine purpose-built for indexing and querying high-dimensional embedding vectors. It provides Approximate Nearest Neighbor (ANN) search at scale, which means it can find the 10 most semantically similar chunks from a corpus of 500,000 embeddings in under 100 milliseconds. Standard relational databases like PostgreSQL can store vectors (via pgvector), but dedicated systems like Pinecone, Weaviate, and Qdrant are optimized for this workload.

For platform teams evaluating AI tooling, the vector database is an infrastructure dependency that often gets underestimated. It requires provisioning, access control, index tuning, and synchronization with your source catalog. When Roadie embeds your TechDocs, the vector storage layer is managed within the platform. You're not standing up a Qdrant cluster alongside your Backstage deployment.

What Is Semantic Search?

Semantic search finds content based on meaning and intent, not keyword overlap. In an IDP context, it's the difference between a developer searching for "payment processor" and finding the checkout-service, billing-api, and stripe-gateway components — even though none of them are literally named "payment processor" — versus a keyword search that returns zero results because the exact string doesn't match any component name.

This matters especially for large catalogs and for developers who are new to the codebase. They don't know the internal naming conventions. They describe what they're looking for in plain English. Semantic search over vector embeddings bridges the delta between how developers think and how services are named.

On its own, semantic search is insufficient for an AI assistant — it retrieves candidates, but the Service Catalog determines which of those candidates are valid, owned, and safe to surface.

Section 3: Platform Data Types (The Context Sources)

Service Catalog Context

Service Catalog context is the structured metadata that lives in your catalog-info.yaml files and gets surfaced through the Backstage catalog API. Fields like owner, lifecycle, tier, tags, system, and dependsOn are machine-readable facts that give an LLM the authority to answer structural questions.

"Who owns the recommendations engine?" gets answered from the owner field. "Is this service production-ready?" gets answered from the lifecycle: production tag. "What services would be affected if the user-profile API went down?" gets answered from dependency relationships in the catalog graph. This data is already structured, already maintained (or should be), and it's the highest-signal context source you have. Poor catalog hygiene directly degrades AI output quality.

TechDocs Context

TechDocs context is unstructured markdown documentation that lives alongside your service code and gets rendered in Backstage TechDocs. It answers the "how" questions that structured catalog metadata can't: how to run the service locally, how to interpret a specific error code, how to onboard to the payments team's workflow.

When ingested into a RAG system, TechDocs pages get chunked (typically into 512-token segments with overlap), embedded, and indexed against their source entity. A developer asking "What does a 503 from the auth service usually mean?" should retrieve the relevant troubleshooting section from the auth service's TechDocs, not a generic HTTP guide. The specificity of the retrieval depends entirely on how well TechDocs are written and tagged. Vague documentation produces vague answers.

Operational Context

Operational context is real-time data injected at query time rather than pre-indexed into a vector database. It includes current PagerDuty on-call schedules, Kubernetes pod health and restart counts, recent Argo CD deployment status, open Jira incidents, and GitHub Actions build logs.

This data changes too fast for batch indexing to keep up. Instead, you pull it live via API calls triggered by the query itself. A developer asking "Why is checkout slow right now?" needs the current K8s resource utilization for the checkout pods, not the documentation about checkout's architecture. Mixing pre-indexed catalog and TechDocs context with real-time operational context is what separates a genuinely useful AI assistant from a documentation search engine.

Operational context informs decisions; it does not imply automated remediation unless explicitly authorized. Observing live state and acting on it are separate trust boundaries.

Golden Path Context

Golden path context comes from your Backstage Scaffolder templates, the opinionated, pre-approved patterns your platform team maintains for creating new services, adding CI/CD pipelines, or spinning up databases. This context feeds the AI's code generation and workflow guidance capabilities.

When a developer asks "How do I create a new Python microservice that follows our standards?" the answer shouldn't come from a generic tutorial. It should come from your actual Scaffolder template, including your team's specific conventions around naming, logging configuration, health check endpoints, and observability setup. Golden path context ensures that AI-assisted code generation produces output that passes your internal review standards on the first attempt.

Section 4: Agentic Capabilities

What Is Agentic Context Injection?

Agentic context injection is the dynamic process by which an AI system decides which data sources to query based on the intent of the user's question, rather than fetching a fixed set of context for every request. It's the difference between a system that always retrieves the top-10 catalog entries regardless of the question, and a system that recognizes "my build is failing" as a signal to pull CI/CD logs, not architecture documentation.

A well-designed agentic system routes queries through an intent classifier first. Questions about ownership route to the catalog API. Questions about procedures route to TechDocs embeddings. Questions about current system state trigger live operational data calls. This routing logic is itself a form of engineering. It determines response latency, token cost, and answer relevance simultaneously.

Without strict boundaries, agentic retrieval increases blast radius: every additional tool or data source expands what the system can surface or misuse. Intent routing must be auditable, deterministic, and permission-aware to be safe in production.

Tool Use and Function Calling

Function calling is the capability that allows an LLM to request the execution of a predefined function, a structured API call, rather than generating a text answer directly. The model outputs a JSON object specifying which function to call and with which parameters; your application executes the call and feeds the result back to the model.

For IDP AI assistants, function calling turns the LLM into an active participant in your platform's API surface. Instead of the model trying to recall what it knows about a service's on-call engineer, it calls get_oncall_for_service(service_id="checkout"), gets a live response from PagerDuty, and incorporates that response into its answer. Functions you'd expose typically include catalog entity lookup, TechDocs page retrieval, incident history queries, and deployment status checks. The LLM becomes a reasoning layer over your actual infrastructure data.

What Is a System Prompt?

The system prompt is the foundational instruction block prepended to every conversation with the AI assistant. It defines the model's persona (a senior platform engineer, not a general assistant), its constraints ("only answer questions about services in this catalog"), its output format preferences, and its access permissions.

For a platform assistant, the system prompt is effectively a policy document. It specifies that the model should cite its sources, decline to speculate about services not in the catalog, and escalate ambiguous ownership questions to a human. A weak system prompt produces an assistant that will confidently make things up. A well-engineered system prompt is a first line of defense against the risks described in the next section. In practice, the system prompt is inseparable from access control. It should reflect the same RBAC assumptions as the IDP itself — otherwise the model’s behavior will drift from the platform’s security model.

Section 5: Quality and Risk Definitions

What Is LLM Hallucination?

Hallucination is when an LLM generates information that is factually incorrect but presented with full confidence. In a platform engineering context, hallucinations take a specific and damaging form: the model invents service names, fabricates runbook steps, cites non-existent on-call rotations, or describes API contracts that don't match the actual implementation.

The primary defense against hallucination is grounding (see above), combined with explicit system prompt instructions to cite sources. If the model's answer can't be traced to a specific catalog entity or TechDocs page, it shouldn't be trusted. Measuring hallucination rate by sampling model responses against the catalog is a useful quality metric for AI-enabled IDP rollouts.

What Is Context Drift?

Context drift is the discrepancy between the data the AI has indexed and the actual current state of your infrastructure. A TechDocs page describing the old three-tier deployment model that your team migrated away from six months ago is a context drift problem. A catalog entry with a stale owner field pointing to a team that was reorganized is another.

Context drift is not a one-time fix. It's an ongoing operational concern. The mitigation is a combination of automated re-indexing (triggering embedding updates when catalog-info.yaml files change) and documentation standards that treat TechDocs as a first-class engineering artifact. An AI assistant is only as current as the data it reads. If your catalog hygiene is poor, context drift will silently produce incorrect answers with no obvious signal that something is wrong.

What Is Context Poisoning?

Context poisoning occurs when low-quality, contradictory, or maliciously crafted documentation gets retrieved and influences the model's output. Two TechDocs pages for the same service that give conflicting deployment instructions will cause the model to blend them into a response that's confidently wrong. A poorly maintained runbook that describes a procedure deprecated two years ago is a context poisoning vector.

The solution is content governance: ownership requirements for every TechDocs page, last-reviewed timestamps surfaced in the catalog, and automated quality checks that flag documentation not updated in over 90 days. The AI doesn't discriminate between trusted and untrusted docs. The retrieval system surfaces whatever scores highest semantically. You own the quality of what gets indexed.

What Is Context Overreach?

Context overreach happens when you inject too much data into the prompt, including irrelevant retrieved chunks that dilute the signal and confuse the model. A developer asking about the auth service's rate limits doesn't need context from the billing service's TechDocs, even if billing is a downstream dependency. Retrieving ten chunks when three would suffice increases token cost, slows the response, and statistically introduces off-topic content that nudges the model toward a less precise answer.

The fix is tighter retrieval: stricter similarity thresholds, metadata filtering (retrieve only docs tagged to the queried service), and re-ranker models that score retrieved chunks for relevance before they enter the prompt. Context budgeting, deciding in advance how many tokens each source type is allowed to consume, is a practical starting point.

What Is Privilege Leakage in AI Systems?

Privilege leakage occurs when the AI assistant returns information about services, infrastructure, or documentation that the querying user shouldn't have access to, because the retrieval layer doesn't enforce the same Role-Based Access Controls (RBAC) as the IDP itself. A junior engineer asking a general question about "our database infrastructure" shouldn't receive details about the security team's secrets management service, even if that service's TechDocs scored highly in the semantic search results.

Preventing privilege leakage requires that your retrieval pipeline filters indexed documents by the user's Backstage permissions before returning results. It's not enough to apply RBAC at the catalog UI layer; the vector search results that feed the LLM must respect the same access policies. This is one of the most commonly overlooked security requirements in IDP AI implementations.

What Are Implicit Trust Chains?

An implicit trust chain forms when a document retrieved as context itself references other documents — runbooks, architecture decision records, external wikis — that are outdated, incorrect, or not indexed by the retrieval system. The model reads the retrieved doc, which cites "the standard deployment procedure in the ops runbook," but the ops runbook lives in Confluence and isn't indexed. The model either ignores the reference, invents what it thinks the runbook says, or generates an incomplete answer.

Auditing your documentation for external references and either bringing those references into your indexed corpus or explicitly removing the links is a necessary part of context engineering. Every document in your retrieval index is implicitly vouching for everything it cites.

The pattern running through every definition in this glossary is simple: context engineering is now a core platform responsibility, not an AI feature bolted on at the edge. LLMs are capable of sophisticated reasoning, but they reason over whatever you give them. Platform teams that invest in clean catalogs, maintained TechDocs, and well-governed golden paths aren't just doing good hygiene. They're building the infrastructure that makes AI actually work.

But, this model also introduces some operational complexity. Architectural decisions that are harmless at a small scale can cause issues when every tenant is running their own Backstage stack. One such decision was how we deployed TechDocs as part of the same backend service as everything else.

In this article, we'll talk about why our original approach stopped scaling, how we redesigned it, and what improved when we split TechDocs out of the monolithic backend.

Our Original Architecture

In our original setup, each Roadie tenant ran a complete Backstage application composed of all frontend and backend plugins bundled together. On the backend side, everything was executed within a single Node.js process.

This meant that for each customer, a dedicated Kubernetes namespace was created to ensure isolation, a single Backstage backend pod was deployed into that namespace, and all backend plugins like Catalog, Scaffolder, TechDocs, and Auth were loaded into the same backend service.

From an architectural standpoint, this resulted in a classic monolith. Every backend plugin shared the same runtime, memory space, CPU limits, and lifecycle. This design was simple to operate and reason about early on, and it served us well for a long time. But, as customer usage patterns evolved, issues began to appear.

The Problem: Resource Contention

As more customers adopted TechDocs and began publishing larger documentation sites, we started receiving alerts that were difficult to explain at first glance.

These incidents typically involved brief periods during which the Backstage backend became unavailable, CPU usage exceeded configured limits, and Kubernetes restarted backend pods from resource exhaustion. What made this particularly challenging was that the failures were intermittent and tenant-specific. Many tenants were completely unaffected, while a handful experienced repeated disruptions, which made it harder to pinpoint the issue.

After analyzing metrics, logs, and pod-level behavior, a pattern emerged. In every affected case, the TechDocs backend plugin was consuming a disproportionate amount of CPU and memory.

Why TechDocs Was the Culprit

We noticed that issues only occurred for tenants with particularly heavy TechDocs usage. This included large documentation sites with many pages and assets, repositories containing multiple documentation sets, and frequent rebuilds triggered by ongoing documentation updates.

This behavior is expected when you look at what TechDocs does under the hood. The backend is responsible for fetching documentation files, rendering Markdown content, and running documentation generators like MkDocs. These tasks are inherently resource-intensive, especially during large or frequent builds.

When TechDocs runs in the same process as the rest of the Backstage backend, resource spikes are not contained. CPU saturation or memory pressure caused by documentation builds directly impacts unrelated functionality, including catalog ingestion, scaffolder workflows, and authentication. The monolithic design of TechDocs became a liability.

The Decision: Split TechDocs Out

To restore stability and regain operational control, we decided to extract TechDocs from the monolithic backend and deploy it as a separate Backstage backend application. The main reason for this was isolation. We wanted TechDocs to operate independently so that its workload characteristics would not interfere with the rest of the system. At the same time, we wanted to avoid breaking existing APIs or introducing fragile custom integrations.

Our TechDocs requirements were simple:

• It needed to be independently deployable so it could evolve on its own schedule.
• It needed to be discoverable by the core backend without hardcoded configuration.
• It needed to scale independently, based on documentation workload rather than overall backend traffic.

The New Architecture

In the new design, each tenant runs two distinct Backstage backend services instead of one.

The first is the core Backstage backend. This service is responsible for handling catalog ingestion, Tech Insights, authentication, and other core APIs.

The second is a dedicated TechDocs backend. This service runs only the TechDocs-related functionality and handles documentation builds and rendering.

The two services communicate using Backstage’s built-in discovery mechanism.

Results

Cleaner Cluster Allocations

By running TechDocs in its own pod, we gained fine-grained control over its resource profile. CPU and memory limits are now explicitly tuned for documentation workloads, and scaling rules can be applied only where documentation usage justifies it.

This prevents overprovisioning the core backend while still allowing TechDocs to scale aggressively when needed.

Improved Stability

Isolating TechDocs eliminated an entire class of failures. Documentation builds no longer put pressure on unrelated backend functionality. Catalog ingestion, scaffolder executions, authentication flows, and core API availability remain stable even during peak documentation activity.

For customers, this translates directly into fewer outages and a more predictable Backstage experience.

Easier Debugging and Operations

From an operational perspective, separating TechDocs clarified boundaries. Resource spikes are now immediately attributable to the correct service. Logs are easier to interpret, and incidents can be diagnosed and mitigated quickly.

This separation also simplifies future tuning and capacity planning. With TechDocs isolated, we can reason about its resource usage independently from the rest of the backend and make decisions based on real workload characteristics rather than worst-case assumptions. CPU and memory requests can be adjusted specifically for documentation builds, and autoscaling policies can be tuned around build frequency, repository size, and peak documentation activity. This also makes forecasting easier: growth in documentation usage no longer forces us to overprovision the core backend. Instead, we can scale and optimize each service independently, reducing wasted capacity while maintaining predictable performance.

What We Gained Overall

Splitting TechDocs out of the monolith forced us to formalize a clear boundary between core platform responsibilities and workload-specific plugins, which in turn improved how we think about backend composition overall. What started as a targeted stability fix turned out to have broader implications for how we structure the backend.

As a result, we now have a repeatable pattern for extracting heavy backend plugins into independent services when their resource profiles or failure modes warrant it. Backend deployments are slimmer, responsibilities are better defined, and each service can be sized and scaled according to the work it actually performs rather than the worst-case behavior of a single plugin. This makes the system easier to reason about both during normal operation and when something goes wrong.

Takeaways

TechDocs was the most obvious candidate for separation due to its workload profile, but this architecture opens the door to further modularization where it makes sense. Backstage provides the primitives needed to support this kind of design. At scale, using them becomes less of an optimization and more of a requirement.

If you're running Backstage in a multi-tenant or high-scale environment and are seeing similar symptoms, it's worth examining your heaviest backend plugins and questioning whether they belong in the same process as the rest of Backstage.

This guide covers every integration surface between Roadie and GitLab, from auto-discovery of catalog entities across hundreds of repos to CI/CD visualization, scaffolding new services, enforcing governance via scorecards, and securely connecting self-managed GitLab instances through the Roadie Broker.

Set Up Roadie with GitLab: A Step-by-Step Guide

Connecting GitLab to Roadie takes about 30 minutes. You'll create a token, configure auto-discovery to populate your Software Catalog, enable CI/CD visibility, and build your first scaffolder template.

Prerequisites

Before you start, make sure you have:

• Admin access to your Roadie instance (URL format: https://<your-tenant>.roadie.so)
• A GitLab account with permissions to create Personal Access Tokens
• Access to at least one GitLab group or project containing repositories.

Step 1: Create and Configure Your GitLab Token

GitLab uses Personal Access Tokens to authenticate API requests. Roadie needs this token to read repositories, discover catalog entities, and interact with CI/CD pipelines.

Backstage needs a GitLab API token for discovery and plugin data. Three token types exist, each with different trade-offs:

Group Access Tokens are the recommended choice for Backstage integrations. They create a bot user scoped to a specific group and all its projects, don't consume a GitLab license seat, and aren't tied to a human user (avoiding breakage when someone leaves the organization). They require GitLab Premium or Ultimate on SaaS. Personal Access Tokens (PATs) provide broader instance-wide access but inherit the creating user's permissions and break if that user is deactivated. Project Access Tokens are too narrow for discovery since you'd need one per repository.

The required scopes depend on the use case. For read-only catalog discovery and CI/CD visualization, read_api is sufficient. For scaffolder operations that create repositories and merge requests, the full set of api, read_repository, and write_repository is needed. Roadie stores tokens securely in its secrets management UI at https://<tenant>.roadie.so/administration/secrets, backed by AWS Parameter Store with per-tenant KMS encryption, rather than requiring them in plaintext config files.

Once you have your token, you can enter it into Roadie.

1. Log into your Roadie instance at https://<your-tenant>.roadie.so

2. Navigate to Administration > Secrets (direct URL: https://<your-tenant>.roadie.so/administration/secrets).

3. Find the secret named GITLAB_TOKEN

4. Click Edit and paste your GitLab token

5. Click Save

The secret update takes 2-3 minutes to propagate. You'll see the status indicator change from "Updating" to "Available" when it's ready.

Configure Auto-Discovery: eliminate manual catalog registration at scale

The core of any Internal Developer Portal is the software catalog. Without auto-discovery, teams must manually register every service, an approach that collapses at hundreds of repositories. Backstage'sGitlabDiscoveryEntityProvider, packaged in @backstage/plugin-catalog-backend-module-gitlab, crawls a GitLab instance, finds repositories containing a catalog-info.yaml file, and registers them automatically.

In self-hosted Backstage, this requires editing app-config.yaml directly:

catalog:
  providers:
    gitlab:
      production:
        host: gitlab.com
        branch: main
        fallbackBranch: master
        skipForkedRepos: true
        includeArchivedRepos: false
        group: my-org                     # Scope to a specific group
        groupPattern:
          - '^platform-.*$'               # Regex: only groups starting with "platform-"
          - 'services'
        projectPattern: '[\s\S]*'         # Regex: match all projects (default)
        entityFilename: catalog-info.yaml
        excludeRepos:
          - my-org/deprecated-service
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }

Roadie replaces this YAML editing with a UI-based configuration at https://<tenant>.roadie.so/administration/settings/integrations/gitlab. Admins add their GitLab instance URL, configure provider rules pointing to specific groups, and entities appear in the catalog within minutes.

The discovery provider supports powerful filtering through regex patterns. The groupPattern field accepts a list of regular expressions OR'd together to select which groups to scan. The projectPattern field applies a regex against each project's path_with_namespace. A legacy discovery processor also exists using wildcard URLs (https://gitlab.com/group/subgroup/blob/*/catalog-info.yaml, where * resolves to each repo's default branch), but the entity provider approach is the current recommended pattern.

For near-real-time updates, the provider supports webhook-driven ingestion, configure GitLab push webhooks to trigger incremental catalog refreshes instead of waiting for scheduled polls. Events can be received via HTTP endpoints, AWS SQS, Google Pub/Sub, or Kafka through @backstage/plugin-events-backend-module-gitlab.

To set up auto-discovery:

1. Navigate to Administration > Integrations & Plugins > GitLab (direct URL: https://<your-tenant>.roadie.so/administration/settings/integrations/gitlab)

2. In the Host field, enter your GitLab instance URL:

   • For GitLab.com: gitlab.com
   • For self-hosted: Your full domain (e.g., gitlab.company.com)

3. Leave API Base URL with the default vaule unless you're using a custom API endpoint

Now, you’ll need to add a discovery provider in Roadie. The discovery provider tells Roadie which GitLab groups or projects to scan for catalog files.

1. In the same GitLab integration page, scroll to Configure GitLab Discovery

2. Click Add Provider Configuration

3. Configure the provider. At the very least, you’ll need to enter your group name. You can also add filters to refine discovery. For example, you can add a project pattern.

4. Click Save

The discovery process runs every hour by default. To trigger an immediate scan, you can refresh the catalog or wait for the next scheduled run.

For Roadie to link catalog entities to GitLab data, you need to add GitLab-specific annotations to your catalog-info.yaml files.

Add one of these annotations to your entity metadata:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: my-service
  annotations:
    # Option 1: Use project ID (found in Settings > General in GitLab)
    gitlab.com/project-id: '12345'

    # Option 2: Use project slug (group/project format)
    # gitlab.com/project-slug: 'acme-corp/my-service'

    # Option 3: For self-hosted GitLab instances
    # gitlab.com/instance: 'gitlab.company.com'

The project ID is the most reliable option. You can find it in your GitLab project under Settings > General at the top of the page.

Now, auto-discovery should be enabled, and your catalog will be populated with projects from GitLab.

CI/CD visibility and contextual linking to GitLab

Once entities are cataloged, the @immobiliarelabs/backstage-plugin-gitlab provides rich GitLab data directly on each component's page in Backstage. This plugin reads the gitlab.com/project-slug or gitlab.com/project-id annotation and calls the GitLab API via a backend proxy to surface:

• Pipeline status table showing the last 20 builds with status (success/failed/running/pending), direct links to the GitLab pipeline URL, and execution time
• Merge requests table with open and recently merged MRs, linking directly to GitLab
• MR statistics for review velocity insights
• Contributors/people card, language breakdown, releases, code coverage, and rendered README

The plugin supports both cloud-hosted gitlab.com and self-managed GitLab instances (configured via the gitlab.com/instance annotation). Multiple GitLab instances can be configured simultaneously in Roadie, useful for organizations running both cloud and on-premise deployments or undergoing migrations between the two.

The GitLab plugin should already be installed in your Roadie instance, but you need to add it to your component layouts.

1. Navigate to any component in your catalog

2. Click the gear icon (⚙️) in the top right corner

3. Click the plus icon (+) to add a new card

4. Select GitLab from the plugin list:

   • EntityGitlabPeopleCard: Shows contributors and languages
   • EntityGitlabPipelinesTable: Shows recent pipeline runs
   • EntityGitlabMergeRequestStatsCard: Shows MR stats
   • EntityGitlabMergeRequestsTable: Shows MRs
   • EntityGitlabReadmeCard: Shows Readme
   • EntityGitlabLanguageCard: Shows repository languages
   • EntityGitlabReleasesCard: Shows recent releases

5. Drag the cards to arrange them in your preferred order

6. Click Save to apply the layout.

If you want consolidated GitLab data:

1. Click the plus icon (+) next to the existing tabs

2. Select EntityGitlabContent

3. Name the tab (e.g., "GitLab")

4. Click Save

This creates a comprehensive GitLab view with all available cards on one page.

Step 4: Create Your First Scaffolder Template

Software Templates in Backstage, powered by the Scaffolder plugin, let developers create new projects through forms that execute predefined actions.

Create a new file in one of your GitLab repositories at templates/basic-service.yaml:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: gitlab-basic-service
  title: Create a New Service (GitLab)
  description: Creates a new service repository in GitLab with a standard structure
spec:
  owner: user:default/<your-user>
  type: service

  parameters:
    - title: Service Information
      required:
        - name
        - owner
        - description
      properties:
        name:
          title: Service Name
          type: string
          description: Unique name for your service (lowercase, hyphens only)
          pattern: '^[a-z0-9-]+$'
        description:
          title: Description
          type: string
          description: What does this service do?
        owner:
          title: Owner
          type: string
          description: User that owns this service
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: [Group, User]

    - title: GitLab Configuration
      required:
        - repoUrl
      properties:
        repoUrl:
          title: Repository Location
          type: string
          ui:field: RepoUrlPicker
          ui:options:
            allowedHosts:
              - gitlab.com
            allowedOwners:
              - <your-group>

  steps:
    - id: fetch-base
      name: Fetch Base Template
      action: fetch:template
      input:
        url: https://gitlab.com/<your-group>/<your-repo>/-/tree/master/skeleton
        values:
          name: ${{ parameters.name }}
          description: ${{ parameters.description }}
          owner: ${{ parameters.owner }}
          repoUrl: ${{ parameters.repoUrl }}

    - id: publish
      name: Publish to GitLab
      action: publish:gitlab
      input:
        repoUrl: ${{ parameters.repoUrl }}
        defaultBranch: main

    - id: register
      name: Register Component
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: '/catalog-info.yaml'

  output:
    links:
      - title: Repository
        url: ${{ steps.publish.output.remoteUrl }}
      - title: Open in Catalog
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}

Create the Template Skeleton

In the same repository, create a skeleton directory with your template files:

skeleton/catalog-info.yaml:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.name }}
  description: ${{ values.description }}
  annotations:
    gitlab.com/project-slug: ${{ values.repoUrl | parseRepoUrl | pick('owner') }}/${{ values.name }}
spec:
  type: service
  lifecycle: experimental
  owner: ${{ values.owner }}

skeleton/README.md:

# ${{ values.name }}

${{ values.description }}

## Getting Started

[Add your setup instructions here]

## Owner

Maintained by: ${{ values.owner }}

skeleton/.gitignore:

node_modules/
.env
dist/

Register the Template

1. Create a catalog-info.yaml in your templates repository:

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: templates
  description: Scaffolder templates for our organization
spec:
  type: url
  targets:
    - https://gitlab.com/<your-group>/<your-repo>/-/blob/main/templates/basic-service.yaml

2. Register this location in Roadie:
   • Navigate to Catalog > Import
   • Paste the URL to your catalog-info.yaml
   • Click Analyze, then Import

If auto-discovery is configured for your templates repository, Roadie finds and registers them automatically.

Test Your Template

1. Navigate to Templates in your Roadie instance

2. Find your template: "Create a New Service (GitLab)"

3. Click Run and fill out the form:

   • Service Name: test-service
   • Description: A test service to verify template functionality
   • Owner: Select an owner from the dropdown
   • Repository Location: Provide a name for the repository

4. Click Review to see a summary of what will be created

5. Click Create to execute the template

The scaffolder will:

• Generate files from your skeleton directory
• Create a new GitLab repository
• Push the generated code
• Register the component in your Roadie catalog

You'll see a link to the new repository and the catalog entry when complete.

Add GitLab-Specific Actions

Roadie supports several GitLab-specific scaffolder actions beyond basic repository creation:

Create a merge request:

- id: create-mr
  name: Create Merge Request
  action: publish:gitlab:merge-request
  input:
    repoUrl: gitlab.com?owner=acme-corp&repo=my-service
    title: 'feat: Initial service setup'
    description: 'Automated setup via scaffolder'
    branchName: feature/initial-setup
    targetBranch: main

Add project variables:

- id: add-variables
  name: Add CI/CD Variables
  action: gitlab:projectVariable:create
  input:
    repoUrl: gitlab.com?owner=acme-corp&repo=my-service
    key: API_KEY
    value: ${{ parameters.apiKey }}
    masked: true
    variableType: env_var

Trigger a pipeline:

- id: trigger-pipeline
  name: Trigger Initial Pipeline
  action: gitlab:pipeline:trigger
  input:
    repoUrl: gitlab.com?owner=acme-corp&repo=my-service
    branch: main

Tech Insights enables governance at scale through GitLab API data

Tech Insights transforms ad-hoc governance (manual audits, spreadsheets of compliance) into continuous, automated checks surfaced directly in the developer portal. The system works through three layers: data sources (fact retrievers that call APIs), checks (boolean logic evaluating facts), and scorecards (collections of checks targeting entity subsets).

For GitLab integrations, common governance questions answered through the API include:

| Governance Check | GitLab API Endpoint | Data Type | |------------------|---------------------|-----------| | Repository has CODEOWNERS | GET /projects/:id/repository/files/CODEOWNERS | boolean | | Default branch protected | GET /projects/:id/protected_branches | boolean | | CI configuration exists | GET /projects/:id/repository/files/.gitlab-ci.yml | boolean | | MR approval rules configured | GET /projects/:id/approval_rules | boolean | | Project description set | GET /projects/:id → description field | boolean | | Open vulnerability count | GET /projects/:id/vulnerability_findings | number |

Roadie's managed Tech Insights provides 100+ built-in data source types and a UI for defining checks and scorecards without writing code. Scorecards can target entity subsets (e.g., "all production services must have CODEOWNERS and branch protection") and surface results as team-level rollups, historical trends, and operational review dashboards. The read_api scope is sufficient for all fact collection.

Connecting Self-Managed GitLab (On-Prem)

Organizations running self-managed GitLab behind a firewall face a connectivity challenge: how does a SaaS portal reach an internal GitLab instance without exposing it to the internet? Roadie's Broker, based on Snyk's open-source broker, solves this with outbound-only connections.

The architecture consists of two components. The Broker Client is a Node.js application deployed inside the customer's infrastructure (via Docker, Helm chart, or npm CLI). It initiates an outbound WebSocket connection to the Broker Server, a tenant-specific endpoint hosted in Roadie's infrastructure. All traffic flows through this tunnel, Roadie never initiates inbound connections, and no firewall ports need to be opened.

Security is enforced through multiple layers. An accept.json configuration file on the client side provides allowlist-based access control, by default, Roadie has zero access to any internal APIs. Only explicitly permitted URL patterns and HTTP methods are proxied. Authentication tokens for internal GitLab remain in the customer's infrastructure and are never transmitted to Roadie. Additionally, the Broker Server restricts connections to customer-specified IP CIDR ranges, and all access is logged for audit.

A minimal Broker Client deployment for self-managed GitLab:

docker run \
  --env BROKER_TOKEN=gitlab-integration \
  --env BROKER_SERVER_URL=https://<tenant>.broker.roadie.so \
  -v $(pwd)/accept.json:/service/accept.json \
  roadiehq/broker

The corresponding accept.json restricts access to only the GitLab API paths Backstage needs:

{
  "private": [
    {
      "//": "GitLab API access for catalog discovery and plugins",
      "method": "GET",
      "path": "/api/v4/*",
      "origin": "${GITLAB_INTERNAL_URL}",
      "auth": {
        "scheme": "token",
        "token": "${GITLAB_INTERNAL_TOKEN}"
      }
    }
  ],
  "public": [
    { "method": "GET", "path": "/healthcheck" }
  ]
}

Broker configuration is managed through Roadie's admin UI at https://<tenant>.roadie.so/administration/settings/integrations/broker, where admins set CIDR ranges and broker tokens.

How Roadie compares to the alternatives for GitLab shops

GitLab's native capabilities leave a significant gap. GitLab offers CI/CD pipelines, project templates, and compliance frameworks, but no unified service catalog, no scorecard-based governance, no docs-as-code portal, and no cross-tool aggregation surface. GitLab's Service Desk is an ITSM ticketing system for external users, not a developer portal. GitLab's own direction page explicitly identifies Backstage and Port as competitive solutions, tacitly acknowledging the gap it cannot fill until at least 2026.

Self-hosted Backstage provides the same plugin ecosystem but demands a dedicated team for maintenance, upgrades, infrastructure management, and plugin curation. Real-world experience confirms that organizations routinely spend months reaching production and still struggle with adoption at scale. Roadie eliminates this burden: automatic monthly upgrades, pre-configured plugins (75+), no-code UI customization, and enterprise features like RBAC, usage analytics, and OpenSearch-powered catalog search that don't exist in open-source Backstage.

Proprietary portals (Cortex at ~$65/user/month, Port with a free tier up to 15 users then $40/seat/month) offer polished UIs and built-in engineering intelligence, but carry significant lock-in risk with proprietary data models. Cortex provides strong DORA metrics and maturity scorecards but has no community plugin ecosystem and no TechDocs equivalent. Port offers maximum flexibility through custom "Blueprints" but lacks native documentation features and treats GitLab primarily as an action backend rather than a deep data source. Neither matches the depth of Backstage's GitLab plugin ecosystem, which includes discovery, org sync, CI/CD visualization, scaffolding, pipeline triggers, code owners display, MR statistics, and coverage reporting across six actively maintained packages.

Roadie sits at the intersection: the open-source extensibility and GitLab integration depth of Backstage, with the operational simplicity of a managed SaaS, at roughly one-third the per-user cost of Cortex.

Common pitfalls and production-hardening advice

Several failure modes recur in GitLab + Backstage deployments at scale. Token expiration is the most common, GitLab tokens expire silently (default max 365 days, extendable to 400 in GitLab 17.6+), causing catalog updates to stop without clear errors. Automate rotation via the GitLab API's POST /groups/:id/access_tokens/:token_id/rotate endpoint. Entity naming collisions break ingestion when two repos use identical metadata.name values; standardize naming conventions early and enforce them through scaffolder templates. Quoted numeric IDs trip up YAML parsing, gitlab.com/project-id: '4521' must be quoted, or YAML interprets it as a number and the annotation match fails.

For discovery at scale (500+ repos), scope discovery to specific groups rather than scanning the entire instance to reduce API calls, and deploy webhook-driven updates to eliminate polling overhead. Use Software Templates to generate correct catalog-info.yaml files for new projects from day one, preventing the catalog drift that occurs when teams must retroactively add metadata to existing repos. Treat catalog-info.yaml as code, reviewed via merge requests, enforced by CI validation, and owned by the service team.

Conclusion

The combination of Roadie and GitLab addresses a gap that GitLab itself won't fill until 2026 at the earliest. Auto-discovery with regex-based group and project filtering eliminates the manual registration burden across hundreds of repos. Scaffolder actions specific to GitLab automate project creation with built-in guardrails, CI configuration, branch protection, and catalog registration happen in a single self-service workflow. Tech Insights scorecards transform governance from periodic audits into continuous, visible compliance tracking. And the Broker architecture extends all of this to self-managed GitLab instances without any firewall changes, using outbound-only connections with allowlist-based access control.

The key technical insight is that Backstage's GitLab integration ecosystem is uniquely deep, six actively maintained packages covering discovery, org sync, CI/CD visualization, pipeline triggers, and scaffolding. Roadie wraps this ecosystem in a managed layer that eliminates the 3,12 engineer operational burden of self-hosting, adds enterprise features absent from open-source Backstage (RBAC, scorecards, no-code layout editing, AI capabilities), and avoids the proprietary lock-in of alternatives like Cortex and Port. For GitLab-centric organizations, it represents the fastest path from repository sprawl to a governed, self-service developer platform.

Next Steps

Now that you understand how Roadie integrates with GitLab, here are practical next steps to get started:

• Start a free trial to connect your GitLab instance and see auto-discovery in action within minutes
• Explore the GitLab integration documentation for detailed configuration instructions and advanced patterns
• Learn how to model your software catalog to represent your system architecture properly with components, APIs, and dependencies
• Create your first Software Template to standardize how teams create new GitLab projects with pre-configured CI/CD and governance
• Set up Tech Insights checks to monitor engineering standards like CODEOWNERS files and branch protection across your GitLab repositories
• Review customer success stories from other organizations using Roadie with GitLab to accelerate their platform engineering initiatives

The stakes are substantial: the friction caused by tool sprawl and context switching acts as a massive tax on engineering velocity. For organizations with 50+ engineers already committed to microservices, the question isn't whether to implement a service catalog, it's how to do it effectively before complexity overwhelms capacity.

The hidden cost of microservices at scale

Modern engineering organizations force developers to juggle a dizzying array of monitoring, CI/CD, and cloud infrastructure tools. This fragmentation forces constant context switching, breaking flow state and burning valuable engineering hours every week. When Expedia Group surveyed their 5,000+ developers managing 20,000 microservices, documentation discoverability emerged as their primary pain point - engineers were spending more time finding information than building features.

The ownership problem compounds over time. Without a system of record, teams accumulate what practitioners call "microservice graveyards", entire clusters of services where the original owners have departed and no one wants responsibility. At Spotify before Backstage, engineers described their workflow as "rumor-driven development", the only way to discover how something worked was asking colleagues who might remember.

Incident response suffers most acutely. FireHydrant's analysis of 50,000+ incidents found that when services have clear ownership attached, mean time to resolution drops by 36%. Motability was able to reduce the creation of new services from 2 - 3 days to minutes after implementing service catalog tooling that eliminated the "who owns this?" question during outages. The pattern is consistent: visibility into ownership and dependencies transforms incident response from frantic Slack archaeology into systematic problem-solving.

Backstage as your microservices operating system

Backstage functions as an internal developer portal, a unified interface that aggregates service metadata, documentation, and operational tooling into a single searchable surface. Created by Spotify in 2016 and open-sourced in 2020, it now manages their 2,000+ backend services and 4,000+ data pipelines with contributions from over 60 internal teams. The CNCF accepted it as an Incubating project in March 2022, signaling enterprise-grade maturity. Created by Spotify in 2016 and open-sourced in 2020, it now manages their 2,000+ backend services and 4,000+ data pipelines with contributions from over 60 internal teams. The CNCF accepted it as an Incubating project in March 2022, signaling enterprise-grade maturity.

The Software Catalog forms the foundation. Every service, API, library, and infrastructure resource gets registered with a catalog-info.yaml file that lives alongside the code:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-service
  description: Handles payment processing for all checkout flows
  annotations:
    # Links this service to its PagerDuty on-call schedule
    pagerduty.com/service-id: P123ABC
    # Connects to the GitHub repository
    github.com/project-slug: acme/payments-service
spec:
  type: service
  lifecycle: production
  # Defines team ownership - answers "who owns this?"
  owner: payments-team
  # Groups service into larger business domain
  system: checkout
  # Declares what APIs this service provides
  providesApis:
    - payments-api
  # Declares dependencies on other resources
  dependsOn:
    - resource:default/payments-db

This declarative approach ensures metadata lives with code and flows through standard git workflows. The owner field answers the 3 AM question definitively. The dependsOn and providesApis fields create a navigable dependency graph. Annotations connect the service to operational tooling, PagerDuty, CI/CD pipelines, monitoring dashboards, creating what Backstage calls a "single pane of glass."

The System Model introduces organizational hierarchy: Domains (business areas like Payments or Search) contain Systems (collections of components that form a product capability), which contain Components (individual services) and APIs (interface boundaries). This taxonomy maps directly to how engineering organizations structure teams and ownership, making catalog navigation intuitive rather than arbitrary.

Golden Paths eliminate the copy-paste tax

Before Backstage, creating a new service at Spotify took 14 days of configuration, pipeline setup, and documentation. Afterward: less than 5 minutes. The difference is the Scaffolder, Backstage's templating system that implements what Spotify calls "Golden Paths".

A Golden Path is an opinionated, supported path to building something, a backend service, a data pipeline, a React application. Rather than starting from a copy-pasted template that's already drifted from current standards, engineers use Software Templates that generate services with current CI/CD configuration, security scanning, logging, and observability pre-wired:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: spring-boot-service
  title: Spring Boot Microservice
  description: Create a production-ready Spring Boot service with CI/CD
spec:
  owner: platform-team
  type: service
  # Parameters define the form users fill out
  parameters:
    - title: Service Details
      required:
        - name
        - owner
      properties:
        name:
          type: string
          title: Service Name
          description: Lowercase with hyphens (e.g., user-auth-service)
        owner:
          type: string
          title: Owner
          description: Team that will own this service
          ui:field: OwnerPicker
  # Steps define what actions to execute
  steps:
    # Step 1: Fetch the template skeleton from a repository
    - id: template
      name: Fetch Template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          owner: ${{ parameters.owner }}

    # Step 2: Publish to GitHub and create repository
    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        repoUrl: github.com?repo=${{ parameters.name }}&owner=acme
        description: ${{ parameters.description }}

    # Step 3: Register in Backstage catalog automatically
    - id: register
      name: Register Component
      action: catalog:register
      input:
        catalogInfoPath: /catalog-info.yaml

The philosophy matters: Golden Paths are recommended, not mandated. Engineers can deviate, but they lose platform team support. This balances standardization with autonomy, the platform provides the easy path, but doesn't constrain innovation. Spotify maintains six Golden Paths spanning backend, frontend, data engineering, machine learning, and web development, each optimized for their specific discipline.

The productivity impact extends beyond service creation. Spotify measured new engineer time to 10th pull request, dropping from 60+ days to 20 days after Backstage deployment. When every service follows consistent patterns, understanding one means understanding all.

Dependency visualization reveals blast radius

The Catalog Graph plugin transforms the static catalog into an interactive dependency map. When planning an API deprecation or infrastructure migration, engineers can trace exactly which services consume an endpoint and who owns them. During incidents, the graph shows upstream dependencies that might be causing failures and downstream services that might be affected.

Example dependency graph showing blast radius when payments-service changes

Relationships are defined explicitly in catalog metadata using dependsOn, providesApis, and consumesApis fields. Backstage automatically generates inverse relationships, if Service A declares it consumes API B, API B's page shows Service A as a consumer. This bidirectional visibility makes deprecation planning systematic: filter by API, identify all consumers, contact those teams, and track migration progress.

The blast radius analysis capability transforms change management. Before deploying infrastructure changes, engineers visualize what breaks if a database becomes unavailable or an API endpoint goes down. Migration wave planning becomes data-driven, identify server clusters where dependency chains can be broken cleanly, then sequence the migration accordingly.

Tech Insights gamifies production readiness

Catalog completeness means nothing if the metadata is wrong. Tech Insights (called Scorecards in some implementations) provides automated fact-checking that validates services against production readiness standards. The system operates on two concepts: Facts (data points collected from various sources) and Checks (rules that evaluate facts).

Common checks include PagerDuty integration verification (ensuring on-call is configured), deprecated library detection, Node.js version compliance, and documentation completeness. Each check runs on a configurable schedule and produces a compliance score visible on service pages:

# Example Tech Insights check configuration
techInsights:
  factChecker:
    checks:
      productionReadiness:
        type: json-rules-engine
        name: Production Readiness
        description: Ensures services meet production standards
        # Define which facts to collect
        factIds:
          - entityOwnershipFactRetriever
          - techdocsFactRetriever
          - pagerdutyFactRetriever
        # Define the rule logic
        rule:
          conditions:
            all:
              # Check 1: Must have a group owner (not individual)
              - fact: hasGroupOwner
                operator: equal
                value: true
              # Check 2: Must have TechDocs documentation
              - fact: hasTechDocs
                operator: equal
                value: true
              # Check 3: Must have PagerDuty integration
              - fact: hasPagerDuty
                operator: equal
                value: true

Dexcom used automated checks to drive catalog completeness from 60% to over 95%. Baillie Gifford, operating in the regulated financial services sector, uses scorecards to track security tool adoption across 250 developers, generating compliance reports that previously required days of manual assembly.

The gamification effect drives adoption organically. When teams see leaderboards showing their scorecard performance relative to peers, competitive dynamics motivate improvement without mandates. Engineering leaders report this "soft governance" approach achieves better compliance than top-down enforcement while preserving team autonomy.

Integration creates the unified dashboard

Backstage's plugin architecture enables what engineers call the "single pane of glass", consolidated visibility across the entire operational stack. Over 200 plugins provide native integrations with common tooling.

The Kubernetes plugin displays deployment status, pod health, and resource metrics directly on service pages. Engineers see crash logs aggregated from all pods, health indicators, and links to deeper investigation tools, without leaving Backstage or requiring kubectl access. The PagerDuty plugin shows active incidents, on-call schedules, and allows triggering new incidents from service context. GitHub Actions, CircleCI, and Jenkins plugins display build status, deployment history, and failure details.

API management uses the same catalog model. OpenAPI, AsyncAPI, and GraphQL specifications register as API entities with full interactive documentation, consumer/provider relationships, and lifecycle management. When API version 2 launches, teams identify v1 consumers directly from the catalog and coordinate deprecation timelines.

The integration pattern is consistent: annotate catalog entities with tool-specific identifiers, and plugins fetch relevant data on page load. A properly configured service page shows ownership, dependencies, documentation, build status, deployment health, active incidents, and on-call, everything needed to understand and operate the service from a single URL.

Choosing between self-hosted and managed options

Self-hosted Backstage offers unlimited customization but demands significant investment: typically 2-3 dedicated FTEs with TypeScript/React expertise for initial buildout and ongoing maintenance. Organizations like Paddle ran self-hosted Backstage for four years before migrating to managed alternatives when the maintenance burden conflicted with driving adoption.

Roadie provides managed Backstage at approximately $20/user/month with same-day setup, 200+ pre-configured plugins, and enterprise features like RBAC included. The tradeoff is reduced customization compared to self-hosted, though standard catalog formats mean organizations can migrate later if needs evolve.

Proprietary alternatives like Cortex ($65-69/user/month), OpsLevel, and Port offer differentiated approaches. Cortex emphasizes AI-powered service discovery and executive reporting. OpsLevel prioritizes fast deployment, 30-45 days typical, with automated catalog maintenance. Port offers maximum customization through a no-code builder but requires significant configuration investment.

For organizations with 50-100 engineers, managed solutions typically deliver faster time-to-value. Above 500 engineers with dedicated platform teams, self-hosted Backstage becomes economically viable if TypeScript expertise exists. Regulated industries should evaluate on-premises options alongside Roadie's self-hosted offering.

Starting your service catalog journey

Successful implementations follow a consistent pattern: start with the software catalog before adding complexity. Import users and teams first so ownership fields work immediately. Choose early-adopter teams willing to contribute catalog metadata, then expand systematically. Platform teams at Expedia put 850+ engineers through Backstage-based bootcamp in their first year, treating adoption as a change management initiative rather than a technology deployment.

Catalog completeness matters more than feature breadth initially. Contentful achieved 90% metadata coverage within one year by making Scaffolder the default service creation path, new services entered the catalog automatically, while existing services received incremental metadata through team contributions.

Measure what matters: time to 10th PR for onboarding velocity, MTTR for incident response improvement, and catalog completeness for adoption tracking. Spotify's Pia Nilsson captured the business case succinctly: "If you have numbers like that in your organization, it's easy to get buy-in for investments in developer experience."

The microservices complexity that created the 3 AM ownership problem also created the opportunity for systematic improvement. Backstage provides the framework; your implementation provides the value. Organizations that treat their service catalog as a product, with dedicated ownership, user feedback loops, and continuous improvement, consistently report the productivity gains that justify investment. Those that deploy and forget find another unused tool in an already crowded landscape.

The choice isn't whether complexity will be managed, it's whether you'll manage it systematically before it manages you.

Next Steps

Ready to implement Backstage in your organization? Here are resources to help you get started:

• Explore Roadie's Catalog - See how Roadie's managed Backstage platform can help you organize your microservices architecture with automated discoverability and ownership tracking.

• Learn About the Scaffolder - Discover how Software Templates and Golden Paths can standardize service creation and reduce onboarding time from weeks to minutes.

• Read Implementation Case Studies - Learn from companies like Expedia Group, Dexcom, and Contentful who have successfully deployed Backstage at scale.

• Compare Deployment Options - Download the whitepaper comparing managed versus self-hosted Backstage to determine the best approach for your organization.

• Book a Demo - See Roadie in action. Request a personalized demo to discover how managed Backstage can tame your microservices sprawl with same-day setup and enterprise-grade security.

Typically, EntityProviders are registered at application startup via the catalogProcessingExtensionPoint. Once the backend initializes, the set of providers is fixed. But what happens when you need to dynamically create new sources of catalog data without redeploying?

Consider these scenarios:

• A multi-tenant platform where each tenant needs isolated entity management
• User-defined integrations that pull data from custom sources
• Dynamic data pipelines that generate catalog entities on-demand
• Self-service onboarding where teams register their own data sources

Backstage doesn't natively support registering EntityProviders after startup. This post explains how to solve this with a provider pooling pattern.

The Challenge

The EntityProviderConnection that allows emitting entities is established once at startup when providers are registered. After initialization, you cannot add new providers—any attempt to call addEntityProvider after the catalog has started will fail.

The Solution: Provider Pooling

Instead of fighting Backstage's architecture, work with it. The key insight: register a pool of providers at startup, then dynamically assign them to consumers at runtime.

┌─────────────────────────────────────────────────────────────┐
│                    Backend Startup                          │
│  ┌──────────────────────────────────────────────────────┐   │
│  │  1. Create pool of N idle EntityProviders            │   │
│  │  2. Register all with catalogProcessingExtensionPoint│   │
│  │  3. Restore any persisted assignments from database  │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  ProviderRegistryService                    │
│  ┌────────────────────────────────────────────────────┐     │
│  │  getProviderFor(id) → assigns idle provider        │     │
│  │  releaseProvider(id) → clears entities, returns    │     │
│  └────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘
                               │
         ┌─────────────────────┼─────────────────────┐
         ▼                     ▼                     ▼
    Consumer A            Consumer B            Consumer C
    (provider-0)          (provider-1)          (provider-2)

The Pooled EntityProvider

Extend the standard EntityProvider with assignment tracking and the ability to clear entities:

interface PooledEntityProvider extends EntityProvider {
  assignTo(id: string): void;
  clearAssignment(): void;
  clearEntities(): Promise<void>;
  updateEntities(entities: Entity[]): Promise<void>;
}

The key method is clearEntities—emitting an empty full mutation removes all entities that provider previously managed:

async clearEntities(): Promise<void> {
  await this.connection?.applyMutation({
    type: 'full',
    entities: [],
  });
}

The Registry Service

The registry manages the pool, handling assignment and release:

interface ProviderRegistryService {
  getProviderFor(id: string): Promise<PooledEntityProvider>;
  releaseProvider(id: string): Promise<void>;
  getAllProviders(): PooledEntityProvider[];
}

When a consumer requests a provider, the registry either returns an existing assignment or finds an available provider from the pool and persists the new assignment.

When a provider is released, the registry clears its entities, removes the assignment, and returns it to the pool.

Catalog Registration

Register all pooled providers at startup via a backend module:

const providerRegistryCatalogModule = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'provider-registry',
  register(module) {
    module.registerInit({
      deps: {
        catalog: catalogProcessingExtensionPoint,
        registry: providerRegistryServiceRef,
      },
      async init({ catalog, registry }) {
        for (const provider of registry.getAllProviders()) {
          catalog.addEntityProvider(provider);
        }
      },
    });
  },
});

Persistence

Store assignments in a database table so they survive restarts. This ensures the same consumer gets the same provider ID, maintaining entity ownership continuity.

Usage

// Acquire a provider
const provider = await registry.getProviderFor('my-integration-123');

// Emit entities
await provider.updateEntities(entities);

// When done, release (clears entities and returns provider to pool)
await registry.releaseProvider('my-integration-123');

Trade-offs

• Memory overhead — Pre-allocated providers consume memory, though instances are lightweight
• Fixed capacity — Pool exhaustion causes failures; size appropriately for your workload
• Provider ID stability — Assignments must persist to maintain entity ownership across restarts

Conclusion

By pre-registering a pool of EntityProviders and dynamically assigning them at runtime, you can achieve flexible, dynamic entity management without modifying Backstage's core architecture. This pattern works for multi-tenancy, user-defined integrations, or any scenario requiring runtime control over catalog data sources.

Someone suggests Backstage. You look at the README, see it's open source, and think you'll just spin it up.

Six months later, you've burned through your platform team's roadmap, your catalog is half-populated, and developers are still opening JIRA tickets for everything.

At my previous company, we built an internal developer portal from scratch. The bill ran into the millions. That experience taught me that every engineering organization over 100 people eventually needs this infrastructure. But it also showed me how expensive it is to build and maintain. Even with AI coding agents to help with development, there’s a thousand small decisions to make about business logic.

Now, as CEO of Roadie, I talk to engineering leaders every week who are making this build vs. buy decision. Most underestimate what self-hosting Backstage actually costs. Not just in dollars, but in team capacity and time to value.

Here's what the decision really looks like.

The Problem You're Actually Solving

The developer portal problem starts small. You're managing 50 services across 30 teams. Someone asks which services depend on the auth API. You don't have a good answer.

Another team needs virus scanning. You think it exists somewhere, but you're not sure.

So you make a spreadsheet. Service name, owner, what it does. Problem solved.

This works for about three weeks. Then nobody updates the spreadsheet. It's out of date. People stop trusting it.

This is the discoverability problem. At 10 engineers, you just shout across the office. At 100+ engineers, that breaks down completely. You need a software catalog that actually stays current.

But discoverability is just the first problem. You also hit:

The self-service bottleneck: A mobile developer needs an S3 bucket. They don't know Terraform. They open a JIRA ticket. Your platform team gets to it in two weeks. The developer either waits or bypasses your platform entirely and creates shadow IT.

The governance gap: Your security team needs to verify every service has proper on-call setup. Your compliance team wants to check access controls. You need automated checks against your entire software catalog, not manual audits.

These three problems drive every developer portal evaluation. The question isn't whether you need a solution. The question is whether you build it or buy it.

What Backstage Actually Gives You

Backstage is not a developer portal. It's a framework for building one.

This distinction matters. When Spotify open sourced Backstage in 2020, they released their toolkit for building developer portals, not a turnkey product. You get a collection of TypeScript libraries and React components. You have to assemble them into something useful.

This creates immediate friction for most platform teams:

You need TypeScript expertise. Most platform teams work in Go, Python, and YAML. Web development is a different skillset than infrastructure engineering. You either hire TypeScript developers or retrain your existing team.

You're building, not configuring. Out of the box, Backstage gives you basic catalog functionality. It doesn't give you role-based access control. It doesn't give you production-grade search (it runs on Postgres full-text search). It doesn't give you most enterprise features. You build those.

You're maintaining a web application. Backstage releases a new version every couple of weeks. Each upgrade can break your plugins. Each new integration requires custom TypeScript code. This is ongoing work, not a one-time project.

The Real Cost: Team Capacity

When we surveyed the Backstage community this year, organizations that reported being happy with their self-hosted deployment had at least three dedicated engineers. Some teams were as large as 12 people.

Let me translate that into budget terms. Three mid-level engineers cost around $450,000 per year in salary, benefits, and overhead. That's the minimum for a successful deployment.

But the real cost is what those engineers aren't doing.

Time to production: 6-12 months. You're not launching next sprint. You're building the catalog model, integrating CI/CD tools, setting up authentication, configuring plugins, and training teams. The organizations we talk to consistently report 6-12 months before they had something teams would actually use.

Opportunity cost. Those three engineers aren't improving your CI/CD pipeline. They're not hardening security. They're not building platform capabilities that differentiate your business. They're maintaining a developer portal.

Missing features you have to build. Need RBAC so your security services aren't visible to everyone? You're building that. Want better search? You're integrating Elasticsearch. Want API documentation? You're configuring and maintaining that integration.

The actual costs break down into several categories:

• 3 engineers minimum at $150k loaded cost each = $450,000/year
• 9 months to production at 60% team efficiency = ~$200,000 in delayed value
• Ongoing maintenance and feature development
• TypeScript training or new hires

First year total exceeds $800,000. Every year after that is still $450,000 minimum, assuming no team growth.

And you still haven't built all the features you need.

When Building Makes Sense

Self-hosting Backstage gives you something valuable: control.

If you have unique requirements that no vendor can handle, being able to modify every line of code matters. If you're integrating with complex legacy systems, having full access to the source code can be critical.

You also get the Backstage ecosystem. The community is active. New plugins appear regularly. If you need an integration with a specific tool, someone in the community might have already built it.

Some engineering leaders prefer ownership for critical infrastructure. They don't want vendor dependencies. They want the source code running in their environment.

These are legitimate reasons to self-host. But they need to justify the cost.

Build if:

• You have genuinely unique requirements no vendor can handle
• You already have a team with TypeScript expertise who wants to own this
• You're large enough (500+ engineers) that control benefits outweigh costs
• You have specific security requirements that mandate on-premises deployment
• Your platform team has capacity to spare

Don't build if:

• Your platform team is already stretched thin
• You want to launch in weeks, not months
• You need enterprise features without building them
• You'd rather focus engineering resources on your actual platform

The key question: What do you want your platform team working on? If the answer is "building platform capabilities that make our engineering organization more effective," you probably shouldn't be maintaining a developer portal.

The Managed Alternative

When we built Roadie, the goal was straightforward: give you Backstage without the team overhead.

Here's what that means:

Day one deployment. You connect your GitHub organization. Your services start populating. No six-month buildout. No TypeScript work. You're configuring, not coding.

Enterprise features included. RBAC, production-grade search, authentication integrations. The pieces you'd have to build yourself are already there, built from feedback across hundreds of deployments.

Automatic upgrades. When Backstage releases a new version, we test it, validate it, and roll it out. You don't manage the upgrade cycle. You don't deal with breaking changes.

No TypeScript team required. You use a web UI to configure Backstage. Your platform team stays focused on platform work.

The financial calculation is simple. Managed Backstage costs a fraction of a three-engineer team. But the more important comparison is what your platform team accomplishes.

Would you rather have three engineers maintaining Backstage, or three engineers improving your CI/CD pipeline and building platform capabilities?

The Hybrid Approach

This is Roadie's actual positioning: the hybrid model.

Proprietary developer portals lock you into their data model. If they don't support your integration, you're stuck. If they sunset a feature, you adapt. You have no control.

Self-hosted Backstage gives you control but requires a dedicated team and significant TypeScript expertise.

Managed Backstage sits in the middle:

• The flexibility of Backstage's open source ecosystem
• Day-one usability and enterprise features
• No team overhead, no TypeScript requirement, no year-long buildout

You're not locked into a proprietary platform. If you decide you want to self-host later, you can. The catalog format is standard Backstage. Your plugins are standard Backstage.

But you also don't need to staff a team just to keep the portal running.

The Path Few Consider

Most engineering leaders frame this as "self-host Backstage vs. buy a proprietary portal." But there's a third option: start managed, migrate to self-hosted later if needed.

You can validate that Backstage solves your problems without burning six months and half your platform team's capacity. Get your catalog populated. Get teams using it. Prove the value.

Then, if you decide you need more control, migrate to self-hosted. The data model is identical. The plugins are the same. You're not locked in.

Several of our customers took exactly this path in reverse. They self-hosted Backstage, realized it was consuming their platform team, and migrated to Roadie to free up those engineers for higher-value work.

Making the Decision

Here's the framework I use when talking to engineering leaders:

Start with team capacity. Can your platform team absorb a year-long project plus ongoing maintenance? If not, you're not really choosing to build. You're choosing to delay.

Calculate opportunity cost. What else could three engineers accomplish in a year? New CI/CD capabilities? Better security tooling? Improved developer experience? Is maintaining a developer portal more valuable than those alternatives?

Consider your timeline. Do you need this solved in weeks or months? If you need it fast, you're buying. If you have a year to spare, you might build.

Evaluate your requirements. Do you have genuinely unique needs, or do you just need a software catalog with good search, RBAC, and common integrations? Most organizations overestimate how unique their requirements actually are.

Think about your team's preferences. Does your platform team want to work on TypeScript and React components, or do they want to work on platform capabilities? Forcing engineers to maintain infrastructure they don't want to maintain leads to burnout and turnover.

The honest answer for most organizations under 500 engineers is that building doesn't make financial sense. You can get Backstage with all the ecosystem benefits for a fraction of the cost, with immediate deployment, and without tying up your platform team.

What This Really Comes Down To

The question isn't whether you need a developer portal. If you're managing 50+ services across 30+ teams, you probably do.

The question is whether building and maintaining that portal is the best use of your platform engineering resources.

For most engineering leaders, the answer is no. Your platform team should be improving your platform, not maintaining a web application.

That's why Roadie exists. You get Backstage without the team overhead. You get the open source ecosystem without the TypeScript requirement. You get day-one deployment without the year-long buildout.

And your platform team gets to focus on the work that actually differentiates your business.

The build vs. buy decision isn't really about cost. It's about what you want your team working on. Choose accordingly.

Next Steps

If you're evaluating Backstage for your organization, here are concrete next steps to help you move forward:

Explore Backstage capabilities: Review our comprehensive guide to Backstage to understand what's possible with the platform and how organizations are using it today.

See how others decided: Read case studies from engineering teams who've made the build vs. buy decision. Learn from Celonis's experience migrating from self-hosted to managed and how Contentful maintained velocity through hypergrowth.

Understand the full cost picture: Dive deeper into the complete cost breakdown of self-hosting Backstage to validate your budget estimates.

Try it yourself: Request a demo to see how managed Backstage works, or start a free trial to experience Roadie firsthand and get your catalog populated in hours, not months.

Compare your options: Review Backstage alternatives and approaches to ensure you're making an informed decision about your developer portal strategy.

We’ve spent the last five years running Backstage in production across dozens of organizations, and building the missing pieces that make it reliable at scale: performance work, background job isolation, governance and scorecards, RBAC, search, TechDocs operations, plugin maintenance, and the unglamorous reality of constant upgrades. If you want to know what’s the real cost and effort involved in standing up a Backstage instance, we’re uniquely positioned to tell you.

This post is a roadmap of that work. If you only have a minute, the table below is the executive summary detailing what will probably need to be done. Think of it as your ultimate to-do list for self-hosting Backstage.

And as always, many of these learnings and insights are captured in this year’s State of Backstage Report, where you can hear firsthand from self-hosted users around their Backstage journey.

Executive summary

Some assumptions about these numbers:

• You’re aiming for a production-grade portal, not a demo.
• You expect meaningful adoption across the organization, not a niche tool used by one team.
• You’ll run a non-trivial plugin surface (CI, SCM, cloud, observability, security, incident tooling).
• Effort is shown as engineering weeks, where one engineering week equals 5 working days of one engineer, and roughly 40 hours.
• Obviously ranges here vary by organizational complexity, catalog size, and how strict your governance/security requirements are.

| Initiative | What you’re building | Effort (engineering weeks) | | --- | --- | --- | | Performance and scalability | Server-side catalog pagination, worker isolation, stability profiling, general optimization, scaffolder scaling | 16-24 weeks (plus ongoing maintenance). This depends substantially on the size of your catalog - this will be on the higher side for larger catalogs with 50,000+ entities | | Catalog customization and data modeling | Configurable catalog UI, decorators/fragments to enrich without PRs, custom kinds/schema, refresh triggers, completeness tracking | 18-30 weeks depending on just how much customization is necessary to meet your requirements. Plus ongoing time required for maintenance | | Search | Operate a real search backend (OpenSearch), indexers and relevance tuning, UX refinements, AI search tie-in | 8-12 weeks for search, plus another 8-12 weeks for AI search if you want a real assistant experience | | Plugins and integrations | Ongoing plugin lifecycle, auth quirks, API drift, version compatibility | 0.5 to 2 weeks per plugin to productionize (auth, permissions, UI polish, support), plus a few hours per upgrade cycle | | Tech Insights and scorecards | Facts ingestion, rule engine, no-code builder UI, built-in checks library, aggregation/history/reporting | ~100 weeks (roughly 6 months for a team of 4 engineers) to build Tech Insights | | RBAC, security and governance | Role mapping, policy engine, admin UI, token issuance/revocation tied to RBAC | ~50 weeks (6 months for a team of 2 engineers) to build RBAC | | TechDocs operations | Hybrid build modes, webhooks/rebuilds, curated MkDocs environment, performance tuning | 2 weeks for initial setup, ongoing maintenance to address any issues | | Developer experience polish | Catalog UI QoL, homepage improvements, admin UX | Ongoing commitment that can very easily be a full-time job for a platform engineer | | AI and MCP | AI assistant over catalog/docs, embeddings/vector store, permission-aware retrieval, MCP servers | 12- 24 weeks, with a significant ongoing investment to continually implement and improve | | Upgrades and release engineering | Test suites across plugin surface, staged rollouts, triage and rollback processes | Ongoing investment of a few hours a week, with significant effort of 5 - 20 engineering days around major Backstage releases |

The rest of this post breaks down each initiative, why it exists, what tends to go wrong in real usage, and the type of engineering work required to make it solid. If you’re a platform engineer, treat it like a checklist. If you’re an engineering leader, treat it like a way to pressure-test the real cost and opportunity cost of self-hosting.

1. Performance and scalability: keeping Backstage fast at 200k entities

Backstage starts out fast enough. It becomes difficult to keep it performant and responsive at scale.

Once you have tens or hundreds of thousands of entities, TechDocs for most services, scorecards, search indexing, CI integrations and so on, the default architecture begins to strain. With several customers with north of 200,000 entities in their catalogs, this is where we had to invest heavily.

Some of the larger pieces you should plan for:

Server side catalog pagination

Vanilla Backstage loads the full result set for catalog pages, then filters on the frontend. That is fine for a few hundred entities. At hundreds of thousands, it become very slow.

We rewired the catalog list to use server side pagination and filtering. That meant changing how queries are constructed, how counts are calculated, and how the UI behaves when it only has a slice of the data. The result is dramatically lower load times for big catalogs. Reproducing that means touching both backend and frontend, and being ready to debug subtle performance and UX regressions when filters combine in unexpected ways.

Factor on at least 200 engineering hours to implement server side catalog pagination. This accounts for a backend refactor, restructuring the frontend table, compatibility handling for pagination edge cases, and API shape changes.

Catalog pagination becomes an absolute necessity as the size of the catalog increases

Background job isolation

In open source Backstage, long running tasks often share a process with the user facing API. TechDocs builds, scaffolder jobs, scorecard computations, data ingestions and so on can all compete with login and catalog requests.

We pulled the heavy backend work out into separate worker containers. That improves stability and reduces resource contention, but it also means you now have to design and operate a small distributed system: one set of pods serving interactive traffic, another running asynchronous jobs, plus the plumbing to schedule, observe, and scale those workers safely.

Effort wise, a reasonable baseline is one to two days for the Backstage specific configuration and app construction, plus at least a few more days for the infrastructure work. All told, expect a small team to spend a week or two on this.

Deep stability work

On the way to running multi tenant Backstage, we have spent a lot of time on the unglamorous work: memory leaks, readiness probes that misreport their status, and so on. Examples include fixing global arrays that never stopped growing, ensuring in memory caches actually drop expired items, and tuning Kubernetes probes so that pods are only marked ready when they really are.

Tenant performance is meticulously logged and continually optimized

These are the bugs that only show up after weeks of production traffic. If you self host, you need to plan for that kind of ongoing investigation with profiling, heap dumps and careful rollouts.

Factor in at least a couple of engineering weeks time off the bat to ensuring all your infrastructure is properly profiled and optimized, and several hours per month of monitoring to ensure everything is running smoothly.

Scaffolder scalability: popularity changes the problem

The Backstage Scaffolder is easy to run when usage is low, but once it becomes the default way engineers create services and automate workflows, it turns into a workflow you need to manage. Scaling isn’t just “add more pods” - it’s making sure you can absorb bursts without the rest of Backstage slowing down, separating template execution from other traffic, and having enough visibility to support it day to day with an understanding what’s running, what’s queued, what’s stuck, what’s failing, and why.

The Scaffolder as a production workflow engine

Preloading, caching and endless optimizations, frontend and back

On the frontend we made numerous changes to how Backstage handles content. This includes optimizing API calls, ruthlessly culling what is loaded, and optimizing how we serve static content such as TechDocs.

In the backend we added visibility and controls around the job scheduler, because once your tooling grows and you’ve got multiple ingestion refreshes, scorecard jobs and TechDocs builds, it is easy to end up with invisible backlogs that impact performance.

Roadie’s scheduler view showing background jobs across the instance

None of these changes is individually complex, but together they form the difference between “Backstage mostly works” and “Backstage feels fast even on a Monday morning with thousands of users.”

It’s challenging to reliably estimate this effort; but even at the lower end for a smaller catalog expect to sink multiple months worth of engineering effort into optimization for Backstage. For us, obviously with multiple Enterprise customers that makes sense to invest the time - for an individual team it’s a challenging cost-benefit discussion to make work.

2. Catalog customization and data modeling

Backstage’s catalog is designed to be endless flexible and extensible, but in practice organizations quickly run into the limits of pure YAML in git.

Over time we have had to turn the catalog into something that behaves more like a product, both in terms of how it’s populated, and how it’s used and consumed by users and services.

Custom columns, tabs and views

Platform teams want to surface critical metadata directly in the catalog table: criticality, lifecycle, tier, compliance score, owner health, and so on. We built configurable catalog columns that can display arbitrary metadata, numeric scales, links and even scorecard results, along with the ability for users to save filtered views as tabs.

Replicating this means building a more dynamic catalog UI and a way to define, persist and share those views. The alternative is endless “export to CSV and slice in a spreadsheet” workflows, which ultimately defeat the purpose of an IDP.

This was one of the most useful features we’ve built for Roadie, and there’s hundreds of hours of engineering effort behind the work.

Custom tabs in action - a must have for the more mature catalog

Decorators and “glue of truth”

Real organizations almost never have a single source of truth. Teams want to combine data from git, SSO, HR systems, cloud providers and ad hoc spreadsheets.

We built the Decorator as an entity decorator, and the Fragments API, which allow extra metadata to be stored in Roadie’s database and merged into entities at runtime, without changing the YAML. That is the foundation for things like business ownership, cost centre tags, custom maturity ratings, and so on. The Decorator works in the UI, while the Fragments API achieves the same thing programmatically.

Skip the YAML - decorate entities with metadata from within the Backstage UI

If you self host, you will need your own answer for how to enrich entities from multiple systems without forcing every change through a pull request. What we’ve learned from experience is that trying to batch annotate entities via organization-wide PRs is unlikely to succeed, hence, empowering the platform teams to decorate these entities at the IDP level. Factor on at least two months of engineering investment to make the data model changes, build the logic and UI elements.

Repositories, products and custom kinds

Very quickly people want to catalogue more than just “services.” They want to represent repositories, shared libraries, products, data models, infrastructure resources and more.

We have added new kinds such as Repository and Product, plus guidance and tooling for extending the entity model safely. Doing this yourself means working with Backstage’s schema system, updating layouts and cards, and then thinking about how all of this will be queried and shown in search.

These are some of the more foundational and impactful changes, and are in the Backstage scheme of things, on the simpler side. Factor on 40-60 hours for the changes to the data model schema and building the custom processor.

Richer kinds unlock richer relationships: the catalog graph is one way to visualize how services, repos, and products connect

Instant updates and completeness tracking

The catalog is a living thing, and it’s important it stays fresh. As usage grows, you will want more control over how your catalog is refreshed, in an idempotent way, from your underlying systems. We use webhooks and APIs from SCM systems to trigger catalog refreshes in near real time, and we track catalog completeness using scorecards that measure ownership, labels and other metadata.

A self hosted team will need some equivalent if you want to trust the catalog as a real time view of your software. Factor on a month or more to get this done.

3. Search: how people can actually find things

Search is one of the most visible parts of a developer portal. If it feels off, people give up quickly. Open source Backstage has gradually improved search, but we found that large organizations quickly needed more than the out-of-the-box search experience.

A real search engine

We moved to OpenSearch for search, with analyzers that handle mixed case names, hyphenation, and partial matches. For example, engineers can type part of a service name, or a fragment of an API route, and still find what they need. This is not the case for out-of-the-box self-hosted Backstage.

That work required running and maintaining a search cluster, setting up indexers, and designing relevance rules. It also meant a pass over search UX, so that results are presented in a way people can actually work with. This can take anywhere from 2-3 months depending on the level of customization required.

AI search

A recently addition, but a critical one - we now make available our MCP tooling to a built-in AI assistant in the Roadie UI. This allows engineers to ask natural language questions about their catalog and have the answers displayed in Roadie. A step change from regular search, and a similar order of magnitude in terms of engineering effort. The real investment here was in the addition of MCP tooling, but factor on a couple of weeks at least here.

AI search uses the same MCP tooling to expose a richer, conversational search experience

4. Plugins and integrations: owning the long tail

Backstage’s plugin model is its greatest strength and one of its sharper edges.

Most large organisations end up using a long list of plugins: Argo CD, AWS, Azure DevOps, GitLab, Jira, Datadog, Sentry, SonarQube, security scanners, incident tools and more. Each of those brings its own authentication quirks, rate limits, API changes and version compatibility issues.

Over the last five years we have:

• Maintained and updated dozens of plugins when Backstage core moved forward or vendor APIs changed
• Built new plugins such as AWS resource ingestion, Wiz security integration, LaunchDarkly enhancements, Shortcut integration and more
• Created a secure connectivity pattern with Snyk using an open source broker to reach on prem systems without opening inbound access

If you choose to self host, the integrations you rely on today will keep evolving. The work is less about “install plugin X once” and more about “own a small product surface for each plugin indefinitely.”

That is not a reason to avoid self hosting, but it is a cost you should be explicit about.

5. Tech Insights and scorecards: turning Backstage into a governance tool

Most organizations adopting Backstage eventually want more than a catalog. They want to use it to drive standards: security adoption, SLO coverage, migration progress, documentation quality, and so on.

Open source Tech Insights gives you some primitives, but it expects you to write a lot of the logic and UI yourself. We turned that into a full product, and it was a six-month engineering lift from a fairly substantially sized engineering team. Factor on at least the same for your own efforts to make Tech Insights into a fully-featured product. Here’s what we’ve built:

A no code scorecard builder

Platform teams can define checks and scorecards in a UI. Under the hood, data is pulled from SCMs, CI, security tools and other sources, stored as facts, and evaluated regularly. We ship a large library of built in checks so that people do not start from a blank page.

Aggregation, history and reporting

Results are rolled up by team and group, graphed over time, and shown either on a dedicated Scorecards section, and/or directly on entity pages and in the catalog. That lets you answer questions like “which team is lagging on SAST adoption” or “how has our documentation coverage changed over the last quarter.”

To replicate this yourself you will need to implement three things: a data ingestion layer, a rule engine, and a UI that surfaces all of it in Backstage. It is very doable, but it is also a multi month engineering effort.

Tech Insights scorecards in action, turning standards into automated checks across your entire software catalog

6. RBAC, security and governance: controlling who can do what

Once you have real adoption, permissions move from “nice to have” to “non negotiable.”

Backstage’s permission framework is flexible, but it does not ship with a full RBAC system out of the box. Building a full RBAC product (roles, admin UI, policy engine, tokens) is typically a six month effort for a small two-person team. We had to build:

Fine grained roles and permission policies

We support custom roles, mapping from identity provider groups to roles, and a policy engine that can express rules like “owners of a service can edit it, others can only view” or “only this group can run these templates.” That is exposed through an admin UI rather than code.

API tokens and service accounts

To allow automation and external tools such as MCP clients, we added API token support tied into the same permission system. That means designing token issuance and revocation flows, and making sure tokens are not a back door around your RBAC rules.

7. TechDocs: documentation without the pain

Docs like code is one of Backstage’s most attractive features, but it can be surprisingly hard to operate in anger.

Problems tend to show up over time: builds that are slow or flaky, large monorepos that generate huge docs sites, Markdown features that people expect but are not configured, and so on.

We addressed this with:

Hybrid build modes

Teams can choose between on demand builds or CI based publishing to a shared bucket. For large or frequently accessed docs, CI publishing gives much better performance.

Autodiscovery and webhooks

When a docs folder changes in git, webhooks trigger a rebuild in the background so that by the time someone opens the page, the new version is already there.

A curated MkDocs environment

We ship a standard set of MkDocs plugins and extensions for diagrams, tabs, admonitions and monorepos. That saves teams from having to build their own MkDocs image and solve the compatibility problems that come with it.

None of this is conceptually complex, but if you skip it you can easily end up with TechDocs that are slow, unreliable, or underused.

TechDocs pages rendered in Backstage, powered by MkDocs and a build pipeline running behind the scenes

8. Developer experience improvements

Developers judge a portal by how it feels in day to day use.

A lot of our work has been on the small things: catalog table layout, sticky filters, configurable columns, a useful homepage, sensible defaults for layouts, certified templates, better error messages, and an admin area that is navigable when you have dozens of integrations.

On top of that we are experimenting with new ways to let teams extend the UI quickly, such as MDX based homepage cards that can fetch and present data without building a full plugin. The direction here is to give engineers “power tools” so they can shape the portal around their workflows.

If you self host, this is the kind of work that rarely makes it onto a roadmap, but that strongly influences whether people actually enjoy using Backstage.

Homepage cards - a quick and easy way to expose information from internal and external APIs in Roadie

9. AI and MCP: the next layer

Finally, there is AI.

We touched on this under search - having an AI assistant that can answer questions about your documentation and catalog, and we have implemented Model Context Protocol servers so that external AI tools can safely talk to Roadie’s APIs. That allows agents in editors to discover templates, understand APIs and query scorecards, all through Backstage as a system of record.

You do not need AI on day one of a self hosted journey. It is, however, part of where the ecosystem is going. If you want Backstage to be a first class participant in your AI tooling, you will likely need to invest in similar capabilities: embeddings, vector storage, permission aware retrieval, and APIs tailored for LLM clients.

10. Saving the best for last - upgrades

If there is one topic that consistently surprises teams adopting Backstage, it is the upgrade burden.

Standing up Backstage for the first time is the easy part. Keeping it running, stable, secure, and compatible with the fast-moving upstream project is where the long-term work lives. Backstage releases frequently. Plugins evolve independently. Breaking changes land often and sometimes without much warning. And because Backstage is a framework rather than a product, the blast radius of every change is potentially wide.

Backstage moves fast

Backstage core typically publishes new releases every few weeks. These releases regularly include:

• Deprecations or removals of catalog processors or entity fields
• Changes to authentication flows and permission boundaries
• Scaffolder API changes
• Search backend rewrites
• Plugin architecture changes (backend plugin framework migration, for example)

If you fall behind, the upgrade path compounds. A one-version bump is manageable. A six-month gap can become a multi-week project.

Plugins evolve at their own cadence

Each plugin (Datadog, Argo, GitHub, Jira, TechDocs, API Docs, AWS, Azure, Wiz, LaunchDarkly, and dozens more) sits on top of Backstage but is not coordinated with it. They evolve independently, and breaking changes in one can cascade across your installation.

A self-hosted team effectively ends up with a plugin garden, each with its own update cycle, bug behavior, API quirks, and upstream issues.

The systems we had to build to keep this sustainable

After years of running Backstage in production, we realized that upgrades needed as much engineering investment as performance or catalog work. We ended up building test suites that run Roadie’s full plugin surface against new Backstage versions before anything reaches customers. This catches breakages early and reliably.

We also never upgrade every tenant or environment at once. Rollouts happen gradually, with internal test tenants and rollback paths if something unexpected happens.

When something does break (and it often does) we have documented processes for diagnosing plugin failures, dependency mismatches, deprecations, and migration regressions, and clear steps for rolling forward or back safely. There’s also the experience that comes from having done this for years across multiple versions - hard won experience that allows our team to quickly triage and resolve highly-specific Backstage issues.

These systems collectively represent hundreds of hours of engineering time. They exist because without them, upgrades would regularly break real customer portals.

The hidden cost: this work never ends

This is the part people most often underestimate. Upgrades are not a one-time project. They are ongoing rent Every new Backstage release, every new plugin version, every upstream API deprecation requires attention.

Self-hosting teams often start enthusiastically, then gradually slow down as the backlog of breaking changes grows. Eventually they end up frozen on an old version, unable to upgrade without major intervention.

Why this matters for your roadmap

If you plan to self-host Backstage, you need to view upgrades as a primary, recurring body of work, not a background task. The cost lives in:

• Testing every combination of plugins and Backstage core on every upgrade
• Managing breaking changes across dozens of moving parts
• Keeping your team aware of upstream changes and migration guides
• Avoiding drift so upgrades remain tractable
• Making sure your internal plugins and catalog processors don’t fall behind
• Ensuring that upgrades don’t take down developer workflows

This is not a warning; it’s a reality. Many teams can absolutely do this. But you should plan for it with eyes open. If there is one lesson from the State of Backstage report, it is this: upgrades are the number one pain point for self-hosters. Not performance. Not TechDocs. Not plugins. Upgrades.

Bringing it together: a realistic roadmap

Taken together, the list above is long. That is the point.

Self hosting Backstage is not simply about standing up a Node process and pointing it at your git repos. It is about:

• Operating a complex web application with many background jobs and external dependencies
• Owning a growing set of plugins and integrations and keeping them healthy
• Deciding how you will model your organisation and software, and then enforcing and evolving that model (we provide extensive engagement and support here through a structured onboarding and ongoing Customer Success motion)
• Providing governance through scorecards and permissions, not slide decks and spreadsheets
• Smoothing the user experience enough that engineers actually want to use the portal
• Keeping up with upstream Backstage changes and broader trends such as AI
• Upgrades. Friction around keeping your Backstage instance up-to-date with the upstream upgrades is a major, major source of pain for self-hosters.

None of this is impossible. Many of our customers could build a lot of it themselves, or did, before migrating to Roadie. Hundreds or thousands of organizations are successfully hosting their own Backstage implementations.

The question is not “can you,” but “which pieces do you want to own, and how much time do you want to commit.” Every engineering hour spent on an upgrade, or an enhancement already offered by Roadie is an hour not spent on improving developer experience, or custom plugins that unlock real value for your internal users. It’s a question of opportunity cost.

Roadie is far more than a hosted wrapper for Backstage - we offer production-grade infrastructure and enhancements that address fundamental gaps in security, governance, performance and developer experience that organizations that want to self-host would inevitably need to replicate.

If you decide to continue your journey on self hosted Backstage, we hope this roadmap helps you plan that work with eyes open. And if you prefer to have someone else carry this complexity for you, that is the role Roadie continues to play: a production grade distribution of Backstage, shaped by years of learning across many organizations. Feel free to request a demo anytime.

Everything worked fine when we had 10 or 20 services. Then we hit 50. That's when people started asking questions we couldn't answer: What is all this stuff? Who owns which service? If something breaks at 2 AM, who do we call? How do we know everything is secure?

We tried a spreadsheet. It lasted about three months before it became hopelessly out of date. So we built a UI from scratch to list services, show where they were running, and let people create new services without going through us for everything.

That was 10 years ago. Today, this problem has a name, internal developer portal, and there's an entire market built around solving it. But after talking to hundreds of platform teams, I've found they all struggle with the same three fundamental problems.

Problem 1: The Discoverability Crisis

The discoverability problem shows up when your organization crosses a threshold. At 10 engineers, everyone knows what everyone else is working on. At 100+ engineers, it's chaos.

Here's what this looks like: Your security team builds a virus scanning service. Your mobile developers need to scan files for viruses, but they don't know this service exists. In an enterprise, they can't just sign up for some random SaaS tool because of compliance requirements. So they open a Slack thread, ask around, wait for responses, schedule a meeting, and maybe get an answer in a week.

Meanwhile, your platform team is getting bombarded: "Do we have an API for X?" "Who owns service Y?" "Which services depend on this database?"

The discoverability problem has three common variations:

Documentation sprawl: Teams scatter docs across Confluence, GitHub wikis, Google Docs, and Notion. No one can find anything. You need to centralize documentation in a software catalog where people can actually discover it.

Dependency mapping: You need to understand which services depend on each other. When you're planning to upgrade a database, you need to know what breaks. Without a dependency graph, you're operating blind.

API discovery: Different teams build APIs, but there's no central place to see what exists, what they do, or how to use them. You need a searchable catalog of API specs.

This problem scales exponentially. We don't work with companies under 50 engineers because they don't face this yet. Most of our customers have 100+ engineers. At that scale, you can't just shout across the office anymore.

Problem 2: The Self-Service Bottleneck

Your platform team becomes a bottleneck. Developers want to get things done without opening a JIRA ticket and waiting days for someone to configure their network rules or provision an S3 bucket.

At most organizations, you're waiting days for basic requests. This creates two problems:

First, developers get frustrated and stop being productive. They're blocked on simple tasks that should take five minutes.

Second, they bypass your platform entirely. This is shadow IT. A mobile developer who doesn't know Terraform just goes into the AWS console and clicks "Create S3 Bucket" because they can't wait. Now you have untracked infrastructure that's not in your Terraform state, doesn't follow your naming conventions, and might not meet your security requirements.

The self-service problem shows up in two main ways:

Project creation: A developer wants to start a new service on your platform. They need a repo, CI/CD pipeline, monitoring, logging, and all your platform integrations. If they configure this manually, they'll get it wrong. You want to give them a template that sets everything up correctly in five minutes, a "golden path".

Infrastructure requests: A developer needs an S3 bucket, a database, or network rules configured. They don't know Terraform and shouldn't have to learn it. You want to let them fill out a form that opens a templated pull request against your Terraform repository. Someone reviews it, merges it, and the infrastructure gets created. The request is tracked, the developer doesn't need specialized knowledge, and you maintain control.

Self-service doesn't mean "let developers do whatever they want." It means giving them fast, easy ways to do things the right way. You're creating guardrails, not removing them.

Problem 3: The Governance Gap

You have hundreds of services running on your platform. You need to know they're secure, reliable, and following your best practices. But you have no single place to check.

Here's a concrete example: You use Incident.io for incident management. Every service should be registered in Incident.io with an on-call person assigned. How do you verify this? You could manually check each service, but that's impossible at scale. You could send a Slack message asking people to audit their services, but half won't respond.

The governance problem shows up in several ways:

Security compliance: You need to verify that all services are scanning dependencies for vulnerabilities, using approved authentication methods, and following your security policies. Without automated checks, you're relying on self-reporting.

Reliability standards: You need to know which services have proper monitoring, alerting, and on-call rotations. When an outage happens, you need to immediately know who to call.

Best practices enforcement: Your platform team has defined standards, code review requirements, test coverage thresholds, documentation expectations. You need to see which teams are falling behind so you can work with them to improve.

The governance problem is about visibility rolled up across your org chart. You want to ask: "Show me the director of engineering who has the most services failing our security checks." Then you can work with that director to improve things.

Some teams also want DORA metrics (deployment frequency, lead time, change failure rate, mean time to recovery) visible in one place. This is harder than it sounds. One of the DORA metrics is deployment frequency, the more you deploy, the smaller your changes are, the less likely they cause issues. But what counts as a "deployment" when different teams use Argo CD, Netlify, and five other deployment tools? Some normalization has to happen before you can just look at a DORA metric. Teams aren't necessarily there yet, and they're expecting a bit of magic that isn't that simple.

The same applies to defining what counts as a "service." That's a hard question to answer, and it's one of the most important challenges when trying to get any developer portal working in your organization.

Why This is Hard to Solve

These three problems, discoverability, self-service, and governance, all stem from the same root cause: you have a lot of software, and you need organized metadata about it.

At Workday, we spent a lot of money building a custom solution. When I left in 2020, I talked to other companies and found they'd all built similar things.

The problem is that building this from scratch takes a year and requires a dedicated team to maintain. You're essentially building a product inside your company.

In 2020, Spotify open sourced Backstage, which gave the world a framework for building developer portals. But Backstage isn't a ready-to-use portal, it's a set of TypeScript libraries you use to build your own portal. This creates new problems:

Language barrier: Platform teams typically work in Go, Python, or YAML. They don't know TypeScript, which is a web development language.

Build time: Because you're building from libraries, not deploying a container, it takes six months to a year to get Backstage into production.

Team requirements: We surveyed the Backstage community and found that teams who report being happy with self-hosted Backstage have at least three dedicated engineers. Large deployments have 12+ engineers working on Backstage full-time.

Missing features: Backstage doesn't include basic features like role-based access control out of the box. The search runs on PostgreSQL full-text search, which is okay but not as good as Elasticsearch. Your search won't be great unless you manage an Elasticsearch cluster as well as your Backstage instance.

Getting Backstage takes a year and a team of five people. That's why the internal developer portal market exists.

How Companies Choose Solutions

When people come to us, they're typically in one of three situations:

They already have Backstage: Someone stood it up at some point and people are using it. They're realizing it's a lot of effort and they don't want to staff that team of five people.

They want Backstage specifically: They want a developer portal, like the idea of Backstage, and don't want to be locked into a proprietary data model. They want to customize their solution because they have legacy tools they need to integrate. But they don't want to staff the team around it.

They just want a developer portal: They don't care if it's Backstage or not. In this case, it's more competitive between proprietary solutions and Backstage-based options.

For the first two groups, they're doing a build versus buy evaluation: how much will it cost us to build and maintain this versus how much will it cost to buy a managed solution?

For the third group, it's more of a feature comparison and proof-of-concept scoring across vendors.

What You Actually Need

Solving these three problems requires a few key capabilities:

A software catalog: Your source of truth for what software exists, who owns it, and how it's configured. The catalog needs to integrate with your existing tools, your repos, your CI/CD, your cloud providers, so it stays up to date automatically.

Self-service actions: Your developers need a UI for common tasks that generates the right pull requests, kicks off the right workflows, and follows your standards. This keeps them moving fast without bypassing your platform.

Automated scoring: You need automated checks that run against everything in your catalog and tell you what's not meeting your standards. This gives you the visibility to work with teams on improvements.

Easy onboarding: Getting services into the catalog can't require each team to manually register everything. You need automated ingestion that pulls metadata from your existing systems. This is table stakes, but it's an area where Backstage is weak compared to proprietary competitors.

The challenge is that every organization has a slightly different definition of what counts as a "service" or a "deployment" or a "team." You need a solution that's flexible enough to adapt to your organization while being opinionated enough to actually work.

The Path Forward

These three problems, discoverability, self-service, and governance, get worse as your engineering organization grows. If you're at 100 engineers now, imagine what happens at 200 or 500. The chaos compounds.

You're not the first platform team to face these problems. Every company with more than 50 engineers hits them eventually. Software catalogs, self-service automation, and governance scoring aren't experimental anymore.

The decision you need to make is whether to build or buy. Building gives you complete control but requires significant investment. Buying gets you there faster but means accepting someone else's opinions about how things should work.

Whatever path you choose, the problems won't solve themselves. Your platform team is already overwhelmed with questions, your developers are already frustrated with bottlenecks, and your managers can't answer basic questions about what's running in production.

These problems only get worse with time. The earlier you address them, the easier they are to solve.

Next Steps

If you're ready to address these platform engineering challenges, here are some practical next steps to consider:

Evaluate Backstage for your organization: Learn more about what Backstage is and how it works to understand if it's the right foundation for your developer portal needs.

See a developer portal in action: Request a demo of Roadie to see how a managed Backstage solution can solve your discoverability, self-service, and governance problems without the overhead of building and maintaining it yourself.

Calculate your total cost of ownership: Use our guide on how much Backstage really costs to compare the build versus buy decision for your specific situation.

Learn from teams who've solved these problems: Read our case studies to see how companies like Contentful, Celonis, and others tackled similar platform engineering challenges.

Start with a free trial: If you're ready to experiment, try Roadie free to get hands-on experience with a fully managed developer portal built on Backstage.

Gartner forecasts that by 2026, 80% of large software engineering organizations will establish platform teams as internal providers of reusable services, components, and tools for application delivery, up from 45% in 2022. The CNCF Backstage project now boasts over 3,400 adopters worldwide. What started as a bunch of internal teams hacking together their own tools has turned into a mature discipline with established best practices, dedicated conferences, and a rapidly consolidating technology landscape.

For VPs of Engineering and Platform Engineering leads who have moved beyond the "why do we need this" phase, the question now is: how do we mature this practice without getting stuck in endless portal maintenance?

The Evolution of Platform Engineering

To understand where we are going, we must briefly understand how we got here. We have moved from the "Ticket Queue" era (where centralized IT provided stability but strangled velocity) to the "DevOps Revolution," where developers gained speed but drowned in infrastructure complexity.

Platform Engineering represents the synthesis of these two extremes. It centralizes complexity without removing autonomy. Platform teams build and maintain the underlying infrastructure, but they expose it through self-service interfaces. This allows developers to move quickly without needing to master every implementation detail, effectively treating the platform as a product with developers as customers.

From Infrastructure to Interfaces

Early platform engineering efforts focused heavily on infrastructure primitives. Teams invested enormous energy in standardizing Kubernetes deployments, building CI/CD pipelines, and creating Infrastructure as Code templates. These were necessary foundations, but they were not sufficient for driving developer adoption.

The modern Platform Engineering conversation has shifted decisively toward Developer Experience. The infrastructure layer still matters, but what sets teams apart is how they present that infrastructure to developers. That's why Internal Developer Portals (IDPs) have become popular.

The Rise of the IDP

An IDP is the storefront of your platform. It gives developers a unified interface to discover services, access documentation, spin up new projects from templates, and view their deployments. Without this interface layer, even the most sophisticated platform remains opaque and underutilized.

While various tools exist, the market has overwhelmingly converged on a single standard. Recent analysis indicates Backstage holds approximately 89% market share among organizations that have adopted an IDP. Originally developed by Spotify and now a CNCF project, it has moved from early experimentation to essential infrastructure.

This dominance is reflected in the project's momentum. Backstage now boasts over 270 public adopters, including global brands like LinkedIn, CVS Health, and Vodafone. It was also the top CNCF project by end-user commits and the fourth most contributed-to CNCF project in 2024, trailing only infrastructure giants like Kubernetes, OpenTelemetry, and Argo.

Platform as a Product: From Philosophy to Practice

The shift to "Platform as a Product" marks a critical maturity point in Platform Engineering. Instead of mandating tools, modern platform teams treat developers as customers, using a competitive dynamic to force a relentless focus on value delivery.

This approach manifests in two concrete practices:

Golden Paths

Rather than offering infinite flexibility, platform teams curate Golden Paths, opinionated, well-supported pathways for common tasks. These paths come with excellent documentation, proven templates, and integrated tooling. Developers can deviate when necessary, but the "Golden Path" represents the path of least resistance and highest support.

Measuring Success

Success measurement has fundamentally changed. Uptime is no longer the only metric that matters. Leading teams now track impact using:

• Adoption rates: Are developers voluntarily choosing the platform?
• Time-to-hello-world: How fast can a new engineer deploy code?
• DORA metrics: Tracking deployment frequency and lead time for changes.
• Satisfaction scores: Using frameworks like SPACE to measure developer sentiment.

For example, Spotify reported that their time-to-tenth-pull-request metric for new developers dropped by 55% after deploying Backstage.

The Tooling Landscape: Consolidation and Standardization

The Platform Engineering technology stack has matured into defined categories. The infrastructure stack is well-defined: cloud providers (AWS, GCP, Azure) at the bottom, Kubernetes for orchestration, and Terraform for Infrastructure as Code. Increasingly, Backstage or a commercial derivative serves as the interface layer on top. Gartner's Hype Cycle for Platform Engineering now tracks dozens of technologies across this stack, reflecting the maturity of the space.

This consolidation is creating a dilemma for the interface layer: Build vs. Buy. Organizations must choose between self-hosting Backstage, purchasing a commercial offering, or using a managed Backstage solution.

Self-hosting Backstage provides maximum flexibility but comes with significant costs. Industry observers report common pitfalls including:

• Long time-to-value: Teams often spend 6-12 months on setup, with complex implementations extending to 18+ months.
• Maintenance burden: The plugin architecture requires continuous maintenance, and breaking changes in recent releases have created upgrade challenges.
• Low adoption:Organizations outside of Spotify struggle with adoption, with average internal rates hovering around 10%—often because teams burn out on maintenance before delivering features developers actually want.

This is not a failing of Backstage itself, but rather a reflection of the reality that building and maintaining a production-quality developer portal requires dedicated ongoing investment. Many organizations discover that maintaining the portal consumes so much of their platform team's capacity that they never get to building the unique platform capabilities that would actually differentiate their developer experience.

Commercial and managed offerings address this challenge. Several alternatives exist in the IDP market, each with different approaches and strengths:

• Solutions like Roadie provide Backstage as a service, eliminating the operational overhead while preserving the extensibility and ecosystem compatibility that make Backstage attractive.
• Red Hat Developer Hub offers an enterprise-grade alternative for organizations already invested in the Red Hat ecosystem.
• Platforms like Port, Cortex, and OpsLevel provide different architectural approaches to the IDP challenge, while Humanitec focuses on platform orchestration.

These approaches allow platform teams to skip directly from concept to value delivery, focusing their energy on the platform logic and Golden Paths that are unique to their organization rather than on maintaining commodity infrastructure.

The Future: AI and the Intelligent Platform

The integration of AI into IDPs is the next frontier for Platform Engineering. Gartner predicts that by 2028, 90% of enterprise software engineers will use AI code assistants, up from less than 14% in early 2024. The role of developers is shifting from implementation to orchestration, focusing on problem-solving and system design. Early implementations are already showing promising results, and the trajectory suggests fundamental changes to how developers interact with their platforms.

Model Context Protocol

The Model Context Protocol (MCP), introduced by Anthropic and rapidly gaining adoption, provides a standardized approach for connecting AI systems with data sources and tools. Platform teams are beginning to expose their capabilities through MCP servers, enabling developers to interact with their platforms using natural language through AI assistants. Rather than navigating a web interface to provision a new service, a developer can simply describe what they need in conversation with an AI agent that understands the organization's platform capabilities.

Agentic AI

Agentic AI takes this further. Instead of passively waiting for developer requests, intelligent platform agents can proactively identify issues, suggest optimizations, and implement routine fixes autonomously. This shifts the platform team's role from building tools to defining the rules and organizational knowledge that enable these agents to operate safely.

These capabilities are not speculative. Organizations are already deploying AI-powered observability that automatically analyzes logs and flags anomalies before they impact production. Scaffolding workflows are becoming increasingly intelligent, generating not just boilerplate code but production-ready configurations tailored to the specific context. Documentation is being synthesized and queried through natural language interfaces rather than searched manually.

The Prerequisites for AI-Powered Platforms

The platform teams who will thrive in this environment are those who have already built strong foundational platforms. AI agents need reliable, well-documented APIs to interact with. They need accurate software catalogs to understand system relationships. They need established Golden Paths that encode organizational best practices. Organizations that have invested in mature Internal Developer Portals have the infrastructure in place to adopt AI capabilities rapidly.

The End of DIY

Platform Engineering is entering its "boring" phase, and this is exactly what success looks like. The fundamental patterns are established, and the technology choices are converging.

The teams achieving the best outcomes have recognized that maintaining commodity infrastructure is not a competitive advantage. They buy or use managed solutions for the interface layer, freeing their platform engineers to focus on the unique Golden Paths and integrations that actually differentiate their developer experience.

Building a developer portal is not the same as building a platform. The portal is the interface; the platform is the substance behind it. Organizations that conflate the two often find themselves with impressive storefronts and empty shelves. The path forward is to source the interface from a specialist and focus your energy on the substance.

Next Steps

If you're ready to move from concept to implementation, here are concrete next steps you can take:

Evaluate Your Build vs. Buy Decision: Before committing significant engineering resources to building and maintaining your own Backstage instance, consider the true cost of self-hosting. Understanding the full scope of ongoing maintenance, upgrades, and support requirements will help you make an informed decision.

Start with a Software Catalog: The foundation of any successful Internal Developer Portal is a comprehensive software catalog. Begin by modeling your existing services, APIs, and resources. Learn about effective strategies for creating a complete catalog to ensure discoverability across your organization.

Implement Self-Service Templates: Reduce friction and standardize best practices by deploying software templates that encode your Golden Paths. This accelerates onboarding and ensures consistency across your engineering organization.

Define Engineering Standards with Tech Insights: Move beyond anecdotal evidence and establish measurable engineering standards using data-driven scorecards. This helps you track compliance, identify gaps, and demonstrate continuous improvement.

Plan Your Adoption Strategy: Technical implementation is only half the battle. Develop a comprehensive adoption strategy that addresses organizational change management, stakeholder engagement, and measuring success metrics that matter to your leadership team.

Explore Managed Backstage Options: If you want to skip the lengthy setup and ongoing maintenance burden, try Roadie to get a production-ready Backstage instance deployed in minutes rather than months, allowing your team to focus on building unique platform capabilities instead of maintaining infrastructure.

Stop building the tool and start building the platform. Get a production-ready Backstage instance today with Roadie.

Here's the uncomfortable truth about internal developer portals in 2025: the market has fundamentally matured, and the old "build versus buy" calculus no longer applies. Companies that chose self-hosted Backstage discovered that "free and open-source" means 3-12 dedicated engineers. Organizations that bought proprietary platforms found themselves locked into data models they can't migrate away from. And those who picked the wrong solution are now facing a painful rip-and-replace.

The IDP landscape has evolved into three distinct approaches, each with radically different total cost of ownership:

• Build: Self-hosted Backstage installations offering maximum flexibility at the cost of $1M+ annual operational overhead.
• Buy: Proprietary SaaS platforms like Cortex and Port with polished interfaces but permanent vendor lock-in.
• Hybrid: Managed Backstage solutions delivering open-source ecosystem benefits without the engineering tax.

This guide evaluates the seven platforms that enterprise engineering leaders are actually deploying at scale, focusing on the strategic tradeoffs that matter three years after your initial decision, when you discover whether you made the right choice.

What Makes a Great Enterprise Developer Portal?

Before diving into specific platforms, let's establish the evaluation criteria that separate enterprise-ready IDPs from tools that work well in demos but fail in production.

Ecosystem and Extensibility

Can the platform integrate with your entire toolchain, or are you limited to what the vendor supports? At enterprise scale, you're likely using 50+ different tools across CI/CD, monitoring, security, and cloud infrastructure. The difference between supporting 20 integrations versus 250+ becomes critical when half your value comes from having everything in one place.

Vendor Lock-In Risk

If you decide to move away from the platform in two years, what happens to your data model? Open-source-based solutions like Backstage use standardized YAML entity definitions that you can export and migrate. Proprietary platforms often use custom data models that trap your organizational knowledge inside their systems.

Maintenance Overhead

Does running the platform require a dedicated team, or can your existing platform engineers manage it alongside their other responsibilities? Self-hosted solutions can consume 3-5 full-time engineers just for maintenance, upgrades, and troubleshooting. This is what we call the "TypeScript tax," the hidden cost of maintaining frontend infrastructure that most DevOps teams aren't equipped to handle.

Enterprise Readiness

Does the platform provide role-based access control (RBAC), single sign-on (SSO), and SOC2 compliance out of the box, or do you need to build these capabilities yourself? For regulated industries or companies with strict security requirements, these aren't nice-to-haves, they're table stakes.

Day 2 Operations

Look beyond the initial setup. How difficult is it to upgrade the platform when breaking changes occur? How do you handle search infrastructure at scale? What happens when you need to migrate to a new backend system? The platforms that look easiest on day one often become maintenance nightmares on day 700.

1. Roadie: The Backstage Platform for Enterprises

Category: Hybrid (Managed Backstage)

Best For: Teams that want Backstage's ecosystem without the operational burden

Roadie delivers the full power of Spotify's open-source Backstage platform as a managed SaaS service. This hybrid approach gives you access to the entire Backstage ecosystem, 211 open-source Backstage plugins, standardized data models, and active open-source development, while eliminating the maintenance overhead that typically requires 3+ dedicated engineers.

The platform handles all infrastructure concerns: hosting, security patches, database management, enterprise-grade search, and complex upgrades like the New Backend System migration that challenged self-hosted installations throughout 2024. Your team focuses on configuring integrations and building workflows, not on TypeScript debugging or React component updates.

Key Features:

• Minimal-maintenance Backstage platform with automated upgrades
• Access to entire open-source plugin ecosystem (211 plugins, 82 supported out-of-the-box)
• Built-in Tech Insights for scorecards and engineering standards (paid add-on)
• Enterprise RBAC (basic RBAC in Teams plan, custom RBAC in Growth plan)
• No vendor lock-in, data model is standard Backstage YAML

Pros:

• No TypeScript Tax: Platform engineering teams can focus on platform capabilities rather than maintaining TypeScript and React frontends.
• Open Ecosystem: Any Backstage plugin works, including community-developed integrations.
• Automatic Migrations: Complex upgrades like the New Backend System transition happen automatically.
• Faster Time-to-Value: Most customers see value within weeks, not months.
• Standard Data Model: Your catalog definitions are portable YAML files, not proprietary formats.

Cons:

• Backstage UI Constraints: Less drag-and-drop flexibility compared to tools like Port, though more structured.
• Requires SaaS Comfort: While Roadie offers secure connectivity options (like the Roadie Broker) for on-premises resources, it is primarily a hosted service.

Pricing: Teams plan starts at $24/developer/month with a 50-seat minimum (50-150 developers). Growth plan pricing is custom with a 100-seat minimum (100+ developers). Only active contributors to your source control management (SCM) incur costs, non-coding team members like product managers and leadership can access for free. Tech Insights is an optional paid add-on. View pricing details.

2. Cortex: The Engineering Metrics Specialist

Category: Proprietary SaaS

Best For: Organizations obsessed with service maturity scorecards and reliability metrics

Cortex offers a polished, opinionated developer portal focused heavily on measuring and improving service quality through scorecards and engineering metrics. The platform excels at gamifying service ownership with detailed maturity models, SLO tracking, and automated scoring based on engineering best practices using Bronze/Silver/Gold levels and point-based systems.

The UI feels modern and intuitive, particularly for teams familiar with SaaS tools like Datadog or PagerDuty. Cortex's scorecard system is more sophisticated than most alternatives, offering fine-grained control over scoring criteria with flexible rule definitions and excellent visualization of engineering standards compliance across your organization.

Key Features:

• Advanced scorecard system with customizable rubrics using Bronze/Silver/Gold levels and point-based scoring
• Strong reliability engineering focus (SLOs, incidents, on-call)
• Polished, modern UI optimized for service discovery
• AI-powered features including Ownership Prediction and Velocity Dashboard for DORA metrics
• 60+ out-of-the-box integrations with major monitoring and development tools

Pros:

• Scorecard Sophistication: Best-in-class service maturity tracking with Bronze/Silver/Gold level visualization and detailed point-based scoring.
• Beautiful UI: Modern design that impresses stakeholders.
• Strong Reliability Focus: Excellent for teams prioritizing SRE practices.
• Fast Initial Setup: Can get basic catalog running quickly.
• AI Integration: New AI features for ownership prediction and metrics analysis.

Cons:

• High Price Point: Known for being expensive at enterprise scale, especially compared to Backstage-based alternatives.
• Proprietary Lock-In: Data model is Cortex-specific, making migration difficult.
• Limited Ecosystem: Rely on Cortex to build integrations, can't leverage community plugins.
• Scaffolding Limitations: Template/workflow capabilities lag behind Backstage's Software Templates.

Pricing: Not publicly disclosed; requires signing up for a demo to receive pricing information. However, the Forrester Total Economic Impact study from July 2024 lists pricing at approximately $65/user/month at scale. Multiple tiers available (Engineering Intelligence, Accelerate, Full IDP, Site License) with features scaling from basic catalog and scorecards to full platform capabilities with AI-powered features. Request pricing.

3. Port: The No-Code Builder's Platform

Category: Proprietary SaaS

Best For: Teams that need to model non-standard assets or want maximum UI customization

Port takes a radically different approach: instead of providing an opinionated developer portal, it gives you building blocks to create your own. The platform's "no-code" interface lets you define custom data models (called Blueprints) for any asset type, not just services and APIs, but also environments, IoT devices, or cloud resources.

This flexibility makes Port uniquely suited for organizations with complex, non-standard infrastructure that doesn't fit typical service catalog patterns. You can build custom views, define relationships between any entity types, and create workflows that match your exact processes. Port has recently rebranded as an "Agentic Internal Developer Portal" with enhanced AI capabilities.

Key Features:

• Fully customizable data models and UI views via Blueprints
• No-code interface for defining entities and relationships
• Strong visualization capabilities for complex systems
• Self-service actions using Cookiecutter templates
• 50+ integrations including DORA metrics tracking
• AI agent capabilities and Engineering360 dashboard

Pros:

• Ultimate Flexibility: Model anything your organization needs, not just traditional services.
• Custom Views: Build exactly the interface your teams need.
• Good for Complex Infrastructure: Excellent for multi-cloud, hybrid environments with diverse asset types.
• Visual Workflow Builder: Create automation without writing code.

Cons:

• Blank Slate Problem: Maximum flexibility means you build everything from scratch; less out-of-the-box value.
• Proprietary Ecosystem: Cannot leverage the Backstage open-source plugin ecosystem, though the Ocean Framework provides extensibility through data integrations and workflow automation.
• Weaker Documentation Features: While it supports Markdown, it lacks the full TechDocs build pipeline and search capabilities found in Backstage.
• Steeper Learning Curve: Teams need time to master the data modeling concepts.

Pricing: Free tier available (up to 15 seats, 10,000 entities). Startup tier at $30/developer/month. Enterprise tier available with premium features including SSO, advanced RBAC, ISO 27001 and SOC2 Type 2 certifications, and dedicated support. View pricing details.

4. OpsLevel: The Service Maturity Tracker

Category: Proprietary SaaS

Best For: Teams focused primarily on service ownership and maturity tracking

OpsLevel started as a service maturity and ownership tool before evolving into a broader developer portal. This heritage shows in its excellent service ownership features and straightforward approach to tracking engineering standards compliance through its "Rubric" system with Bronze/Silver/Gold levels, plus separate Scorecards for team-specific standards.

The platform offers the fastest time-to-initial-value for basic service cataloging, with typical deployments completing in 30-45 days. You can have a working catalog with ownership information and basic checks running within hours. However, this simplicity comes at the cost of extensibility, OpsLevel's feature set is more constrained than platforms built on extensible architectures. Recent additions include AI-powered features for check generation and catalog enrichment.

Key Features:

• Fast setup for basic service catalog (typical 30-45 day deployment)
• Strong focus on service ownership and maturity rubrics
• Good integration with CI/CD systems for automated checks (60+ integrations)
• AI-generated checks and AI-enriched catalog
• Package version inventories for SBOM visibility
• Clean, straightforward UI

Pros:

• Quickest Setup: Get a basic catalog running faster than any alternative.
• Service Ownership Focus: Excellent for clarifying who owns what.
• Maturity Rubrics: Good built-in templates for measuring service quality.
• Intuitive Interface: Less complex than more feature-rich platforms.
• AI Integration: New AI features streamline catalog management.

Cons:

• Limited Extensibility: Can't add community plugins or build custom integrations easily.
• Rigid UI: Less flexible than Port or Backstage for customization.
• Narrower Feature Set: Primarily focused on cataloging and checks, less emphasis on docs or scaffolding.
• Proprietary Lock-In: Migration path unclear if you outgrow the platform.

Pricing: Not publicly disclosed, pricing based on team size with custom quotes. Per-developer pricing model with volume discounts available. Includes SOC2 Type 2 compliance and SAML-based SSO. Pricing customizable based on needs including self-hosted options and support levels. Request pricing.

5. Atlassian Compass: The Jira Companion

Category: Proprietary SaaS

Best For: Organizations deeply invested in the Atlassian ecosystem

If your company runs on Jira, Bitbucket, and Confluence, Compass offers the most seamless native integration you'll find. The platform leverages Atlassian's identity system, pulls in data from other Atlassian products automatically, and feels like a natural extension of your existing toolchain.

Compass provides automated service health monitoring, tracking metrics from integrated tools and surfacing problems before they escalate. For teams already paying for Atlassian products, Compass represents an incremental cost with minimal integration effort. The platform has scorecards with a new "Maturity Levels" feature added in 2025.

Important Note: Atlassian deprecated the Templates/scaffolding feature on December 1, 2025. This represents a significant capability reduction for teams requiring self-service service creation.

Key Features:

• Native integration with Jira, Bitbucket, Confluence, and Opsgenie
• Automated service health monitoring
• Component tracking with Atlassian-native data models
• Integrated incident management through Opsgenie
• Scorecards with Maturity Levels feature
• Built on Atlassian Forge with GraphQL APIs for extensibility

Pros:

• Atlassian Integration: Unbeatable if you use Jira for everything.
• Familiar UX: Feels consistent with other Atlassian products.
• Automated Health Monitoring: Good automated health checks.
• Standalone or Bundled: Available as standalone product or with some enterprise packages.

Cons:

• The Atlassian Trap: Struggles with non-Atlassian tools like GitHub, GitLab, or CircleCI.
• Scaffolding Removed: Templates feature deprecated December 1, 2025, no longer available for service creation.
• Proprietary Ecosystem: Can't extend with community plugins.
• Not a Full IDP: More of a service catalog with add-ons than a complete platform interface.

Pricing: Free tier available (3 full users, unlimited basic users). Standard tier at $8/user/month includes basic features. Premium tier at $25/user/month includes IP Allowlisting, advanced integrations, 99.9% uptime SLA, and premium support. Discounted rates available for teams above 101 users. Compass is a standalone product with separate billing from other Atlassian tools. View pricing details.

6. Backstage (Self-Hosted)

Category: Open Source (Self-Hosted)

Best For: Large enterprises with dedicated platform engineering teams and specific compliance requirements

Spotify's Backstage is the industry-standard open-source developer portal framework. It powers IDPs at Spotify, American Airlines, Pinterest, and thousands of other organizations. The platform offers ultimate flexibility: you own the code, control the infrastructure, and can customize anything.

But this flexibility comes with significant operational costs. Self-hosting Backstage requires 3-5 dedicated engineers to manage infrastructure, handle upgrades, maintain search systems, and keep up with the rapidly evolving codebase. Roadie's survey of the Backstage community found that users reporting satisfaction with self-hosted deployments had at least three engineers dedicated full-time, with some companies staffing teams of 12 people just for Backstage. Breaking changes occur regularly with monthly releases, and major migrations like the New Backend System transition that completed in 2024 consumed months of engineering time for self-hosted teams.

For perspective on the true cost of building similar developer portals, Zalando invested over $4 million across four years developing their internal platform before open-sourcing their work as part of Backstage.

Key Features:

• Fully open-source with Apache 2.0 license
• 250+ community plugins covering every major tool
• Extensible architecture for custom plugins and integrations
• Active community and regular monthly releases
• CNCF Incubating project with strong enterprise adoption

Pros:

• Ultimate Control: You own everything and can customize without limits.
• No License Costs: Free to download and use.
• Massive Ecosystem: Largest community and plugin library (250+ plugins).
• No Vendor Dependency: Run anywhere, modify anything.
• Industry Standard: Backed by CNCF and major enterprises.

Cons:

• Operational Overhead: Requires 3-5 FTE engineers minimum for successful deployments, some teams reach 12 FTEs.
• The TypeScript Tax: Most DevOps teams lack frontend skills for React/TypeScript customization.
• Upgrade Complexity: Monthly breaking changes and major migrations (like New Backend System) require significant engineering investment.
• Day 2 Operational Burden: You manage databases, search infrastructure (Elasticsearch), monitoring, and security patches.
• Hidden Costs: The real cost when factoring in engineering time, opportunity cost, and infrastructure can reach millions of dollars. At typical senior platform engineer compensation ($250K/year fully loaded), 3-5 engineers cost $750K-$1.25M annually, plus infrastructure costs ($12K-$24K/year). TCO typically exceeds $2M+ over three years when including engineering time and opportunity cost.

Pricing: Free (open source under Apache 2.0 license). However, total cost of ownership includes 3-5 engineer salaries ($750K-$1.25M+ annually at senior platform engineer compensation of $250K/year fully loaded) plus infrastructure costs ($12K-$24K+ annually for hosting, databases, and search infrastructure). TCO typically exceeds $2M+ over three years when factoring in engineering time, opportunity cost, and infrastructure.

7. Configure8: The Universal Catalog Platform

Category: Proprietary SaaS

Best For: Organizations prioritizing discovery and cost analytics

Configure8 positions itself as a "universal catalog" that can ingest and relate data from virtually any source. The platform emphasizes discovery features, helping teams understand what they have and how it's interconnected. It also offers strong cloud cost integration, surfacing spending data alongside technical resources.

While Configure8 has solid core features including 30+ integrations and workflow-based Self-Serve Actions, its smaller market presence and proprietary nature make it a riskier choice than platforms with larger ecosystems or open-source foundations.

Key Features:

• Universal catalog supporting diverse asset types
• Strong discovery and search capabilities
• Cloud cost analytics integration
• Relationship mapping across resources
• Workflow-based Self-Serve Actions
• Available as SaaS or on-premises deployment

Pros:

• Comprehensive Discovery: Good at helping teams understand what exists.
• Cost Integration: Unique focus on cloud spending alongside technical resources.
• Multi-Source Ingestion: Can pull data from many systems.
• Deployment Flexibility: Offers both SaaS and on-prem options.

Cons:

• Smaller Ecosystem: Less community support and fewer integrations (30+) than larger platforms.
• Proprietary Lock-In: Data model is Configure8-specific.
• Limited Track Record: Fewer public case studies and enterprise deployments than alternatives.
• Market Position Uncertainty: Smaller player in a competitive market.

Pricing: Free tier available (up to 10 users for scorecards). Paid tiers available with SOC2 certification and RBAC features. Enterprise pricing available with additional features and volume discounts. Available as both SaaS and on-premises deployment. Contact Configure8 for detailed pricing information. View pricing page.

Comparison: At a Glance

| Platform | Foundation | Maintenance | Ecosystem Size | Lock-In Risk | Enterprise Features | | :--- | :--- | :--- | :--- | :--- | :--- | | Roadie | Open Source (Backstage) | Minimal (Managed) | 211 plugins | Low (standard YAML) | RBAC, SSO, SOC2 Day 1 | | Cortex | Proprietary | Minimal (SaaS) | 60+ integrations | High (proprietary) | Strong | | Port | Proprietary | Minimal (SaaS) | 50+ integrations | High (proprietary) | Good | | OpsLevel | Proprietary | Minimal (SaaS) | 60+ integrations | High (proprietary) | Basic | | Compass | Proprietary | Minimal (SaaS) | Atlassian-centric | High (proprietary) | Good (if Atlassian) | | Backstage | Open Source | High (3-12 engineers) | 250+ plugins | None | DIY | | Configure8 | Proprietary | Minimal (SaaS) | 30+ integrations | High (proprietary) | Moderate |

How to Choose the Right Developer Portal

The right IDP depends on your organization's constraints, technical culture, and platform engineering maturity:

• Choose Self-Hosted Backstage if you have a team of 10+ platform engineers, unlimited budget, and specific requirements that absolutely cannot be met by managed solutions. You're willing to invest significant engineering time in maintenance and customization for maximum control. Be prepared for 3-12 dedicated engineers and $2M+ total cost of ownership over three years.
• Choose Cortex or Port if you value polished UI above all else, don't mind proprietary lock-in, and want specific workflow capabilities their platforms emphasize. Budget for potentially higher costs at scale (~$65/user/month for Cortex, ~$30+/user/month for Port enterprise tiers).
• Choose OpsLevel if you need a basic service catalog immediately and your primary use case is tracking ownership and maturity, not building complex workflows or maintaining extensive documentation.
• Choose Compass if you live entirely in the Atlassian ecosystem and can accept its limitations with non-Atlassian tools. Note that scaffolding/templates were deprecated December 1, 2025. The integration efficiency may outweigh the platform's constraints if you're already in the Atlassian ecosystem.
• Choose Roadie if you want the industry standard (Backstage) with its entire ecosystem and community support, but you value your engineers' time too much to spend it on infrastructure maintenance. It's the "golden path" for enterprises that want modern platform capabilities without the platform tax or the $2M+ cost of building from scratch.

The key question isn't which platform has the most features, it's which platform lets your engineers focus on building platform capabilities instead of maintaining platform infrastructure. At the 150+ engineer scale where IDPs become critical, that distinction determines whether your portal becomes a force multiplier or just another thing to maintain, potentially at a cost exceeding $2M over three years if you go the self-hosted route.

Next Steps

Ready to implement a developer portal for your organization? Here are some practical next steps to guide your journey:

Evaluate Roadie with a Free Trial: Experience the power of managed Backstage firsthand. Start your free trial to explore Roadie's features, integrations, and see how quickly you can get value without the operational overhead.

Learn from Customer Success Stories: See how other enterprise teams have successfully implemented developer portals. Read our case studies to understand real-world adoption strategies, challenges overcome, and measurable results achieved.

Dive Deeper into Backstage: If you're new to Backstage or want to understand its capabilities better, explore our comprehensive guide to Backstage and learn why it's become the industry standard for developer portals.

Plan Your Implementation: Moving from evaluation to production requires careful planning. Our guide on planning and implementing Backstage from Day 0 to Day 2 provides a roadmap for successful adoption.

Compare Your Options: Still weighing self-hosted versus managed solutions? Read our detailed comparison of managed vs. self-hosted Backstage to understand the true total cost of ownership.

Schedule a Demo: See Roadie in action with your specific use cases. Request a personalized demo to discuss your requirements, explore integrations, and understand how Roadie can accelerate your platform engineering initiatives.

Teams kept deploying more stuff onto our platform. The security team had services running. There was a virus scanning team. Dozens of other teams I didn't even know existed were all running their software on our infrastructure. At around 50 services, people started asking questions we couldn't answer.

Who owns this thing? If it goes down at 2am, who do I call? Is it secure? How many resources should we allocate to it?

The Spreadsheet Solution

To help answer these questions, we started with what everyone starts with: a spreadsheet. One column for service names, another for owners, another for what it does.

It lasted about three weeks before it became useless. Nobody updated it. The information got stale. We had services running in production that weren't in the spreadsheet and entries for services that had been decommissioned months ago.

So we did what any self-respecting infrastructure team would do, we built a custom solution.

The $2 Million Developer Portal

We built a UI that could list all the services, show where they were running, and let people create new services with the click of a button. You'd fill out a form, and you'd get a repo with code that would run perfectly on our platform. You could configure your networking rules without going through ten Slack messages trying to find the right person. It was genuinely useful.

When I look back at the team size and time involved, I estimate we spent about $2 million USD building that thing from scratch. That isn't an exaggeration; it’s standard math when you factor in the salaries of the engineers, product managers, and designers required to build and maintain an internal product over several years.

And here's the thing: when I talked to people at other companies, they'd all built some version of the same solution to the same problem.

The Turning Point for Developer Tools

Fast forward to 2020. I'm leaving that role, and I can't stop thinking about this problem. Companies are clearly willing to spend millions solving it.

Around the same time, Spotify open-sourced Backstage, their internal developer portal that they'd been using since 2016. Suddenly, there were two approaches:

• Proprietary solutions like Cortex and OpsLevel were building closed-source developer portals and selling them as SaaS products. Quick to set up, but you're locked into their data model and their way of doing things.
• Open-source Backstage gave you maximum flexibility. You could customize everything. But it came with a catch.

The Hidden Cost of "Free"

Here's what people don't realize about Backstage until they're deep into implementation: it's not a developer portal. It's a framework for building developer portals. It's TypeScript libraries that you use to construct your own solution.

That's a problem for platform teams because most of them don't know TypeScript. They know Go and YAML and infrastructure-as-code. Not React and frontend frameworks.

When we surveyed the Backstage community, we found something interesting: people who reported being happy with their self-hosted Backstage deployment had at least three engineers dedicated to it full-time. Some companies had teams of 12 people just working on Backstage.

Think about that. You're trying to solve the developer productivity problem, and you need to staff an entire team to maintain your solution. That's not solving the problem, that's just moving it around.

And keep in mind, there are features Backstage doesn't give you out of the box. Role-based access control? Build it yourself. Better search than Postgres full-text? Run and maintain your own Elasticsearch cluster.

It takes about a year to get a self-hosted Backstage instance into production. A year and a team of five people. Or you could go with a proprietary solution and be live in weeks, but give up all the flexibility and customization that made Backstage so attractive in the first place.

The Best of Both Worlds

We built Roadie to give people the power of Backstage's open-source ecosystem, all those plugins, the community contributions, and the flexibility, without requiring them to staff a team around it.

When you use Roadie, you get access to the hundreds of open-source plugins that the Backstage community maintains. Every time someone contributes an improvement to the Backstage core, you get it automatically. You're part of this massive ecosystem that's constantly getting better.

But you don't need a team of TypeScript engineers. You don't need to spend six months setting it up. You log in on day one and start using it.

We handle all the infrastructure, all the upgrades, all the features that Backstage doesn't include by default. We've added role-based access control, better search, and scorecards for tracking standards.

Who Comes to Roadie and What Problems We Solve

Here's what we see when companies come to us. They're usually in one of three situations:

They already have Backstage. Someone got excited about it, stood up an instance, and now they're realizing it's eating up way more engineering time than they expected. They want to flip it to Roadie without starting over.

They want Backstage specifically. They love the open-source ecosystem and the flexibility, but they don't have the bandwidth to build and maintain it themselves. They're doing the build-versus-buy calculation and realizing Roadie is cheaper than five full-time engineers.

They just want a developer portal. They don't particularly care if it's Backstage or something else. They have the discoverability problem—too many services, nobody knows who owns what—and they need to solve it.

For the first two groups, the decision is easy. For the third group, it comes down to whether they value flexibility and avoiding vendor lock-in. If they do, Backstage through Roadie makes sense. If they want something more opinionated and are okay with proprietary data models, other solutions might work.

These companies are trying to solve three main problems:

• Discoverability. This is the main complaint we see, and the same problem we had when I frist worked on developer portals. "We have too much stuff and nobody knows what any of it is." This extends to centralizing documentation, mapping service dependencies, cataloging API specs. It all falls under the umbrella of making it possible to find things.
• Self-service automation. Platform teams want developers to be able to get things done without opening JIRA tickets and waiting days. Need to create a new service? There should be a button for that. Need an S3 bucket? Fill out a form that generates a Terraform pull request instead of waiting for someone to do it manually. Our software templates feature handles this.
• Governance and standards. Leadership wants to know: is our software secure? Is it reliable? Are teams following our best practices? They want automated checks—is this service properly configured in our incident management tool? Does it have someone on-call? Are there critical security vulnerabilities we haven't addressed? Tech Insights makes this possible.

A developer portal doesn't magically solve these problems. But it gives you a place to tackle them systematically instead of through a patchwork of spreadsheets, Slack messages, and tribal knowledge.

What No Tool Can Fix

The biggest mistake I see companies make is thinking this is just a technical problem. They'll come to me and say "We want DORA metrics in our portal" and I'll ask "What's a deployment in your organization?" and then we discover that half their teams deploy through ArgoCD, some use Netlify, and others have their own custom pipeline.

You can't measure deployment frequency across your organization until you agree on what a deployment is. No tool can solve that for you, not Backstage, not Roadie, not anyone. That's an organizational alignment problem disguised as a technical one.

The same thing happens with service catalogs. "Show me all our services" sounds simple until someone asks "What's a service?" and three engineering managers give you three different answers.

Developer portals give you a forcing function to solve these problems and a place to encode the answers once you've figured them out.

Where Roadie's Headed

Two things are taking up most of our roadmap energy right now.

AI Integration

When you think about the questions people want answered, "Who's the engineering manager for the virus scanning service?" or "Which services depend on our authentication API?", they're perfect for conversational interfaces. You should be able to type that into a chat and get an answer. We're building that.

Automated Ingestion

This is less sexy but maybe more important. Right now, getting data into Backstage is too manual. You have to go to each team and say "put your stuff in the catalog" and that doesn't scale. We need to automatically discover and ingest services, infrastructure, and dependencies. We're making Backstage better at this than the proprietary competitors, which is crucial for wider adoption.

The Hybrid Approach

Here's how I explain Roadie to people: self-hosted Backstage sucks because you need a team of five engineers. Proprietary developer portals suck because you're locked into their data model and limited customization.

Roadie gives you the best of both. You get the massive open-source ecosystem that's constantly improving through community contributions. But you don't need to staff a team around it. You don't need to spend a year bringing it into production.

We built Roadie because we lived through building it from scratch. We know exactly how much it costs and how long it takes. And we knew there had to be a better way.

Request a demo or start a free trial to see how it works for yourself. You can also check out this case study to see how Celonis switched from self-hosted Backstage to Roadie and the results they achieved.

This adage has never been truer than it is for Spotify's Backstage in 2025. While the repository is free to clone, turning that code into a secure, scalable, and adopted Internal Developer Portal (IDP) is a massive financial and operational undertaking.

For engineering leaders, the question isn't "Can we build this?" As Roadie CEO David Tuite highlighted at BackstageCon, the real question is: "Should we spend millions of dollars a year to maintain this?"

In this guide, we break down the real math behind self-hosting Backstage. We look beyond the initial setup and examine the deep technical challenges, from the misalignment of skills on your platform team to the relentless pace of upgrades, that make "free" software incredibly expensive.

1. The Headcount Reality

Before discussing technology, we must look at the human capital required. Backstage is not a tool you install; it is a product you build and maintain.

Our research into the State of Backstage found that organizations satisfied with their self-hosted setup typically dedicate 3 to 12 full-time engineers to the platform.

If you assign a single engineer to "figure it out," they will spend 100% of their time on maintenance—dependency updates, security patches, and keeping the lights on—leaving zero capacity for feature development or driving adoption.

2. The "TypeScript Tax": A Fundamental Skill Mismatch

Beyond the number of heads, there is the issue of the type of skills required. Backstage is not a binary you configure with YAML; it is a set of libraries written in TypeScript and React that your engineers can combine together to build an IDP.

This presents a critical problem for most infrastructure and platform teams.

The DevOps Skill Set: Your platform engineers are likely experts in Go, Python, Terraform, and Kubernetes. They live in the terminal and think in terms of infrastructure-as-code.

The Backstage Requirement: To customize Backstage, you need to write React components, debug frontend hooks, manage CSS themes, and handle complex Node.js build pipelines.

The Reality: When you ask a Senior Cloud Engineer to center a <div> or debug a React context error, you are not only misusing their expensive skillset, but you are also setting them up for frustration. Even if they can technically write the code needed to customize Backstage, these engineers won't have the user experience skills required to produce a portal that your teams love to use.

To run Backstage successfully, you essentially need to hire Frontend Platform Engineers, a niche and expensive role, or constantly borrow time from product teams who would rather be shipping features.

3. The Hidden Infrastructure Bill

Getting Backstage running on a laptop takes ten minutes. Getting it ready for the enterprise takes months. When you choose to self-host, you take ownership of a distributed system that requires significant "plumbing."

Authentication & Security

Backstage provides the framework, but you build the doors.

The Work: You must manually register OAuth applications (Okta, Google, GitHub), configure backend resolvers, handle session cookies, and manage CORS policies.

The Risk: You are responsible for vulnerability management. Backstage relies on hundreds of Node.js dependencies. When a security advisory comes out for a nested dependency, you are the one who has to patch, rebuild, and redeploy.

The TechDocs Pipeline

TechDocs (docs-like-code) is a flagship feature, but it requires external infrastructure to work at scale.

The Work: It is not "plug and play." You must provision cloud storage (AWS S3, GCS), configure IAM roles, and build a dedicated CI/CD pipeline to generate static HTML from Markdown and push it to storage.

Search Infrastructure

The default search engine is basic. For a production-grade experience, you need to manage a dedicated Elasticsearch or OpenSearch cluster, adding another layer of cost and maintenance.

4. The Maintenance Treadmill (Day 2 Operations)

Backstage moves fast. According to the 2025 State of Backstage report, 56% of adopters cite upgrades as their biggest pain point.

Unlike other infrastructure tools where an upgrade is often just bumping a version number in a config file, upgrading Backstage often requires code changes.

Breaking Changes: You will frequently need to refactor your App.tsx or backend wiring to accommodate changes in the core APIs.

The New Backend & Frontend Systems: In 2024, Backstage introduced a completely new backend architecture followed closely by a new frontend system. Teams self-hosting Backstage are currently spending months refactoring their entire plugin ecosystem and UI logic to migrate to this new system.

Plugin Rot: Open-source plugins are often abandoned. If a community plugin you rely on breaks, you become the maintainer.

5. The Financial Math: Calculating TCO

Now, let's translate that technical effort into dollars.

The Cost of Waiting (Time to Value)

First, consider the cost of not having an IDP while you build it. Self-hosted implementations typically take 6 to 12 months to reach production. That is a year of paying salaries without seeing value. In contrast, managed solutions like Roadie typically go live in under a month, accelerating your ROI significantly.

The Hard Costs 

The single largest cost of self-hosting is Headcount.

 You are building an internal product team. Based on data from hundreds of implementations, a "minimum viable" self-hosted setup requires:

• Year 1 (Build & Launch): 3 Full-Time Engineers (FTEs).
• Year 2+ (Maintenance): 2 FTEs to handle upgrades, migrations, and support.

The 2025 Math: According to 2025 market data, the fully loaded cost (salary, equity, benefits) of a Senior Platform Engineer in top tech hubs averages $125,000.

| Cost Category | Year 1 Cost (DIY) | Recurring Annual Cost (DIY) | |--------------------------------------------------------|-------------------------|----------------------------------| | Engineering Salaries (3 FTEs Year 1, 2 FTEs Year 2) | $750,000 | $500,000 | | Cloud Infrastructure (Hosting, DB, Elasticsearch) | $12,000 | $12,000 | | Maintenance & Upgrades (20% of time) | Included in Salary | Included in Salary | | Total TCO | $762,000 | $512,000 |

6. Comparison: Self-Hosted Backstage vs. Roadie

Roadie provides the exact same underlying technology (Backstage) but removes the technical implementation layer.

| Feature / Requirement | Self-Hosted Backstage | Roadie (Managed) | |---------------------------|----------------------------------------|----------------------------------------------------| | Time to Value | 6 to 12 Months | < 1 Month | | Required Skills | TypeScript, React, Node.js | No Code / Config Only | | Team Required | 2–3 Full-Time Engineers | Part-time Admin | | Upgrades | Manual Code Refactoring | Automated / Handled by Roadie | | Search Engine | Manage your own Elasticsearch | Included & Optimized | | Security / SOC2 | You handle the audit | SOC2 Type 2 Certified |

7. The Opportunity Cost

This is the cost that doesn't show up on a spreadsheet, but it's the one that kills momentum.

If your best Platform Engineers are spending their weeks debugging React components, upgrading Node.js versions, or fixing broken TechDocs pipelines, they are not building your platform.

They aren't optimizing your Kubernetes clusters, they aren't improving your CI/CD pipelines, and they aren't helping product teams ship faster. They are maintaining a CMS tool.

Conclusion: Buy the Outcome, Don't Build the Tool

Backstage is the standard for IDPs, but "free" software is expensive.

Roadie allows you to buy the outcome, a centralized, adopted, and secure developer portal, without paying the tax of building and maintaining it yourself.

Ready to see how Roadie can transform your developer experience? Contact us for a personalized demo or start your free trial today.

Next Steps

If you're evaluating whether to self-host Backstage or use a managed solution like Roadie, here are some recommended resources to guide your decision:

• Getting Started with Roadie - Learn how to quickly set up your Internal Developer Portal with minimal overhead and start seeing value in days instead of months.
• Explore Backstage Plugins & Integrations - Discover the 80+ pre-configured plugins and integrations that Roadie supports out of the box, eliminating weeks of setup and configuration work.
• Tech Insights: Scorecards for Engineering Standards - Understand how to define and track engineering standards across your organization without building custom tooling.
• Software Templates (Scaffolder) - Learn how to create golden paths to production that embed best practices and accelerate service creation.

If you are looking for alternatives, you have likely realized that adopting Backstage is a major engineering undertaking. It requires a dedicated team, TypeScript expertise, and months of setup. For many teams, the operational overhead simply outweighs the value.

However, the problem Backstage attempts to solve is unavoidable. As organizations grow, they inevitably hit a ceiling known as the Dunbar Number Effect. Anthropologist Robin Dunbar proposed that humans can only maintain a stable social network of about 150 people. Beyond this number, tribal knowledge evaporates, ownership becomes unclear, and the informal communication channels that worked for a startup turn into chaos.

You need a system to solve this chaos—a "system of record" for your engineering team. But you don't necessarily need the complexity of self-hosted Backstage to get it.

This guide covers the best Backstage alternatives for 2026. We will outline the three paths you can take, Build, Buy, or Hybrid, and help you find the right solution to conquer the Dunbar Number without drowning in maintenance work.

The Three Paths to an IDP: Build, Buy, or Hybrid

Before we dive into the specific tools, it's crucial to understand the three fundamental approaches you can take:

1. Build (Self-Hosted Backstage): You take the open-source Backstage project and dedicate a team of engineers to build, customize, and maintain your own portal. This offers ultimate flexibility but comes with significant headcount and operational costs.
2. Buy (Proprietary IDPs): You purchase a SaaS solution from a vendor like Cortex or Port. These platforms are often quick to set up and feature-rich, but they lock you into a proprietary data model and ecosystem.
3. Hybrid (Managed Backstage): You use a service like Roadie, which handles the hosting, maintenance, and enterprise-grade features for Backstage. This gives you the power of the open-source ecosystem without the operational burden, offering a "best of both worlds" approach.

Backstage Alternatives: At a Glance

| Tool | Core Technology | Hosting Model | Key Strength | Ideal For | |---------------------|------------------------|----------------|-------------------------------------------------|-----------------------------------------------------------------------------------------------| | Roadie | Backstage | SaaS (Managed) | The Backstage ecosystem without the overhead. | Teams who want Backstage’s open-source power but need a fast, managed, enterprise-ready solution. | | Cortex | Proprietary | SaaS | Engineering metrics & scorecards. | Organizations focused on measuring and improving service quality and engineering performance. | | Port | Proprietary | SaaS | Developer-friendly API & flexibility. | Teams that want to build custom workflows and integrations on a flexible platform. | | Atlassian Compass | Proprietary | SaaS | Deep Atlassian ecosystem integration. | Companies heavily invested in the Atlassian stack (Jira, Confluence, Bitbucket). | | OpsLevel | Proprietary | SaaS | Service maturity & reliability checks. | SRE and platform teams focused on enforcing production readiness standards. | | Self-Hosted Backstage | Backstage (OSS) | Self-Hosted | Ultimate customization and control. | Large organizations with a dedicated platform team (5+ engineers) to manage the instance. |

The Hybrid Approach: Managed Backstage

Roadie

Roadie isn't just an alternative to Backstage, it's a different way to adopt Backstage. It’s built on the belief that you shouldn't have to choose between the power of a vibrant open-source community and the convenience of a SaaS product.

As we discussed in our founding story, the "aha!" moment for Roadie was realizing that while Backstage is a fantastic framework, it requires a dedicated team of 3-12 engineers and a 6-12 month investment to become production-ready. Roadie solves this by providing a secure, scalable, and fully managed Backstage experience out of the box.

• Best for: Teams who have decided on Backstage for its extensibility as their platform but want to accelerate time-to-value and reduce their operational burden.
• Key Features:
  • Get a production-ready Backstage instance in minutes, not months. Roadie handles upgrades, security, and maintenance.
  • Includes critical features missing from open-source Backstage, like Role-Based Access Control (RBAC) enterprise-grade search, and scorecards, from day one.
  • Instantly access and install all of the best open-source Backstage plugins without the hassle of rebuilding your instance.
• Considerations: Because Roadie uses Backstage as its foundation, it shares the same data model and core experience. Teams looking for a completely different, highly opinionated UI may be better served by a proprietary vendor.

The "Buy" Approach: Proprietary IDPs

Cortex

Cortex has established itself as a leader in the IDP space, with a strong focus on service quality, reliability, and engineering metrics. It excels at helping platform teams define standards and drive adoption through its powerful Scorecards feature.

• Best for: Organizations focused on establishing and tracking engineering standards and service maturity.
• Key Features: A central inventory for all microservices, applications, and APIs; scorecards to define rules and initiatives for service health; and a scaffolder to create new services from templates.
• Considerations: Cortex is a proprietary platform. While powerful, you are locked into its data model. Migrating away in the future could be a significant undertaking.

Port

Port is designed for flexibility, centered around a developer-friendly API that allows teams to ingest any data and build custom workflows. It positions itself as a platform for building a developer portal, rather than a rigid, out-of-the-box solution.

• Best for: Platform teams with strong development capabilities who want to build highly custom developer experiences and workflows.
• Key Features: A highly flexible "blueprint" model to define any asset, a self-service hub for custom actions, and scorecards to track quality and security metrics.
• Considerations: Port's flexibility is its greatest strength but can also mean a steeper learning curve and more initial setup compared to more opinionated platforms.

Atlassian Compass

Atlassian Compass is Atlassian's entry into the developer experience space. Its primary advantage is its seamless integration with the broader Atlassian ecosystem, making it a natural choice for teams already standardized on Jira, Confluence, and Bitbucket.

• Best for: Companies deeply embedded in the Atlassian suite of tools.
• Key Features: A component catalog to track ownership, health scorecards to monitor best practices, and deep, native integration with other Atlassian products.
• Considerations: Its greatest strength is also its weakness. If you are not an Atlassian-centric organization, Compass may feel less integrated and compelling compared to other options.

OpsLevel

OpsLevel is a mature player that focuses heavily on service ownership and reliability, making it a favorite among SRE and platform teams. It helps organizations answer the question, "Is our software ready for production?"

• Best for: SRE-driven organizations looking to enforce service maturity standards and improve on-call processes.
• Key Features: A complete service catalog, an extensive library of automated maturity checks, and integrations with on-call tools like PagerDuty.
• Considerations: OpsLevel's focus is more on reliability and standards than on developer self-service and scaffolding, which are stronger in other platforms.

The "Build" Approach: Self-Hosted Backstage

Choosing to self-host Backstage is a significant engineering commitment. It should be treated as building an internal product, not just deploying a tool.

• Best for: Large organizations with a well-funded, dedicated platform team (5+ engineers) that has a clear mandate to build and maintain a highly customized developer portal.
• Key Features: You have complete control over the code and can mold the platform to your exact specifications, and you own and control your instance and data model.
• Considerations: This path carries a high cost of ownership. As detailed in our cost analysis of self-hosting Backstage, you must account for the fully-loaded salaries of a dedicated engineering team, the 6-12 month initial build time, and the ongoing operational burden of maintenance and upgrades.

How to Choose the Right Path for You

Making the right choice depends entirely on your organization's priorities, resources, and philosophy. Ask your team these questions:

1. How important is the open-source ecosystem to us?

If you want to avoid vendor lock-in and leverage the innovation of a massive community, your choice is between self-hosting Backstage or using a managed service like Roadie. If you prefer the all-in-one experience of a single vendor, a proprietary option like Cortex or Port is a better fit.

2. What is the size and skill set of our platform team?

If you have a team of 5-10 engineers with TypeScript experience and a mandate to build a custom portal, Self-Hosted Backstage is a viable path. If your platform team is smaller, or you want them focused on other priorities, Roadie or a proprietary vendor is the more efficient choice.

3. What is our most critical "job to be done" right now?

• If you prefer a proprietary, opinionated model for scorecards and don't mind vendor lock-in, tools like Cortex or OpsLevel offer polished but rigid solutions.
• If you want to build custom workflows from scratch and are comfortable within a closed-source ecosystem, Port offers a flexible proprietary API.
• If your organization exists entirely within the Atlassian suite and you don't require broad third-party integrations, Atlassian Compass is a natural extension of that walled garden.
• If you want enterprise-grade features (Scorecards, RBAC, Self-Service) combined with the freedom and extensibility of the open-source ecosystem, Roadie is your answer.

Ultimately, an Internal Developer Portal is a long-term investment in your developer experience. By understanding the fundamental trade-offs between the "Build," "Buy," and "Hybrid" approaches, you can make a decision that will empower your teams for years to come.

Ready to see the hybrid approach in action? Try Roadie for free.

To get an accurate picture of how teams are navigating this shift, we surveyed 105 active Backstage practitioners across industries, company sizes, and maturity levels. The State of Backstage 2025 Report brings together those insights and paints a clear picture of where Backstage excels and where teams continue to struggle.

A few themes stood out strongly.

Hosting choices are shaping outcomes more than ever. Teams running Backstage themselves describe a very different day to day reality compared to those using managed platforms, especially around stability and upgrades.

Upgrades remain a defining challenge. Even experienced teams report friction in keeping pace with Backstage’s release cadence, particularly when plugins and customizations accumulate.

Automation is quickly becoming the heart of successful Backstage implementations. As teams lean into templates and workflow automation, the platform shifts from a visibility layer to a genuine control plane.

And maturity isn’t simply a matter of time. Some organizations advance rapidly with the right structure and investment, while others stall even after years of use. Catalog health, governance, and sustained ownership are emerging as the real differentiators.

These findings reflect a framework entering a new phase. Backstage is stabilizing, expectations are rising, and the questions are becoming more operational and long term. The report explores these patterns in detail and highlights what separates thriving implementations from those that struggle to sustain momentum.

You can read the full State of Backstage 2025 Report here:

State of Backstage 2025 Report

If you want to explore how these trends map to your own platform journey, we’re always happy to talk.

As soon as these reports surfaced we initiated a full dependency review covering all our services, builds and publish pipelines:

• We scanned for any use of known malicious or compromised packages.
• We compared our versions of all npm dependencies against published indicators of compromise.
• We verified none of our services include packages with the propagation or “dead-man’s switch” behaviour described in the reports.
• We maintain continuous monitoring of our package supply chain for new threats.

We can confirm: we are not affected by this incident.

We do not use any of the compromised versions described in public reports; any overlapping package names in our dependency tree are on safe, non-compromised versions.

We will continue to monitor developments, update our scanning and tooling, and share any relevant security guidance. If your team is reviewing internal risks or supply-chain posture and would like supporting detail from us, please reach out.

More Roadie MCP Server(s)

Last month we launched a set of Model Context Protocol servers to provide data to any LLM-powered MCP client that can accept authenticated external servers, like Cursor or VSCode with Copilot.

This month we've extended that offering to include a few more of the bells and whistles you get with Roadie: backend config and the ability to manipulate Catalog entities via the Decorator.

For those not au fait with the current boom in MCP server creation, MCP is a protocol created by Anthropic last year to help standardise the way data is sent to LLMs and how models can access tools. Via an MCP server (or servers) you can give LLM-powered application access to data and tools that the foundational LLMs could never have.

We've added two new ones:

• The Backend Config Server provides specialized MCP tools for managing and querying backend configuration in Roadie.
• The Catalog Decorators Server provides MCP tools for managing catalog entity decorators/fragments in Roadie.

Now you can, from the comfort of your IDE or MCP client, in addition to accessing the Catalog and the Scaffolder, you can:

• List out proxies, lists ones which are correctly configured, and returns the path for any given proxy
• List the available secrets that can be used in proxy configurations
• Query and then append additional information to Catalog entities via a fragment, like new specifications, metadata, etc

All you need to get going is an API token from Roadie and away you go.

Full docs for all released MCP servers can be found here: https://roadie.io/docs/details/roadie-mcp/

API tokens for all

We've added a new role to api-token-creator which, when added to a user, allows them to (wait for it...) create API tokens.

This has previously been a permission attached by default to the Admin role, and for customers not paying for Role-based Access Control they wouldn't have been able to delegate that permission to non-Admins easily.

That API token opens up all the MCP server capabilities we've been working on, so it makes a lot of sense for us to make it as easy as possible to get a token.

En Masse Changes in the UI

To make it easier to add that role to all users (and general make mass changes for various Admin tasks) we've introduced en masse changes.

You can now make mass changes to:

• Add or remove roles to sets of users
• Mass deletion or cleanup of Locations if they're no longer necessary

Simple.

Moving to the AI SDK from Mastra

This is a quasi-behind-the-curtain update from us, but as we move forward with AI feature building we've made the choice to move from Mastra, a way to build agents in TS, to Vercels AI SDK, a more fully-fledged . Mastra wrapped AI SDK v4 and provided some added bells and whistles, but as this space moves so darn fast Vercel caught up (and actually out-paced Mastra) and we found ourselves looking longingly at the features we could unlock by adopting the AI SDK directly.

What this means for those beta testing new AI features from us is:

• Streaming of chats (and generally a 'hey this feels like ChatGPT or Cursor' feel)
• History, UI elements and a much more dynamic chat
• Chat as a pop-up or slideout (coming soon)

OpenSearch hits GA

OpenSearch has hit GA for Roadie so now all customers can benefit from our new search backend.

We'll be making tweaks to result ranking and indexing over the next few weeks and months as we gather more feedback from users, so let us know if your results are not what you'd intuitively expect.

Anything else?

No.

Happy Summer everyone (or at least for those in the Northern hemisphere)! 😎⛱️

But like everything in (platform) engineering, adoption isn’t linear. It’s messy, exploratory, and sometimes wildly inventive. Teams are hacking together tools, wiring up LLMs, and figuring out what “AI” actually means in the day-to-day of a developer experience and platform engineering. It’s the wild west at the moment, and we thought we’d share some of our observations from the front lines.

What Platform Teams Are Trying

We've seen a few consistent experiments in the wild:

• RAG over TechDocs: Letting engineers ask natural language questions over their internal documentation using Retrieval Augmented Generation (RAG). More than one customer has built a version of this themselves using OpenAI and their TechDocs content, wired through the developer portal UI or third-party integration (Slack is especially popular).
• Slack + Catalog Q&A: “Who owns this service?” is still one of the most common questions. Several teams have built Slack bots to pull ownership annotations from the catalog, and some are layering in code insights too - like linking to recent deployments or PRs.
• Agent-like automations: Some teams are building agent workflows and expressing interest in agent-driven automations. For example, the idea of automatically creating Jira tickets for failing scorecards has come up in multiple conversations as a natural evolution from today’s Tech Insights visibility.

What’s Hard Right Now

Despite the momentum, we’re also hearing consistent friction:

• Context engineering: AI outputs are only as good as the context they get. Teams are struggling with what data to pass, how to structure it, and how to keep it fresh across multiple sources. Too much context can overwhelm the model; too little leads to incomplete answers.
• LLM hallucination: Without a structured source of truth, LLMs invent scaffolder actions, make up ownership metadata, or reference outdated or non-existent docs. The quality is unpredictable.
• Tool fragmentation: Documentation lives in TechDocs, Confluence, Notion, Google Docs. Data is siloed. One customer told us: “We want developers to be able to ask a question, and it shouldn’t matter where the answer lives - TechDocs or Confluence or Slack or wherever.”
• Lack of discoverability and reusability: Internal agents are often one-offs. Teams struggle to expose them in a central place. There’s no “catalog of agents” yet - just siloed agents. All the usual caveats here about reusability and discoverability apply.
• Governance and ownership: Once you’ve got three agents running, who owns them? Who updates them when the data model changes? This quickly becomes tech debt if left unstructured.

What We’re Doing About It at Roadie

We’re leaning into this space because we believe AI + platform engineering is a natural fit. Here's how we’re helping:

1. Cataloging Agents and MCP Servers

As AI agents start creeping into more parts of the internal developer experience, answering questions, triggering automations, guiding workflows, a new kind of infrastructure is quietly taking shape. These aren’t services or systems in the traditional sense, but they’re just as operationally important. And yet, for now, most teams are flying blind.

Right now, AI agents live in the shadows. Someone built a Slack bot last quarter. Who owns it? Someone else built RAG for TechDocs. Where’s the codebase? A third team spun up a tool that opens Jira tickets based on scorecard failures. These AI tools are useful, but they’re not documented, discoverable, or governable. Nobody knows who owns them. Nobody knows if they follow security and reliability best practices, or even if they still work.

If it sounds familiar it’s because it’s the same chaos we saw with microservices before developer portals came along. So here’s the idea: what if we treat internal agents the same way we treat services or systems? What if we give them a proper home in the catalog, with metadata, ownership, links to their code, a purpose tag, and maybe even their scopes and capabilities?

We’ve already started floating this model in conversations with customers. The feedback is consistent: “Yes, please. We need to know what exists, who owns it, what it does, and whether or not we can trust it. It’s maybe not a huge problem now that we only have two or three, but it’s going to be an issue soon enough.”

And this isn’t just for visibility. Once agents are modeled in the catalog, a lot of downstream benefits open up:

• Discoverability: Engineers can browse or search for “agents that help with onboarding” or “Slack bots connected to Roadie.”
• Governance: Platform teams can track which agents are connected to which datasets, where PII might flow, or whether an agent is deprecated.
• Ownership and auditing: Just like services, agents need owners. Catalog entries can help enforce that.
• Integration: Once agents are cataloged, they can show up in scorecards, dependency graphs, and documentation—just like any other component.

This is an initial observation, but it fits a broader pattern: as AI agents become part of the developer experience fabric, they deserve the same treatment as other pieces of platform infrastructure, and to be treated like first-class citizens.

Any mature IDP can help by making agents first-class citizens in the developer ecosystem: discoverable, governed, and owned just like services. When agents are in the catalog, they’re easier to monitor, trust, evolve, and reuse. At Roadie, we’re building native support for this because we see it becoming table stakes for running AI at scale.

2. MCP Server: Making Roadie’s Metadata AI-Ready

One of the biggest issues platform teams run into when trying to use LLMs is that the data they need isn’t structured or accessible in a way that AI can reliably work with. You end up with hallucinations, brittle prompts, and tools that kind of work, sometimes, if you’re lucky. And that’s not good enough, especially for workflows where trust and correctness matter.

That’s why we’re building MCP (Model Context Protocol) servers into Roadie. Roadie is already the canonical source of truth for your software, so the idea is simple: make everything inside Roadie (your catalog, your scorecards, your API specs, your scaffolder actions) available to other tools, including AI agents and copilots, through a structured and queryable API surface.

The use cases are compelling:

• Developers working in their IDE ask copilot to “generate a Python client for the Streetlights API,” and having the LLM automatically retrieve the correct OpenAPI spec via MCP.
• Slack agents that can respond to questions like “Who owns this service?” by pulling ownership directly from Roadie’s catalog, without users having to context switch and log into another window.
• Agents that can look up Tech Insights data (like whether a service is failing a scorecard) and take action or report it, without needing human intervention.

An IDP should act as the live, structured source of truth that AI agents and developer tools can trust, in much the same way it already serves human developers. Exposing this data through standard protocols like MCP means your automation and AI layers always work from the latest, correct information. Roadie has already begun implementing this so our customers’ portals can be both the human and machine-facing source of truth.

3. Multi-Source RAG: Unifying Your Docs into One Knowledge Graph

Ask any platform team what slows down developer onboarding, and you’ll hear the same thing: “We have docs, but they’re all over the place.” There’s TechDocs, Confluence, internal wikis, Notion, Google Docs, even Slack threads. It’s no wonder teams are looking at RAG to unify this mess and make information queryable with natural language.

We’re hearing from teams who want their developers to be able to ask questions like “How do I provision a new database?” and get a real answer, without needing to know where that answer lives!

But this only works if the RAG system can reach across all your documentation sources. Most tools today are single-source: they might work over TechDocs, but not Confluence. Or vice versa. That’s why we’re looking at multi-source RAG support as part of Roadie’s broader AI strategy.

The intention is to build a unified knowledge graph that spans TechDocs, Confluence, and other internal documentation sources. That way, developers can get answers in any channkel (Slack, their IDE, within Roadie) regardless of where the answer is, and platform teams don’t have to duplicate effort or field the same Slack questions over and over.

We’ve heard teams ask for this explicitly: “We want our engineers to ask a question once and get the right answer no matter which system it’s in.”

One of the most valuable things an IDP can do is act as the unifying layer for knowledge, irrespective of whether it lives in TechDocs, Confluence, or anywhere else. Multi-source RAG turns the IDP into a single place where developers (and their AI copilots) can find answers without hunting. Roadie’s goal is to be that connective tissue for our customers.

4. Making Scaffolding AI-Ready

Roadie’s Scaffolder is basically a UI form that does stuff - an easy way for developers to fill in parameters and create components or request infrastructure changes. But with the rise of AI-assisted development, platform teams are thinking about how the Scaffolder might be a programmable interface, something that can be invoked by agents or copilots directly from an IDE or chat environment. This reframing turns the Scaffolder into a backend surface, not just a frontend tool.

We’ve heard from teams who want to use LLMs to help author templates from scratch, generate the right parameters, and even guide developers through the scaffolding process conversationally. The challenge they face is that LLMs often hallucinate or suggest unsupported actions - because they don’t have access to the source of truth.

This is where Roadie can help. Since Roadie maintains the canonical configuration of Scaffolder actions per tenant, we can expose that metadata (through an MCP server) in a structured, machine-readable format. That allows developers to point their LLMs at a live, up-to-date catalog of what’s actually supported in their environment.

The result? AI that’s context-aware and grounded in real platform capabilities. Instead of manually digging through docs or trial-and-error authoring, developers get accurate suggestions, faster feedback, and the ability to trigger safer automation from within their IDEs.

Teams are imagining workflows like:

• “Create a new microservice” prompts in Slack that create a template based on the available actions and automatically open a PR.
• IDE-based copilots that suggest valid actions and parameters based on intent as you write a new template.
• AI agents that help debug broken scaffolder flows by understanding action compatibility.

Any IDP can add value by making its automation surfaces - whether that’s Scaffolder, Ansible playbooks, or other tools - accessible to both humans and AI in a safe, structured way. That means exposing live metadata, enforcing supported actions, and reducing the gap between what teams want to automate and what they can safely automate. At Roadie, we’re starting with Scaffolder but see this as a broader pattern.

What Comes Next?

If your team is experimenting with AI, building internal agents, or even just asking “Where do we start?”, we’d love to chat. You’re not alone; the best practices are being invented right now, and we’d love to explore that together.

Roadie MCP Server(s) for your LLM-powered Applications

We're launching a set of Model Context Protocol servers that can provide data to any LLM-powered MCP client that can accept authenticated external servers, like Cursor or VSCode with Copilot.

For those not au fait with the current boom in MCP server creation, MCP is a protocol created by Anthropic last year to help standardise the way data is sent to LLMs and how models can access tools. Via an MCP server (or servers) you can give LLM-powered application access to data and tools that the foundational LLMs could never have.

It's a client-server architecture, so you can then connect those MCP servers to whatever MCP client you wish. MCP clients range from chat interfaces like the Claude App to full blown AI IDEs like Cursor.

Now you can, from the comfort of your IDE or MCP client, do all kinds of things:

• Access Catalog entity data, relationships, and documentation
• Find, validate, and execute scaffolder templates
• Discover and retrieve API documentation and specifications to build API clients
• Access Tech Insights fact data (TI subscription needed, of course)

All you need to get going is an API token from Roadie and away you go.

Full docs for all released MCP servers can be found here: https://roadie.io/docs/details/roadie-mcp/

AI-enabled search [in beta]

In 2024 we open-source the RAG AI plugin to make AI features in Backstage as widely available as possible.

Implementing those features in Roadie though has taken some time. We wanted to generally improve search before bringing AI into the mix. With OpenSearch powered search results now fully rolled out, we turned out attention to integrating the RAG AI plugin fully into our search.

You can:

• Ask natural language questions of your Tech Docs and Catalog, like Summarise the Streelight service API docs or How does the Streelight service interact with other services?
• Focus on a specific Doc or Catalog entity to narrow results and ask detail questions only about that specific entity.

General availability expected in September / October.

Vibe code your own UI inside Roadie with MDX [in beta]

As part of a Support task, we accidentally allowed anyone to vibe code their own UI inside Roadie.

That might sound silly, but hear me out...

To cover a simple use-case of 'I just want to call and API and render some data back' we exposed an ApiViewerCard to call an API from the frontend (as long as you have a proxy configured for that call).

Then we added MDX configurability to that card and it morphed into the MdxPluginCard.

Then we exposed some common Roadie components to make it easier to compose a card that looked and felt at home in the rest of the UI.

When you put it all together (as some early customers have now started to do) you get some fairly nice cards that do exactly what you need them to.

Not only that, given the widespread use of MDX, you can also get an LLM to vibe code them for you.

For simple use-cases where you don't want to build a full custom plugin, but you do want some ability to pull in data that is not currently supported by a card or plugin: this is your guy.

The next step in this direction? The MDX can be fully generated from prompts inside Roadie, so you don't even have to leave your instance.

Then fully generated entity page layouts...

The future is now...

Tasks now exposed via the Scheduler

To take a small step back from AI-AI-AI-AI, ALL THE TIME, EVERYWHERE, we also introduced a new Scheduler page to the Admin interface. This allows you to see the active jobs and self-serve re-triggering them if necessary.

We're continuing to expose Admin controls like this as a necessary part of an AI work. As AI starts to take over Roadie, we want Platform teams to have more control and visibility, not less.

Improved Launch Darkly plugin

The Launch Darkly plugin we span up last year was looking a little lacklustre and so we've spruced it up.

For those that use Launch Darkly for feature flag management:

• There's now a card displaying environment statuses for each feature flag
• Updated visual indicators for feature flag status (pills rather than text)
• A general spruce of the card UI to make it easier to understand what's happening at a glance

More info can be found in the docs: https://roadie.io/docs/integrations/launchdarkly/

Anything else?

No.

That was quite a lot for one month :)

Introducing the Roadie MCP Server

Roadie has always served a wide range of users, but first and foremost we're a software catalog and automation portal for software engineers.

As such, Roadie (and Backstage, upon who's shoulders we stand) has always been a good dashboard, acting as a single pane of glass for all kinds of fragmented dev tools and scattered pieces of documentation.

But it's always been a tool that you had to step out of your development workflow in order to access.

The classic flow for most engineers using Roadie was to either have Roadie open as a broswer window as they worked, or to step out of the IDE to ask questions of the Catalog, get wider context on your work, or provision some new service using the Scaffolder.

That context switch has always been a problem we'd like to solve.

Luckily the answer is here: it's the Roadie Model Context Protocol (MCP) Server(s).

What's an MCP Server?

MCP is a protocol introduced by Anthropic in November 2024.

The protocol itself is based on a simple client-server architecture. MCP Clients - like the new wave of IDEs (Cursor, Windsurf, VSCode with Copilot, etc) - act as clients that can call MCP servers to get additional information.

MCP servers allow the LLMs at the heart of newer software development applications to receive additional, specific context about the query that a user is making which wouldn't be well represented in the foundational model. MCP servers are therefore great for enriching responses when a base LLM simply wouldn't be able to generate a good enough answer.

For example, if you ask ChatGPT or Claude to generate you a Backstage Scaffolder Template it will do a good-ish job because it was trained on enough open-source templates to do that. The format will be largely correct, for example. But, it will also hallucinate actions because it thinks they might logically exist. It also won't know which scaffolder actions you have installed. This isn't great if you want to actually run that template or automate a development flow involving the scaffolder.

If you tried the same case again but this time with an MCP which returned useable scaffolder actions upon request, then the LLM has a much better chance of generating a functional Template.

How can I use it?

The Roadie MCP server works with any tools that can act as an MCP client, including Claude Desktop, VSCode Copilot, and Windsurf Editor.

If you have your own MCP client with the capability to interact with remote MCP servers with authentication headers then this should also work.

You'll need a Roadie API key. Admins have access by default. There's a role in Roadie which, when enabled, can give users the ability to create their own API key.

Docs on how to get started can be found here: https://roadie.io/docs/details/roadie-mcp/

What can the Roadie MCP servers do?

With the four MCP servers we've currently exposed, you can:

• Access catalog entity data, relationships, and documentation
• Discover and retrieve API documentation and specifications
• Find, validate, and execute Backstage scaffolder templates
• Access operational metrics, security data, and compliance information from Tech Insights (if you're a Tech Insights customer)

Example: OpenAPI specs.

By asking questions in your MCP client, you can integrate two systems, assuming the one you're integrating with exposes OpenAPI specs documented in Roadie.

You would say something like this to you LLM:

“I need to integrate with our User Management system”

The MCP client will then make a call to our MCP server. The steps will look something like this:

• Searches for user-related APIs using find-api-specs
• Retrieves specifications for relevant APIs
• Explains available endpoints, authentication, and schemas
• Provides integration guidance and code examples

You'll then get an implementation of an integration between your current code base and the User Management system, that takes into account how the User Management API actually works.

Example: Creating a new project using the scaffolder

You would say something like this to you LLM:

“Create a new React frontend application”

The MCP client will then make a call to our MCP server. The steps will look something like this:

• Searches for React templates using find-scaffolder-templates
• Shows available templates and their requirements
• Guides user through providing necessary parameters
• Validates inputs using validate-template-values
• Executes template using run-scaffolder-template
• Monitors progress and reports results

You'll then get an complete application based on the provided template.

How does authentication & permissions work?

Getting permissions right in the context of MCP Servers, and AI and agentic systems more broadly, can be tricky.

To solve that for the Roadie MCP Server(s), we've architected the Roadie MCP to have access to the authentication and permissions that the same user would have in the web app. We do that by combining two things:

• Roadie RBAC plugin
• Roadie API using user-scoped tokens

A user generates an API token in the web app for use in their MCP Client of choice. That token inherits the permission that the user has. It's evaluated on each call to ensure the permissions stay in sync. Tokens expire after 1 year.

When will it be generally available?

Folks who have opted in to our early-access AI features will have access shortly. After that we'll begin a wider rollout. Existing customers will be sent a message in the communication tool of choice (Slack or Teams) to onboard them.

OpenSearch for all

The OpenSearch rollout began last month and has not reached general availability.

If you missed it last month that means:

• Casing is better handled: camelCase, kebab-case and snake_case are factored in, so the user doesn’t need to know which was used when they’re searching
• Partial search terms and descriptions are better handled: for example, skipping some-word in camel-some-word-case and just typing camel case will be returned as a result.
• Word fragments allow for partial word matches. We’re currently using 4-7 chars for the ngrams, so fragments of words should be well matched

But that's not the only thing happening with search...

Unified Search enters [beta]

We've also tweaked the UI to create a new search UI that captures more information and provides a generally slicker interaction with search.

The new UI allows you to:

• Use keyboard shortcuts (Cmd-K from anywhere in the app to bring it up)
• Navigate as well as search (try searching for “Catalog”) for example.
• Do things a little faster than before (performance is improved)

Unified search not only enables greater discoverability within Roadie, but is also our first step in enabling RAG AI-enhanced searching of the Catalog and Tech Docs. In fact, it'll be the entry point for a few different AI features we're dreaming up.

We're already seeing search volume and click-through to an entity that then doesn't result in a new search (i.e. "Yay, I found what I was looking for") go up as result.

More on that over the next couple of months as our AI journey continues...

Supporting both GitHub.com & GitHub Enterprise Server

As our customers grow, so do the number of different SCMs deployed in any given organisation.

For a while we've dealt with the GitHub Cloud <> BitBucket combo.

Now we can handle the GitHub.com <> GHES combo too. Aside from setting up both SCMs there's nothing additional you'll need to do - it's all handled by a new GitHub switcher.

Nice.

Roadie Local [beta] begins

Roadie Local, the locally runnable version of Roadie you can deploy on your infrastructure, has entered beta.

We weren't expecting a long beta but who knows when you get into the weird and wacky world of on-prem deployment. More to come here I'm sure...

Rootly Plugin

The Rootly plugin recently released v1.0.0 of their plugin, which we've dutifully upgraded. It comes with a lot of new fun stuff, including the ability to:

• View and search a list of entities
• View and search a list of services
• View and search a list of functionalities
• View and search a list of teams
• View and search a list of incidents

Rootly <> Roadie customers: enjoy. https://roadie.io/docs/integrations/rootly/

Terraform Plugin

We also introduced the Terraform plugins to help Terraform Cloud users manage their infrastructure.

Docs on it can be found here: https://roadie.io/docs/integrations/terraform/

Upgrade to v1.39

Last but not least, Roadie is now on Backstage version 1.39.

Full change notes for that update can be found here: https://backstage.io/docs/releases/v1.39.0/

AI is coming...

We've been playing around with AI features in Roadie for around 18 months now, ever since we open-sourced the RAG AI plugin way back in 2024.

We've struggled to understand how though.

A breakthrough came this month as we got stuck into the developments that have taken place in late 2024 and early 2025, notably the improvements in foundational models and in the introduction of the Model Context Protocol (MCP) paradigm.

We'll be pulling at a few different threads over the next few weeks and months to see what's feasible in the short-term, so watch this space for things like:

• RAG-AI-powered search
• MCP servers to return data and actions from your Roadie instance
• Integration of common (but complex) actions into the UI, powered by AI

Plugins, plugins, plugins

In the meantime, we've introduced two substantial plugin improvements into Roadie that radically improve ingestion of entities.

Kratix

The Kratix Platform has grown in popularity recently as more folks a) adopt the idea of an internal platform and b) want an open-source alternative to the proprietary platforms that exist out on the market.

That's something that we want to support and foster. We're big believers in open-source, as you'd imagine given we're built on top of an open-source product and regularly contribute to various oss projects.

Kratix plugins are now available in Roadie. You'll need to be a paid Kratix user to access them, as is the model for the folks behind Kratix (Syntasso, in case you were wondering) but we now suport:

• Kratix FE plugin
• Kratix BE plugin
• Kratix scaffolder actions

Enjoy. https://roadie.io/docs/integrations/kratix/

Terasky K8s Ingestor and Crossplane plugins

We also folded many of Terasky's Kubernetes plugins into Roadie.

Terasky have managed to thread the needle with improving the way Backstage interacts with Kubernetes and done so in order to ingest entities directly from K8s clusters.

They've also open-sourced Crossplane plugins to allow Backstage to:

• Ingest Kubernetes Components
• Ingest XRDs
• Ingest Claims

Check out our docs here.

Introducing OpenSearch for better discoverability

As the volume of data in the Catalog grows, we often see customers hit a discoverability problem: how do you find items when the Catatlog is tens of thousands of, often very similar seeming, entities?

Well: search. Duh.

But, how do you make sure search results are fast, accurate and scalable? Backstage supports different search engines, but the better ones (Elastic-like ones like OpenSearch) tend to get expensive at scale, especially when you run almost isolated stacks for each customer as we do.

This month we dedicated a bit of time to figuring out how to do it at scale, and voilà! Better search has arrived!

We stripped out the old search engine and brought in OpenSearch, an Elastic-search-like search engine (an open-source one, of course).

That means you get a few improvements out of the box:

• Casing is better handled: camelCase, kebab-case and snake_case are factored in, so the user doesn't need to know which was used when they're searching
• Partial search terms and descriptions are better handled: for example, skipping some-word in camel-some-word-case and just typing camel case will be returned as a result.
• Word fragments allow for partial word matches: We're currently using 4-7 chars for the ngrams, so fragments of words should be well matched

We also have brought ourselves a lot more headroom here for optimisation - we'd reached the end of the line with the pg_search after many years of trying to force it to be the search engine that we'd have liked it to be. OpenSearch has a lot more we can tweak.

So what's next? We'll be rolling out OpenSearch progressively to customers, starting this month.

Mobile UX gets some love

Being able to search for something in the Catalog is one thing, but if you can't readily view it or the experience is poor then that's not much use.

Roadie is largely optimised for web (big tables of software == a laptop or desktop-based experience is always going to be best) but there's a baseline for usability that should be met to make some key use-cases function.

This month we focused on the use of tech docs on mobile. If it's 4am, you've just been woken up by an incident, and you need to quickly read a runbook to see what to do next: probably best to not have to get your laptop out when you have your phone right there.

Tech Insights historical data ingestion gets 12x more data-ness

Tech Insights push-based Data Sources allow for historical data ingestion for our customers when migrating from other systems to Roadie. That's been in place for a while, but this month we bumped the volume of data you can ingest.

Over the last few months we've seen more and more customers want to ingest a lot of data. Part of that is the growing maturity of scorecards as a concept within the wider industry, but also part of the wider shift towrads managed Backstage as the IDP of choice for many orgs.

Down to brass tacts though: the push-based data source can now ingest one year of data. Previously it was 1 month.

DX Plugin

Last but not least, we've introduced the DX plugin to Roadie to help sync the Catalog between the two.

More details here for those that are interested: https://roadie.io/docs/integrations/dx/

This is something we're expecting to see a lot of in the coming months: dev tools that need a software Catalog simply sync to a Catalog like Roadie/Backstage rather than build it themselves. Tools like DataDog and Sentry are dipping their toes into the software Catalog world but it's a tricky business keeping up to date with all the different sources of truth and ingestion mechanisms: that's why open-source tools like Backstage will always have the edge.

More on this trend on the New Stack for those that are interested.

A version of this article appeared on the New Stack: https://thenewstack.io/the-hidden-costs-of-multiple-service-catalogs-in-development/

Developer tooling often requires service catalogs to scope the data that is created within them, especially when that tooling relates to every piece of software created by your organization. Because of this, you often end up with multiple different service catalogs for services you’re brought.

Commonly, you’ll see a service catalog at the heart of classes of tools like:

• Data and observability, like DataDog and New Relic
• Incident management, like Incident.io, PagerDuty, and FireHydrant
• Developer Experience, like DX

Manually creating a catalog in these applications takes valuable time away from Platform and Software teams.

It’s also not the core of any of these services, so the quality of the catalog ingestion, visualization and manipulation is rarely optimal. Some form of a catalog describing the types of software built by an organization is required to make them work effectively for your organization, but that doesn’t mean that that piece of software is focused on creating a fantastic ingestion mechanism for catalog creation.

You can easily sleepwalk into having multiple service catalogs in multiple places with multiple scope levels. This is inefficient and quickly these catalogs fall out of sync.

It’s a pain.

Why you should use an IDP to build one catalog

To avoid multiple catalogs you need a single source of truth. One catalog to rule them all. There’s a strong case for an Internal Developer Portal (IDP) to be that catalog. IDPs like Backstage, Port, and Cortex are all, at their core, software catalogs. They have some other important features (scorecards, some automation runne,r etc) but the bread and butter is making a service catalog easy to create, configure, and use.

Information from various systems is surfaced to development teams to create a single pane of glass. In building an IDP, organizations inherently create an integration and enrichment point for data about their software, which in turn can be used as part of a wider and more complex data flow.

Think in terms of data flows:

• Metadata about software goes in, either auto-ingested or manually added.
• Rich objects are created for each piece of software.
• Structural information is defined and included as part of the data model for that catalog, allowing a graph of software to be constructed of pieces of software and how each one relates to other pieces.

That catalog is then a rich store of information about the software you have built. It’s just a single step away from being the source of truth for that information to other services that require it.

Enter: Roadie

Backstage comes with a few built-in advantages as an IDP that help it excel at this use-case. As the dominant IDP on the market, it garners a lot of support from the third-party service providers you then need to connect to, as well as providers of catalog information (like AWS, who are particularly active plugin developers):

Plugin ecosystem. Third parties are constantly building new options for supporting this use-case. These plugins support either the visualization of information in the catalog or often, more crucially, the ingestion or extraction of catalog data from Backstage.

Auto-ingestion. Backstage has For example, AWS recently released a plugin that supports auto-ingestion of resources like S3 buckets and RDS instances that make completing your software catalog much easier than using another service.

Ease of editing. Backstage comes with a slew of simple enrichment options, leaning heavily on democratically edited yaml files in a format that

Extraction of data. The Backstage Catalog API and plugin ecosystem make it easy to get data out of Backstage when you’re ready to connect to a third-party system.

How to use the Roadie Catalog as a source of truth

Let’s take a look at some examples of how this can be done with examples from incident management, data visualisation and developer experience:

• DataDog
• Incident.io
• DX

DataDog

Using catalag-info.yaml files

The core of the Backstage software catalog is a series of yaml files stored alongside code in your source code management (SCM) tool of choice (Backstage supports them all). These are often simply referred to as catalog-info.yaml files. They’re basically just service metadata and reference keys to other services. DataDog maintains it’s own ingestion mechanism that uses these catalog-info.yaml files to ingest Catalog information. The integration constantly scans repositories in your SCM for Backstage YAML files named service.datadog.yaml and catalog-info.yaml — which you create when you add your service to the Backstage Software Catalog. The code snippet below shows an example of catalog-info.yaml.

You’ll need to enable the GitHub integration for this example

Using DataDogs API

You can also POST Backstage YAML files to the Datadog API. This allows you to programmatically send Backstage service definitions that may not exist in your GitHub repositories. The Backstage Catalog API can respond with your whole Catalog (or just a subset of it), so syncing the two is possible using this route.

https://www.datadoghq.com/blog/service-catalog-backstage-yaml/

Incident.io

Incident.io maintains a variety of different ways to connect their internal software catalog to sources of truth.

Using catalag-info.yaml files

Incident.io works in a similar way via their catalog-importer . The catalog-importer is a little more involved though, so it’s worth taking a look at. The importer can pull data from a variety of sources, “catering for all the ways people normally store their catalog data” as they so delightfully put it. One option is GitHub. This works in much the same way as the DataDog ingestion mechanism outlined above.

Using the Catalog API

Another option is to read Catalog information directly from Backstage itself, via the Backstage Catalog API. This in essence makes a GET /entities call to your Catalog and retrieves information directly. You can filter that as you see fit to make sure you’re only extracting the subset of information that’s relevant for Incident.io.

DX

DX takes a different approach. They’ve built a full Backstage backend plugin to handle the extraction of data from Backstage.

Using a Backstage backend plugin

The DX Backstage backend plugin sets up jobs within Backstage to sync the DX catalog Those jobs make a call to the DX API in order to send Catalog information. As this can be a lot of data (at Roadie we routinely see Catalogs with 300k entities), you probably want to use the optional params for filtering. You can set these in your app-config.

app-config.yaml

dx:
  catalogSyncAllowedKinds: [API, Component, User, Group]
You may also want to control the schedule of the sync, so as not to spam your Catalog. Again, just a bit of config in app-config.

app-config.yaml
dx:
  schedule:

frequency: minutes: 45

Using the Roadie API

At Roadie we run a managed SaaS version of Backstage for many different customers. We often are asked how to make it as easy as possible to use Backstage as a source of truth for other systems. We spend a lot of time making it as simple as possible to take catalog data out of Backstage and use it meaningfully in other applications and workflows.

To help, we expose several endpoints to allow easy syncing with different systems (either to ingest new information or pull catalog information out):

• Catalog endpoints like /entities endpoint allows you to query the Catalog API and programmatically access your software catalog in its entirety. And a /fragment endpoint allows you to sync different third party systems with the Catalog (i.e. ingest Slack handles for your users) and fluidly update your data s you see fit
• A set of endpoints for scorecards and software standards
• A set of endpoints to expose Scaffolder template information

Performance Improvements: Catalog Pagination & Service Workers

We mentioned in the previous Changelog that we've been working on a buffet of different improvements to both performance and stability to enable us to handle large Catalogs.

Top of mind this month has been performance of the backend and response time when users make simple requests of the Catalog. That means: pagination time.

The first pass of pagination has proven we can dramatically reduce page load times without impacting the current UI or UX. For the first customers to receive a paginated Catalog we're already seeing a minimum of ~300ms decreases in load time for smaller Catalogs but at least 66% for XL Catalogs (i.e. 100s of thousands of entities).

This is comparing the P95 scores for page load that we see across all our customers. Red is the rollout where we migrated a few customers over but some customers remained on the older, unpaginated Catalog. Green is the period where the paginated Catalog was fully rolled out.

We've also tweaked the filtering UI to handle more filters, while disabling some (Some of the in-table filtering and sorting options are not possible with a paginated catalog).

And last but not least we’ve also changed the UI a little around filtering to clean up how crowded it can become if you have a table with lots of potential filters in place.

We're also separately using service workers to pre-load some elements of the application to further speed up page load. More on that to come.

The rollout of the paginated Catalog has already begun, so more on this soon.

Fun stuff!

All hail the All tab

On a similar theme, the Catalog has also received a major update in the form of an All tab.

About a year ago we started the effort to make the Roadie Catalog as composable as possible.

The Backstage Catalog is extremely extensible and configurable, but sometimes the UI belies that flexibility and makes it feel a little restrictive.

We wanted to highlight the power and extensibility of Backstage in the Roadie-fied version, but also make it significantly easier to make changes to the Catalog.

That's why we have Catalog Tabs (for making easy filters of the Catalog) and the Decorator (to make changes to entities via the UI or the API, without having to use YAML).

One issue with this novel approach is it created a long-tail of issues with plugins upstream. OSS plugins sometimes assume things like a Domain filter can be applied over the Catalog and a filtered list of Domains will be presented to the user. We don't really have that concept by default (and we allow users to remove it) so we've hit a few bugs over the last few months.

The All tab solves that.

The All tab shows you your whole, unfiltered Catalog. You can apply filters over the top of it, as you like. Previous cards that linked through to filtered lists of Domains, Systems etc, will now land here.

We like it a lot - we hope you like it too.

Enjoy!

Admin Area glam-up [in beta]

The Admin area has also seem some significant changes this month.

The old Admin area had grown into a bit of a sprawl in recent times. 70+ plugins and integrations all mixed together, doesn't lend itself to a clean and easy to navigate UX.

We decided a revamp was in order. We focused on the discoverability of plugins - namely moving to a card-based layout and refreshing the search. We also took the opportunity to make changes to the information architecture of this area to chunk up the config options into different tabs.

Next up we'll be standardising each page to make adding secrets, seeing which cards and tabs that plugin exposes are being used, and understanding at a glance how it is configured. After that we'll be looking at how simple health checks and status indicators for each plugin can give greater feedback to users about the health of that connection.

Enjoy.

But that same flexibility comes with a cost. Backstage doesn’t come pre-configured for your organization’s needs - it’s a toolkit; more a framework for building an internal developer platform than a platform in and of itself. Moreover, it requires customization and ongoing maintenance, and crucially, it demands TypeScript expertise - a skillset that many platform teams don’t typically have in-house. Unlike infrastructure-as-code or cloud automation tools (which use more familiar Python or Go), Backstage development requires working directly with a React and TypeScript codebase. This mismatch in skillsets is one of the biggest hurdles teams face when trying to operationalize Backstage, and it’s a key reason why many underestimate the effort required - especially beyond the initial deployment.

It’s helpful to think of the Backstage journey following a lifecycle: the planning and setup phase (Day 0), the deployment and integration work (Day 1), and the ongoing operations, scaling, and optimization (Day 2 and beyond). Each of these phases comes with its own challenges, and if you’re not thinking ahead, it’s easy to get stuck and see momentum stall.

Day 0: Planning and Strategy - Laying the Foundation for Backstage

Before diving headfirst into writing YAML, it makes sense to ensure Backstage is set up to solve real problems for your engineering teams. Many organizations install Backstage expecting quick wins, only to find that a half-filled service catalog and a couple of integrations deliver little immediate value. A successful implementation starts with understanding what engineers actually need from Backstage - whether it’s faster onboarding, better service discoverability, or standardization across teams. Without this clarity, adoption stalls before it even begins.

Key considerations for Day 0

Defining success. What problem are you solving? Service observability? Developer self-service? Compliance automation? What’s the burning problem that you’re hoping an IDP will solve? Speak to your broader engineering organization and find out if the major source of frustration is a lack of centralized information around services, painful and lengthy environment creation times, or a lack of engineering standards. Knowing what problem you’re solving helps prioritize your Backstage implementation and rollout.

Building organization alignment. One of the most underestimated challenges isn’t technical at all - it’s cultural. Backstage success relies on organizational buy-in, and many teams struggle with adoption of the developer portal concept itself. Engineers may be skeptical about whether Backstage (or any portal) will truly save them time or improve their workflows. A strong internal advocacy plan is just as important as a technical implementation plan. As such, it’s important to consider who owns the portal? What teams need to be involved? How will you drive adoption? Successful Backstage implementations have a clear owner who can lead this - whether that’s the platform team, developer experience group, or an infrastructure lead.

Understanding complexity. Are you dealing with multiple repositories? Do you have existing metadata in spreadsheets? What integrations do you need from day one? Backstage thrives in complex engineering environments, but that complexity also makes implementation harder. If your services are spread across multiple repos (because you’re in the process of changing SCMs, for example), you’ll need to define a strategy for catalog completeness. Similarly, if engineering teams have historically tracked metadata in spreadsheets or Confluence, plan for how that information will migrate into Backstage.

Choosing a hosting model. Do you have the bandwidth to stand up and then maintain a self-hosted Backstage instance, or would a managed solution free up your team? Self-hosting Backstage means setting up your own infrastructure, handling upgrades, and troubleshooting outages - responsibilities that many platform teams don’t want to take on. A managed solution like Roadie allows you to focus on adoption and feature development instead of infrastructure headaches.

How Roadie helps on Day 0

• Expert-led onboarding. We guide teams through best practices and pitfalls based on real-world experience with different customers across multiple verticals.
• Pre-built integrations. Get out-of-the-box support for GitHub, Jira, PagerDuty, and more without spending months setting up plugins.
• No infrastructure headaches. Roadie fully manages Backstage, so you don’t need to worry about performance, upgrades, security patches, or downtime.

Many teams get stuck in Day 0, unsure of where to begin. With Roadie, you don’t have to navigate this alone. Our Solutions team has helped hundreds of organizations set up Backstage Proof of Concepts, and helps ensure a good fit for Backstage and your organization from the very beginning.

Day 1: Deployment and Integration - Getting Backstage Live

Once the strategy is in place, it’s time to make Backstage a reality. This phase involves setting up the initial Backstage implementation, including the software catalog, integrating with existing developer tools, and ensuring that authentication, access control, and self-service workflows are functional.

Key considerations for Day 1

Service catalog population. Where will Backstage source its metadata? Do you need GitHub sync, YAML definitions, or both? A common mistake is launching Backstage with an incomplete catalog, which can hamper efforts at driving adoption. If engineers search for a service and can’t find it, they won’t come back. You need a strategy for automatic catalog population so that new services appear without manual effort. Putting Backstage into the production path is also a good way to ensure long-term catalog completeness - if developers are required to register their new services in Backstage, it builds a positive feedback loop to drive catalog completion.

Tooling integrations. Which services (CI/CD, observability, cloud platforms) should be connected? Backstage is most powerful when it integrates with the tools developers use daily. Are you pulling in logs from Datadog? Linking to on-call rotations in PagerDuty? Surfacing deployment insights from ArgoCD? The sooner Backstage provides visibility into a team’s actual workflow, the sooner it becomes indispensable. One of Backstage’s most compelling propositions is of the central pane of glass, reducing the need for developers to switch windows. This is only possible if you’ve integrated all the necessary tools into your Backstage environment.

Authentication and RBAC. How do you ensure secure, role-based access control? Not every engineer needs full edit access to Backstage and all of its various moving parts. Define your authentication strategy early - will you use GitHub SSO? Okta? A custom identity provider? Role-based access control (RBAC) should align with your engineering org’s structure so that only relevant teams can modify services and templates while ensuring visibility remains open. Fine grain control allows you to go deeper, restricting access to sensitive entities or applying sensible limits on who can (for instance) create new scaffolder actions.

Scaffolder workflows. Speaking of which, what self-service templates will accelerate developer productivity? The scaffolder is often underutilized because teams don’t set up useful templates from the start. Think about what engineers repeatedly request from platform teams (or check your ticket volumes) - service creation, infrastructure provisioning, spinning up a new API - and build those workflows into Backstage’s scaffolder. The goal is to reduce developer toil by automating common tasks**.**

How Roadie helps on Day 1

• Instant deployment. A fully managed Backstage instance, set up in minutes instead of months.
• Automated catalog population. GitHub sync, YAML registration and the Roadie catalog API streamline the onboarding process.
• Pre-configured RBAC. Secure access controls without needing to build policies from scratch. Fine-grained control gets you the granularity you need on top of sensible defaults.
• Scaffolder templates and best practice. We can help get you going with advice and guidance on workflows for service creation, infrastructure provisioning, and more.

By handling the operational complexity, Roadie ensures that Day 1 is about getting value from Backstage, not wrestling with configuration files.

Day 2: Operations and Optimization - Keeping Backstage Running Smoothly

Great, you’ve got Backstage live, but now the real work begins! More than anything, for teams approaching Day 2, adoption becomes the focus, but there’s still a significant governance and performance management component. Without a strong Day 2 plan, Backstage can quickly become outdated, underutilized, or a maintenance burden.

Key considerations for Day 2

Monitoring and performance. Is Backstage responsive, scalable, and available when developers need it? Engineers are notoriously performance sensitive, so Backstage should feel snappy, not sluggish. A slow portal can affect adoption - monitoring page load speeds, request latency, database queries, and plugin performance ensures that Backstage remains fast and scalable as more teams onboard, and the size of the catalog grows.

Governance and compliance. Are service owners keeping metadata up to date? Is technical debt being tracked? Backstage is only as useful as the data inside it. If ownership details, API versions, or compliance statuses are outdated, trust erodes. Setting up Tech Insights to track stale metadata and enforce policies helps maintain catalog health.

Driving adoption. Are developers using Backstage as their central hub? How do you encourage engagement? A Backstage rollout is only successful if engineers actually use it. This means marketing it internally, gathering feedback, and ensuring that the portal delivers real value - whether that’s through dashboards, integrations, or self-service features.

Ongoing feature development. What additional capabilities and custom plugins should you roll out over time? Backstage evolves. As teams become comfortable with the service catalog, they may want to expand into TechDocs, scorecards, security insights or even custom plugins specific to their needs. Having a roadmap for feature adoption keeps momentum going.

How Roadie helps on Day 2

• Automated upgrades and maintenance. Always running the latest, most secure version - no manual updates required.
• Tech Insights for governance. We help you identify gaps in ownership, enforce security policies, and track catalog health.
• Performance and scaling support. Roadie continuously optimizes for speed and reliability, even as usage grows.
• Ongoing guidance and best practice. Access to our team of solution engineers and customer success for troubleshooting, strategy, and long-term platform growth and adoption.

Maintaining Backstage isn’t just about keeping it running - it’s about keeping it useful. Many teams struggle with the high maintenance effort and slow feature development velocity, as each upgrade or new integration requires engineering time. With Roadie, we handle upgrades, performance optimization, and feature rollouts for you, ensuring your Backstage instance stays modern without burdening your team.

A Simpler, Faster Path to Backstage Success

Backstage adoption isn’t just about getting to Day 2 - it’s about ensuring long-term success through proper planning, a smooth launch, and continuous improvement.

Without Roadie, teams often find themselves bogged down in maintenance, debugging, and internal support. With Roadie, Backstage is effortless to deploy, easy to scale, and always up-to-date. That’s why to date, we’ve had several customers migrate from self-hosted Backstage to a Roadie-managed solution.

If you’re considering Backstage or struggling with an existing implementation, let’s talk. Book a demo today and see how Roadie accelerates your journey from Day 0 to Day 2 and beyond.

It’s no secret that identifying misconfigurations or vulnerabilities early can save engineering teams huge amounts of time and risk. But doing that efficiently means centralizing actionable security data in the same place developers already work. That’s where this new plugin comes into play.

According to Roadie Software Engineer Irma Solakovic (who built the plugin), it’s all about speed and clarity, and a single pane of glass:

“As an engineer, you want to shorten the time it takes to spot issues - whether that’s a misconfiguration or a vulnerability. Save time and stop hopping between different platforms - if you're already looking at your services in Backstage, why not see what's wrong there, too? Now teams can see Wiz issues alongside their services, documentation, and CI/CD pipelines.”

What Does the Plugin Do?

The plugin surfaces Wiz’s prioritized security findings - such as vulnerabilities, compliance risks and misconfigurations - right inside Backstage, giving teams a unified overview of their software landscape and its security posture. Key highlights include:

• Single Pane of Glass: No need to jump between separate tools. Check your critical Wiz findings the moment you inspect a service in Backstage.
• Contextual Issue Prioritization: Wiz assesses vulnerabilities and misconfigurations in the context of your specific environment, ensuring that the most pressing threats are highlighted first.
• Comprehensive Risk Insight: Access up-to-date, Wiz-scanned data so you can focus on the most pressing issues first.
• Open Source & Managed: The plugin is open source for self-hosted Backstage users, and Roadie customers benefit from automated token renewal and streamlined setup.

See your Wiz issues by status, from inside Backstage

Certified by Wiz

This plugin has been certified by Wiz, so it meets Wiz’s standards for reliability and quality. Organizations using Wiz can be confident that their data is displayed accurately in Backstage - letting them act on the right information at the right time.

According to Irma, the Wiz team provided valuable support during the development process:

“We got great feedback from the Wiz engineers, especially around handling API usage guidelines. Their insights were really helpful in making this plugin both accurate and efficient.”

The feeling goes both ways, and the Wiz team were equally excited by the development of the plugin by Roadie: "We’re thrilled to see Roadie’s Wiz Plugin seamlessly integrate prioritized Wiz security insights into Backstage, empowering developers and security teams to collaborate more effectively, quickly identify critical issues, and streamline remediation directly within their existing workflows."

Why Choose Roadie for Wiz + Backstage?

While anyone can adopt the open-source plugin, Roadie offers a managed solution and we can assist with initial setup and ongoing usage of the plugin. Additionally, Roadie allows for managing Wiz secrets via the UI, while open-source users will need to add these to their configuration file (app-config.yaml). With Roadie, you spend less time fiddling with integrations and plugins, and more time actually fixing issues.

Get Started

Ready to give it a spin? Check out our resources:

• Roadie users:Roadie Wiz Plugin Docs
• Self-hosted users: Install via NPM (@roadiehq/plugin-wiz-backend) and follow our guide.

We’re excited to hear your feedback, and look forward to seeing how the open community continues to improve and enhance this plugin.

Our engineering team tackled a few key problem areas - memory leaks that were leading to crashes, liveness probes that were restarting services unnecessarily, and background jobs that were sometimes slowing things down. Let’s dig into what was happening, how we fixed it, and why it matters.

The Challenge: Unstable Backends, Memory Leaks, and Unnecessary Restarts

Like any complex platform, Backstage and Roadie’s multi-tenant setup requires ongoing maintenance to stay performant. Over time, we started noticing some recurring issues. Some customers were seeing intermittent backend errors. Our monitoring showed that some instances were restarting more than they should, and memory usage in certain environments was creeping up in a way that suggested something wasn’t being cleaned up properly.

Diagnosing the Issues: A Deep Dive Into Backstage Stability

The first step was figuring out exactly what was happening. We leaned heavily on our monitoring tools, using Grafana dashboards, CloudWatch logs, and Prometheus metrics to track memory usage, CPU performance, and garbage collection behavior. We also took a lot of heap dumps—essentially snapshots of what was in memory at a given time—to compare them over time and spot patterns.

Some issues were straightforward and easy to resolve quickly, but others were far more subtle. While some fixes were immediate, the more nuanced memory leaks and stability concerns required deeper investigation. These problems took time to surface and were harder to track down, requiring careful analysis across days or even weeks of data. Debugging them meant running long-term comparisons of heap dumps, identifying small but persistent changes, and testing fixes incrementally to ensure they actually solved the issue without introducing new problems.

Fixing Memory Leaks

One of the biggest culprits turned out to be a global constant that was continually appended to with the same value, causing unbounded growth in memory usage. Instead of replacing old data, the service kept adding to the same constant over time, gradually increasing memory usage until the instance would crash.

const A_CONSTANT = [];

export class CustomProcessor implements CatalogProcessor {
  async preProcessEntity(
    entity: Entity,
    location: LocationSpec,
  ): Promise<Entity> {
    // The following causes a memory leak
    A_CONSTANT.push('blah');
  }
}

Additionally, express request handlers were starting un-awaited promises, which in certain edge cases could remain unresolved, holding onto memory and preventing proper cleanup.

const makeLargeObject = async () => {
  // assign large object and sleep.
}
router.get('/consume-memory', async (_, res) => {
  makeLargeObject(); // unwaited promises can cause memory leaks
  return res.status(200).send();
});

Backstage’s default in-memory cache behavior also contributed to the problem. Expired cache items were only removed when accessed, meaning if something wasn’t retrieved after expiring, it would just sit there, taking up space indefinitely. To fix all of this, we cleaned up how constants were stored, replaced the problematic setTimeout loops with Backstage’s built-in scheduler, made sure all promises were handled correctly, and implemented a scheduled cache cleanup process to prevent memory bloat.

Tuning Kubernetes Liveness Probes

Liveness probes are a great feature in Kubernetes, but they need to be carefully configured to reflect the actual health of a service. Previously, our probes were reporting success too early in the startup cycle, meaning that broken or incomplete processes were being marked as healthy and starting up successfully—even when they weren’t fully ready. To fix this, we separated out readiness and liveness probes, ensuring that Kubernetes only considered a service healthy once it had completed its startup process correctly.

Worker Separation for a More Reliable API

Backstage, by default, runs background jobs in the same process as the API. That means if something in the background is taking a while - say, a Tech Insights check - it could impact API performance. We saw cases where this led to slow response times or, in extreme cases, crashes. The fix here was relatively straightforward in theory: move background jobs into separate worker processes so they no longer affect the API’s stability. Now, even if something in the background is taking longer than expected, it won’t slow down the user-facing parts of the platform.

The Impact: A More Stable and Resilient Roadie

The results of these changes were immediate. One of the most noticeable improvements was the immediate end of Sunday night crashes. Previously, during the week, our continuous updates to customer Backstage instances meant that services were restarted regularly, which helped keep memory leaks from accumulating. But over the weekend, with no updates happening, memory leaks would build up unchecked, leading to crashes by Sunday night or Monday morning with some level of predictability. Now, with these fixes in place, memory usage remains stable throughout the week, and these weekend crashes have disappeared.

Backend restarts are also happening much less frequently, and overall system stability has improved across the board. API performance is more predictable, memory usage is under control, and customers are seeing fewer errors and disruptions.

For platform teams using Roadie, this means a smoother experience with fewer unexpected issues. If you’re running an IDP, you shouldn’t have to worry about whether it’s up and running—it should just work. And that’s what these improvements were all about.

Lessons Learned & What’s Next

One of the biggest takeaways from this work is just how important it is to keep a close eye on certain key metrics. If Kubernetes is restarting your pods more than usual, something’s probably wrong. If Backstage’s processing queue is growing, that’s a sign that background tasks aren’t keeping up. And if global constants aren’t being handled properly, they can lead to slow, creeping performance issues that only become obvious after weeks or months.

We also learned a few best practices along the way. Freezing global constants with Object.freeze helps prevent unintended modifications. Handling all promise rejections properly ensures that errors don’t silently accumulate in memory. And separating background jobs from API processes is a simple but powerful way to improve overall system stability.

Looking ahead, we’re continuing to refine our observability and monitoring, making sure that stability issues can be detected even earlier. We’re also exploring ways to improve how Backstage handles large-scale processing workloads so that performance remains smooth even as usage grows.

Roadie: Making Backstage Better for Platform Teams

The goal at Roadie is to make sure platform teams get the best possible experience with Backstage. That means not just adding new features, but constantly refining and improving what’s already there. These stability fixes are part of that ongoing effort, ensuring that Backstage is as reliable and hassle-free as possible for teams who depend on it.

If you’re using Roadie today, these improvements are already live and working for you. If you’re thinking about adopting Backstage and want to ensure you’re getting a battle-tested, highly available implementation, we’d love to chat.

Over the years the ability to self-host Roadie has come up time and again.

Platform teams for security-conscious, government-facing, or generally pro-self-hosting would often approach us about using Roadie but were blocked. They couldn't access all the Roadie goodies that our SaaS customers know and love, like scorecards, RBAC, and a no-code UI for crafting entity pages and enabling plugins.

Even folks who simply wanted to spin up a quick PoC of Backstage would approach us and ask if we could help. There isn't a default image for running Backstage and it can be tricky to even stand up an instance, let alone craft a version of Backstage that reaches parity with proprietary offerings like Cortex or Port in a PoC timeframe.

All that's changing.

We're launching self-hosted Roadie.

We call it Roadie Local.

• You'll be able to self-host it.
• You'll be able to use it for free (for less than 15 users).
• And it's currently in beta.

Why self-host Roadie?

Running Roadie in a security-conscious system

Maybe you're a government department, defense contractor, high-frequency trading firm, or simply an organization that takes security very, very seriously: you need the software you use to be on-prem or at the very least hosted in your cloud account.

Roadie Local is that. We're even working on an air-gapped solution.

Spinning up Backstage, fast

Early in the cycle of adopting an Internal Developer Portal (IDP) like Roadie or Backstage you want to kick the tires and test things out. In fact, you're usually testing a whole bunch of different IDPs to understand the market and what each one offers. From completely open-source offerings like Backstage, proprietary offerings like Cortex or Port, and hybrid open-source-but-with-support-and-some-nice-features like Roadie.

Spinning up Backstage for these kinds of PoCs usually took a bit of time though, and you don’t necessarily get to feature parity with the proprietary offerings in the time allotted.

With Roadie Local you can stand up a best-in-class IDP, based on open-source Backstage, in minutes.

Migrating from Self-hosted Backstage

Migrating from self-hosting Backstage to Roadie is now a well-trodden path, but the act of switching is still non-trivial. That is often because of the snags from moving to SaaS. Moving from self-hosting Backstage to self-hosting Roadie is a much simpler adoption path.

How it works

We've tried to keep it simple:

• You request a license key
• We give you a license key
• You pull the Roadie Local image and install it
• You have a local version of Roadie

That's abstracted a little 👆, but not much.

What's in the box?

In essence, everything that you know and love now as well as everything we're building for the SaaS version of Roadie. There will be no meaningful difference in terms of feature availability.

That means:

• A composed and polished version of Backstage that you can configure both in the UI and with file-based config options for added flexibility.
• All the usability and quality of life improvements that we've built over the years, like no-code page editing and no-YAML ways to add catalog data.
• 70+ plugins to help you get going on Day 1.
• Scorecards that let you document and enforce engineering standards.
• And built-in RBAC to control who sees what and who can trigger certain actions.

The best part? Roadie Local is free for less than 15 contributing users.

We've never had a fully free tier for Roadie.

We offer 30-day free trials of the SaaS product but the costs of running a full, isolated Backstage stack for each SaaS tenant we have meant that the economics never worked out for us to offer Roadie to smaller teams, en masse, for free.

The economics are different for Roadie Local, so we get to be different.

Roadie Local will be free for <15 contributing users*.

• What's a Contributing User? Someone who has committed code to a piece of software tracked in the Catalog in the last 3 months.

If you're playing around or running a quick Proof of Concept, then it's free.

If you're a smaller team and you have 10 engineers, then it's free.

Simple.

What's next

We don't like to spend long in beta, so expect Roadie Local to reach general availability in Spring 2025.

We're targeting BackstageCon EU in April 2025 for a big, ol' fashioned product launch.

And then?

In short, deployment options:

• Helm charts
• Images and deployment options available on the AWS, Azure, and GCP Marketplaces
• Other, non-Kubernetes, non-cloud provider-based deployment options

💈 Roadie Local [coming soon]

We're going on-prem.

We've been toying with this idea for a long-time and we've decided to go all-in on enabling Roadie to be run on customer infrastructure.

We call it Roadie Local.

More soon - we're aiming for a beta launch in March, moving to general availability in Q2 2025.

💈 State of Backstage launches

We're also launching a survey.

When a community reaches a certain level of maturity getting a good lay of the land each year becomes a good idea. Backstage has hit that point - maybe we're even a little late to realise it - so we thought it was time to launch an annual State of Backstage survey.

This is a non-commercial activity for us. It’s solely to foster a greater sense of community and information sharing.

We’ll publish the report from this survey in Spring 2025 for free (of course).

Check it out on the state of backstage.io site.

🔌 Plugins & Integrations roundup

• Announcements plugin: we pulled in the Announcements plugin to give customers the ability to post updates and requests to their userbase within Roadie. It's a plugin that's been around for some time, but it's developed markedly in the last year or so so we thought it was time. Enjoy!

Backstage has grown, diversified and differentiated a lot since it was open sourced back in 2020.

We have new features, new patterns of adoption, and many, many new members of the Backstage community.

One downside for having a sprawling community of business adopting Backstage is that we often don’t have much insight into how it is actually being used. Organisations tend to keep that information close to their chest.

There are some exceptions and some information is in the public domain to help new adopters and those who are looking to super-charge their instance. For instance, case studies are extremely useful for spreading the word about patterns of adoption. The Backstage Discord channels is great for questions about individual features and plugin adoption. Consultants and SaaS providers like Roadie are good for seeing a wider subset of good patterns.

Yet even commulative, all of these channels mean that information is fragment and the flow of information and knowledge is inconsistent.

We lack a systematic and widespread data capture mechanism to ping the community for its current state.

That’s why we’re launching the State of Backstage survey.

This is a non-commercial activity for us. It’s solely to foster a greater sense of community and information sharing.

We’ll publish the report from this survey in Spring 2025 for free, on the state of backstage.io site.

It aims to provide a general map of the community and an understanding of its current state.

A novice Backstage user, new to the community, maybe even standing up Backstage for the first time, should be able to understand at a glance understand current best practice, plugin adoption, and patterns that are working for organisations successfully adopting backstage.

Check it out (and fill it in) at stateofbackstage.io!

Day 2 with Backstage is about moving beyond the initial setup and figuring out how to turn Backstage into a tool your developers use as part of their daily or weekly workflows, and in time, come to rely on. Many teams start with specific use cases like service discoverability or software governance, but these don’t always translate neatly into widespread adoption. That’s because solving a single pain point doesn’t necessarily create long-term habits. To make Backstage stick, you need a strategic approach that aligns its features with your developers’ daily workflows. So, let’s explore why Backstage adoption often stalls, and how to set your organization up for sustained success.

Build it and they ~~will~~ might come

It’s easy to assume that once Backstage is deployed, adoption will naturally follow. After all, it solves critical pain points: creating a unified service catalog, automating repetitive tasks with templates, and introducing governance tools to improve software quality. But in practice, adoption requires much more than simply solving these problems. It requires building habits, demonstrating value, and integrating Backstage into the fabric of your engineering culture.

Strategy: treat Backstage like a product

The key to overcoming adoption challenges is to treat Backstage like a product, not just a one-off project. This mindset is at the heart of successful adoption and ensures that Backstage evolves over time to meet the needs of its users. Treating it as a product means committing to continuous improvement, guided by user feedback and measurable outcomes. That does mean that if you’re a platform engineer, you may need to put on a product manager hat, thinking strategically about what your users need, prioritizing features, and continuously communicating the value of Backstage to developers and leadership alike. This means balancing the technical work with a clear focus on solving problems and delivering value internally, which may represent a big change.

It needn’t be overwhelming though - start by engaging your most important internal users - your developers. Communicate with them to understand their workflows, pain points, and priorities. Backstage adoption is a collaborative effort, and the more involved your developers feel in shaping its direction, the more likely they are to embrace it. Use tangible metrics to track success, such as catalog completeness, daily active users, or ROI from key features like templates (engineering time saved for each template run, for instance). These metrics not only highlight areas for improvement but also help demonstrate value to leadership, ensuring sustained investment.

The ‘treat it like a product’ approach to a developer portal such as Backstage is really well articulated by Adam Rogal, who leads Developer Productivity and Platform at DoorDash. In the podcast episode “Bootstrapping a Developer Portal,” Rogal shares how his team built their developer portal, DevConsole, with a clear focus on delivering immediate value to their engineering customers. By engaging engineers early and iterating based on their feedback, the platform team ensured that DevConsole directly addressed real pain points. This approach not only drove higher adoption but also built trust with their internal users.

Rogal also underscores the importance of creating a community - not just of users but of contributors. By empowering teams to actively participate in shaping the portal, DoorDash fostered a culture of collaboration and continuous improvement. This communal effort allowed the portal to evolve alongside the organization’s needs, ensuring its long-term relevance and success.

By adopting this product mindset and prioritizing user engagement, platform teams can transform Backstage into an indispensable tool that enhances productivity and aligns with organizational goals. The experience at DoorDash serves as a powerful example of how this approach can drive both adoption and satisfaction.

Tactics for sustaining adoption

While adopting a product mindset gives you the strategic foundation for long-term Backstage success, tactics offer some actionable focus areas on a day-to-day basis. Let’s explore some key tactics to help drive and sustain adoption.

Automate catalog completeness

A rich and accurate catalog is the foundation of Backstage, but manually maintaining it is time-consuming. The best approach is automation. By using integrations with GitHub or AWS, you can automatically populate the catalog with metadata from your repositories or clusters. In our experience, organizations with the highest levels of catalog completeness (90% or more) have all made extensive use of automations such as templates and scripts, and entity autodiscovery and ingestion to reduce the manual effort involved.

Leverage templates for early wins

Templates are one of the easiest ways to showcase the value of Backstage. They can automate repetitive tasks, like setting up a new microservice or creating a CI/CD pipeline, saving developers significant time and ensuring software governance and best practice is baked in. The ROI here is often immediate and measurable - reducing a process from months to minutes is something both developers and leadership can get behind.

Measure and communicate value

Metrics are your best friend when it comes to adoption. Track data like catalog completeness, template usage (and time saved), and daily active users to understand what’s working and what needs improvement. Share these metrics with leadership to demonstrate the impact of Backstage and justify further investment.

Make adoption a cultural effort

Adoption isn’t just a technical challenge - it’s a cultural one. Evangelism plays a crucial role here. Host “lunch and learns,” demos, or office hours to show developers how Backstage can make their lives easier. Engage other engineering teams and create internal champions who can advocate for the platform within those teams. Adoption grows when developers see Backstage as part of their workflow, not an extra step.

Invest in custom plugins

While Backstage’s open-source plugins are a great starting point, one of Backstage’s biggest selling points is its near limitless extensibility through custom plugins. These plugins can address your organization’s unique workflows and challenges, creating a toolset that developers can’t find elsewhere.

Plan for Day 2 from Day 1

The real value of Backstage lies not in its deployment but in its adoption. To unlock this value, you need to approach Backstage as a product - gathering feedback, iterating on features, and aligning its capabilities with your organization’s goals. Adoption doesn’t happen overnight, but with a strategic mindset and sustained effort, Backstage can become an indispensable tool for your engineering teams.

So, as you set up Backstage, don’t just think about the first deployment. Think about what comes next - the Day 2 experience. Plan for it, invest in it, and you’ll set your organization up for long-term success.

💈 Performance, Stability & Pagination

We've been working on a bunch of different improvements to both performance and stability.

First among them is pagination of the Catalog, without losing filtering.

We'll be tweaking the filtering UI in the next few weeks and months but the first pass of pagination has proven we can dramatically reduce page load times without impacting the current UI or UX. We're already seeing ~300ms decreases in load time and increased stability for giant Catalogs (i.e. 100s of thousands of entities).

We're separately using service workers to pre-load some elements of the application to further speed up page load. More on that to come.

Last but not least, we've also been chunking up Backstage into separate pods to help us scale the application as usage grows. More on that to come in blog posts, but if you're running a giant instance or have reached scale with Tech Insights, splitting individual parts of the application out into separate pods is generally a good idea.

Fun stuff!

Tech Insights Facts: Aggregated

Sometimes you want to just see a Fact in Tech Insights. You don't want to look at whether it passed or failed a check or whether it is meeting a Scorecard requirement: sometimes you just want the data.

An entity-by-entity view is one thing (which we implemented with Fact Tables) but often you'll want to see a team-by-team view of fact data.

For checks and scorecards we have roll-ups, and now for Facts we have Fact Cards.

Say you're tracking DORA metrics or want to see mean time to resolution for incidents. Now you can :)

GitHub Enterprise Server (GHES) support

We now support both GHES using the Roadie Agent and GHES combined with github.com (if that floats your boat).

Previous iterations of our GHES support have relied on a connection to the GHES instance over the public internet. This is ok for some organizations but not all, and usually if you're using GHES you're pretty pro-security and pro-don't-send-things-over-the-public-internet.

In the past for these use-cases we used the Roadie Agent, which is based on the open-source Snyk Broker, to create a long-lived websockets connection that is fully controlled by the customer. That hasn't, up until this point, worked with GHES though.

Now it does! :)

For good measure we threw in the ability to configure both GHES and github.com simulatenously to allow folks who are migrating from one to the other to support both SCMs on their Roadie instnace.

Enjoy!

🧘‍♂️ Scaffolder Actions table

We have also just launched some good ol' fashioned documentation.

The Scaffolder is an extremely powerful tool, but it doesn't have the same level of documentation or cataloging that plugins receive. It can be tricky to find out which open-source actions exist and how to use them.

To solve that, we created the Scaffolder Actions Directory. We searched the internet high and low to find all the actions we could. Most of them we support, some are hot of the press and not yet included in Roadie. We'll keep updating it as the Actions ecosystem evolves.

Check it out: https://roadie.io/docs/scaffolder/scaffolder-actions-directory/

🔌 Plugins & Integrations roundup

• Wiz plugin: Our Wiz plugin is now certified! The new Wiz security frontend plugin surfaces Wiz data inside Backstage. We've worked with Wiz to give it ol' seal of approval. It's available both inside Roadie and as an OSS plugin. 👩‍🚀🚀

Instead, a successful IDP emerges when you carefully pinpoint and address the actual challenges developers face. Is discoverability a major and constant headache, with engineers spending hours trying to figure out who owns which service, or whether a certain piece of functionality already exists somewhere else? Are operations teams buried in tickets because developers need help every time they want a new environment or test database spun up? Do your devs waste time hopping between half a dozen interfaces to track deployments, check logs, and find documentation?

These are the signals to tune into when you’re thinking about building an IDP. Aligning your IDP closely with well-understood problems ensures every feature is both purposeful and valued. Equally important, it sets the stage for adoption. Developers and team leads won’t embrace a platform just because it’s there—they’ll embrace it if it truly solves their everyday pain points. By narrowing your focus to the challenges that matter, you can go deep on solutions that genuinely improve how your team works, rather than going wide and hoping something sticks.

Focus on the Three Big Challenges

1. Discoverability

The Challenge:

Have you ever seen teams re-implement a piece of functionality simply because they didn’t know it already existed somewhere else? Or maybe someone spends half a day sifting through old wikis, outdated Confluence pages, and random Slack threads looking for an API endpoint. That’s poor discoverability in action. It’s not just about knowing what services exist, but also understanding who owns them, what their dependencies are, and where the latest documentation or runbooks can be found. When this information is hard to find, it leads to frustration, wasted time, and sometimes unnecessary duplication of effort.

What to Prioritize and Implement:

A Service Catalog that brings together all your services, their owners, docs, performance data, and runbooks in one easily searchable location. Using something like Backstage, you create a living directory of what your organization offers internally, so developers spend less time hunting and more time building.

Measuring Impact and ROI:

Track how often teams ask questions like, “Do we have a service that does X?” or “Who’s responsible for Service Y?” If these inquiries drop significantly, you’ve hit a milestone. Similarly, if your onboarding times shrink—new hires who previously took weeks to understand the landscape now feel comfortable in days—that’s your IDP delivering tangible value.

2. Self-Service

The Challenge:

Ever see a feature get stuck in limbo because the developer can’t get the right environment spun up? Or watch an ops team drown under a pile of infrastructure requests that never seem to end? Without self-service capabilities, your velocity takes a hit. Developers have to wait on someone else’s schedule to get a test database or a staging cluster. By the time the resource is ready, the developer may have lost context or moved on to something else. Multiply that by all the teams and projects in flight, and it’s a huge drag on efficiency.

What to Prioritize and Implement:

Template-driven provisioning and aautomated pipelines that let developers handle common requests themselves. Pre-approved templates can ensure that every provisioned environment adheres to best practices and security standards, so you’re not just removing a bottleneck—you’re also improving consistency and reliability.

Measuring Impact and ROI:

Start by noting how long it currently takes to get a new environment—maybe it’s three days. After introducing templates, see if you’ve brought that down to a few hours or less. Fewer infrastructure-related tickets and faster environment turnarounds mean your teams can maintain their momentum, delivering features and fixes quicker than before (never mind the increase in developer productivity).

3. Developer Experience (DX)

The Challenge:

Picture a developer’s daily routine: they log into one tool for CI/CD pipelines, another for metrics, another for logs, and a separate browser tab for documentation. This constant context-switching slows them down and increases cognitive load. Over time, this fragmented experience can lead to frustration and reduced morale. It’s not that your teams don’t have the tools—they might have too many, scattered across different interfaces with inconsistent user experiences and integration points.

What to Prioritize and Implement:

An IDP such as Backstage that has all the necessary plugins and integrations properly configured and working becomes a Single Pane of Glass that consolidates these critical elements. By giving developers one place to view logs, metrics, deployments, code reviews, and documentation, you’re streamlining their workflow and cutting down on wasted mental effort. Having an IDP that pulls in data and functionality from multiple sources, presenting it in a coherent, intuitive manner is a surefire way to improve developer experience for your internal engineering teams.

Measuring Impact and ROI:

Survey your developers before and after implementation. Ask how easy it is to find information, how often they switch tools, and how smoothly they can move from coding to testing to deploying. You can also keep an eye on DORA metrics—if the team starts shipping more frequently or fixing issues faster, your integrated interface may be part of the reason why.

Tying It All Together

These three challenges—discoverability, self-service, and DX—often feed into one another. Better discoverability saves teams from reinventing the wheel, reducing the complexity of what you need to maintain. Self-service capabilities speed up delivery, easing the workload on ops and freeing developers to move quickly. Improved DX lowers friction, streamlines daily workflows, and keeps developers happier and more productive. Together, these improvements create a virtuous cycle that helps your engineering organization move faster and build more resilient services.

With that as a baseline, your IDP can really step up and take things a step further. For example, Roadie’s Tech Insights can add another layer of value to your IDP by providing a data-driven, real-time view into the overall health and quality of your services. Rather than relying on anecdotal evidence or gut feelings, Tech Insights surfaces concrete metrics—like compliance scores, dependency health, and adherence to security best practices—within the same platform your developers already use. This makes it easier for engineering leaders to identify hot spots, measure improvement over time, and align investments with areas of greatest need.

Ultimately, you don’t implement an IDP just to say you have one. You implement it because you’ve identified specific, costly problems and want to solve them. Start by pinpointing the real issues—where are you losing time, where are developers frustrated, where are processes too complex or opaque? Then map each problem to a targeted solution—like a service catalog, a set of ready-made provisioning templates, or an integrated interface—and track the results. By focusing on the areas that matter most, you ensure that your platform isn’t broad and shallow, but narrow and deep—truly making a difference where your teams need it most.

Get the “why” and the “what” clear first. From there, your IDP will have real purpose, real value, and real staying power within your organization. It’ll be something people rely on, not just another system they’re forced to use.

For engineering teams thinking about adopting Backstage, this is great news. Here’s why:

Skip the procurement hassle - Dramatically simplify the procurement process and get Roadie through the AWS Marketplace. No need to onboard a new vendor, no unnecessary paperwork, and no invoicing headaches - enjoy simplified billing, consolidated into your existing AWS bill.

Faster adoption - Roadie’s hosted Backstage solution is quick to set up and effortless to maintain, allowing you to skip the complexities, hosting and maintenance of self-hosted Backstage. Purchase Roadie through AWS Marketplace and you can get started even faster by avoiding lengthy approval and onboarding processes, which means you get to delivering value to your developers sooner.

Built for AWS Users - Roadie works seamlessly with your AWS environment, making it easy to manage services, track operational metrics, and boost productivity and DevEx across your teams.

Maximize your AWS discounts - As an added benefit, your Roadie spend on AWS Marketplace also counts toward AWS discount programs, helping you get even more value from your existing Cloud spend.

Visit Roadie on AWS Marketplace and see how we can help your team unlock the full potential of Backstage.

Orchestration tools like Jenkins, Ansible or Salt have been around for a while. Yet the ease of use and accessibility for engineers sets the Backstage Scaffolder apart. Unlike traditional tools, it is built directly into an IDP platform, making it readily available to the entire engineering organization and low cost. It also supports role-based access control (RBAC), which allows fine-grained control over who can create, modify or use the templates.

The Backstage Scaffolder was built originally around the idea of templating repositories in Source Control Management systems to create “golden path” standardization in organizations. However, as with many orchestration tools with broad integrations and API scripting opportunities, the Scaffolder can be effectively used for a broad set of orchestration workflows within cloud native and even more traditional companies with HTTP API access to internal systems and orchestration layers.

Let’s look at some examples of what you can do with this tool as well as some of its limitations in comparison to its rivals.

Creating Accessible Golden Paths

Golden paths in software organizations are essentially agreed-on best practice templates for how to develop, build, deploy and monitor software. Templates could simply be a workflow, such as a best practice CI/CD deployment pipeline or a skeleton code repository for certain workloads, like an HTTP server.

The Backstage Scaffolder was initially designed around this use case — templating new code repositories with a golden path. For instance, when a backend developer wants to create a new AWS Lambda function for a ticket, instead of writing it from scratch or making their own decisions on repository layout, tooling, testing and language, they can run a Scaffolder template that gives them a standardized starting point with a common layout, build tooling and CI/CD workflows.

The Scaffolder has integrations to all the major cloud source control management systems like GitHub or Azure DevOps as well as support for Nunjucks and Cookiecutter templating languages. This allows engineers to get new projects off the ground in a matter of minutes using a clear and secure architecture and workflow. It can embed testing standards and approaches in a codebase to save engineering time and reduce bugs.

At the organizational level it can ensure easy transfer of developers between teams as each team has similar codebase structures and workflows allowing them to get off the ground faster.

Golden Paths at Every Level

But it also can be used  for much higher level golden path orchestrations such as AWS account creation, infrastructure bootstrapping on top of IaC automation tools and even orchestrating workflows in other automation tools such as GitHub Actions.

There are a dizzying array of open source Scaffolder actions available to use as well as a relatively straightforward path to writing custom ones or combining existing generic steps like calling an HTTP API and parsing the response to achieve almost any workflow.

For instance, you can write a scaffolder template that sets up a new team with AWS accounts and users using Terraform and GitHub Actions, stub Confluence docs with their own set of templated internal team documentation such as engineer onboarding checklists, create team Slack channels and document some initial team Objectives and Key Results like getting set up and releasing the first bit of production software.

Templates can save days of engineering and management time as well as ensuring standardized approaches that embed governance tooling and security standards in every team by default.

The Discoverability Problem

There are other tools in this area that allow templating repositories, such as Yeoman. Combined with an internally available CI/CD workflow that can be bookmarked, golden path templating can be achieved for code repositories without the Backstage Scaffolder.

However, discoverability is a key factor in how much these kinds of golden paths are actually used. Discoverability in the CI/CD tooling space is generally very poor as they are often associated with individual repositories or lack search and accessible metadata such as a clear title, description and tags.

The Backstage Scaffolder places discoverability at its core, listing templates in a searchable and filterable page with metadata displayed on cards.

If we compare this to GitHub Actions, a popular cloud-based CI/CD workflow tool, we can see the difference clearly.

Without easily accessible and discoverable templates that are front of mind, it can be a challenge to get engineers and managers using templates, even if they are there.

Combining Sub-Orchestration Workflows

The Backstage Scaffolder can call any other specialized orchestration tool that exposes an HTTP API, chaining together orchestration workflows into larger automation processes with GitHub Actions, Terraform Cloud, Jenkins etc.

Other orchestration tooling can of course do similar things with options including Chef Courier, Jenkins, Salt or Ansible Automation Platform. However these tools are often gated and accessible only to DevOps engineers or infrastructure engineers partly due to pricing models, thereby restricting their impact on day-to-day engineering practices. By contrast Backstage is intended  for all members of an engineering organization on a daily basis. Having a powerful automation tool like the Scaffolder front and center in a daily tool makes driving adoption of golden path templates and time-saving workflows much easier.

Limitations

While the Backstage Scaffolder can be immensely powerful as an orchestration tool, it does not compete with dedicated enterprise-grade orchestration tools in certain areas such as the ability to create scheduled workflows, perform complex logic branching in an easy-to-visualize way or run shell commands. Additionally, it must be manually set up using Typescript code in your Backstage instance with integrations added individually with code changes. If you need RBAC you will need to implement and configure Backstage RBAC with code again, which can be time consuming.

Alternatively managed Backstage solutions like Roadie can provide a ready-to-use Scaffolder with RBAC integrated out of the box and secure runtime features. This is similar to tools like Ansible, which has been packaged with additional UI layers in RedHat’s Ansible Automation Platform. Roadie’s Scaffolder comes with additional features such as easy configuration of template groups, self-serve proxy creation for use with the HTTP Request action and certified templates that help users know which templates are stable and ready to use.

Originally published on The New Stack.

Image from Jevanto Productions on Shutterstock.

Last week the platform world decamped to Salt Lake City, Utah to attend the annual BackstageCon & KubeCon North America conference.

Roadie joined in, of course, and contributed to a lively debate about the future of both spaces. We didn't present at either conference this time around (more from us in this space in April next year when BackstageCon & KubeCon return to the EU), but some great names did.

BackstageCon

Organised by the fine folks at Amazon (Byran Landes) and RedHat (Balaji Sivasubramanian), this years North American edition of BackstageCon was everything we've come to expect and enjoy from a BackstageCon. We heard from Microsoft, Booking.com, and Roku, as well AWS, Redhat and Spotify.

Key take-aways:

• Adopting a product mindset for your Platform continues to be the best way to drive adoption
• A lot of first-time adopters were present, learning about the journey to adoption and the best path to get to a complete Backstage instance
• There was a big focus on defence and security and how Backstage can help organise information for companies in those sectors

Our pick of the sessions:

• Himanshu from Harness gave a great overview of what it takes to drive adoption of Backstage, based on his experience both at Spotify and Humanitec. Key for us was the practical tips in the final third of the talk.

All the videos for the BackstageCon sessions can be found here.

The full schedule of the event can be found here.

KubeCon

KubeCon is inherently a much broader, less Backstage-focused affair, but Backstage still made a lot of appearances in other talks. As an enabler of broader platform initiatives or as the necessary precursor to platform functionality to development teams, Backstage is still the #1 go-to product for Platform teams looking to make an impact.

Key take-aways:

• CNCF officially launched the Certified Backstage Associate
• Security rose in importance, with a full day of Keynotes addresses dedicated to it, and Wiz and Akamai particularly prominent sponsors
• Patrik and Ben from the core maintainer team ran their update session on upcoming changes, focusing on the new declarative frontend 💪

Our pick of the sessions:

• Benson Phillips and Rob Heckel from Toyota Connected ran through their implementation of a platform composed solely of CNCF projects. They've focused heavily on ArgoCD and Backstage when building out their platform and gave a great overview of what they've been up to.

All the KubeCon sessions will be uploaded soon and can be found here.

The full schedule of the event can be found here.

In this blog we’ll explore the Backstage System Model and how you can extend it if you need to.

Why do we need a system model?

Catalogs require at least some structure. If you don’t have a common taxonomy for how to describe each element inside it then it lacks coherence, like a library with no labels on the shelves (or worse yet, contradictory labels). You could pour in all of your various repositories, components, gateways, resources and clusters into a catalog and it will closely resemble a giant blob of nothing.

In a Catalog, information needs to be sorted to have value. Decisions need to be made about what gets included and what does not, and you need an idea of what goes where - how things are categorized now and how they should be categorized in the future.

The Basics

The Backstage data model is made up of nodes ("entities") and edges ("relationships").

Entities

The Backstage data model is built around "entities." Entities are the core units within the Backstage catalog that represent various elements of your software ecosystem.

Each entity is defined via metadata (name, description, labels etc), spec (custom properties), and relations (connections with other entities). In OSS Backstage this information is often piped into Backstage via YAML files that adhere to Backstage's entity specification. Sometimes entities can also come from "Providers" which provide the entity from some source of truth (i.e. Users and Group entities from Okta)

This model allows teams to maintain a structured, discoverable Catalog by distributing the load across every team who owns part of the Catalog.

Friction Warning:

• Backstage advocates for distributed ownership (i.e. each team owns the information in the Catalog that represents the software that it owns) so it can be tricky to update your model and change it over time. For example, if you wanted to replace a Kind all of the various teams would need to update their catalog files. To get around this, a lot of self-hosted Backstage users have built API-based methods for mass updates.

Kinds

Entities are grouped into Kinds. Kinds are like a aisle at a supermarket - everything within it is broadly cohesive and organised around similar principles.

Kinds have a schema and they require a processor to correctly ingest them into the Catalog.

You get some core Kinds out-of-the-box with Backstage, like:

• Domain: Defines larger business domains, organizing systems and components
• System: Higher-level abstraction representing a collection of components working together
• Component: Represents deployable units like services, websites, or libraries)

Friction Warning:

• In OSS Backstage you can extend existing Kinds or write new Kinds to include whatever you’d like, but you need to build or modify a processor each time. That means writing code.
• You will also need consider the long-term impact of a new Kind. You’ll likely be supporting that Kind for a long time unless you want to deprecate it and force entities that use that Kind to fail.

Types

Kinds have Types, allowing grouping within these larger buckets.

Types can be defined on-the-fly. Nothing special is needed to make Types work, any team can create a new Type just by articulating it in their catalog-info.yaml file.

Friction Warning:

• This can lead to a Cambrian explosion of Types, so you may want to introduce some constraint there. Validation of Types is common.
• Annoying errors can creep into Types (i.e.Website and Wesbite) unless you’re validating them in some way.

Relationships

Relationships exist between entities to provide the connective tissue of the Backstage Catalog.

Each Kind has a preset series of permissible relationships that are built when the processor runs for that Kind.

For example, a simple Component might have some API relationships and dependencies defined:

kind: Component
metadata:
  name: artist-web
  description: The place to be, for great artists
spec:
  type: website
  lifecycle: production
  owner: artist-relations-team
  system: artist-engagement-portal
  dependsOn:
    - resource:default/artists-db
  dependencyOf:
    - component:default/artist-web-lookup
  providesApis:
    - artist-api

The Core System model

Out of the box, Backstage comes with a lot of built-in Kinds with attendant relationships so you can get started as quickly as possible.

Some Kinds, like software templates and Locations are effectively atomic and compartmentalised away from other Kinds. The remainder are tied to how the Catalog is built and used to represented entities.

They in effect represent "The Spotify Way" to model software. That’s not for everyone and won’t necessarily work perfectly for you.

If that’s the case, you have two options:

• Force it a little: aka shoehorn your existing concepts into Spotify’s version. This works in a lot of cases, but is necessarily a compromise.
• Re-model: if that doesn’t do the trick, you need to get to work remodeling Backstage entity Kinds and types to fit your needs. Some can be done without code changes, but some need you to get your hands dirty.

Going beyond the basics and extending the Backstage System Model

The Backstage framework is designed to be highly extensible, allowing you to modify or add new Kinds, Types, and Relationships based on the requirements of your organisation.

That said, there are a few things you need to think about when extending the model:

1. No code extensibility

Backstage has flexibility baked on for a large degree of software definition. Using Types or built-in relationships handles for most situations when you want to model your software inside the Backstage System model. 80-90% of the time this will do the trick, but will often come with some degree of compromise. For example, let’s say you want to articulate Value Streams as a top level concept, but have to make do with Value Streams being a Type associated to the Domain Kind. It’s imperfect, but it’ll do in a pinch.

At Roadie, we evaluate and extend the System Model for our customers regularly. That works a lot of the time, but sometimes customers have niche requests that we don’t feel would benefit all our users. This is non-optimal. We want to customers the freedom to extend the model without talking to us or writing code. To achieve that we’re building a fully self-serve, no-code UI for dynamically generating Kinds and defining a system model that can be as arbitrary as you’d like: if you want a Kind called purple-monkey-dishwasher you should be able to have one.

2. Extending the framework using code

Backstage is built around providers and processors. Providers pull data in, processors manipulate and validate that data to build the Catalog entities and relationships.

You can create wholly new providers to handle the ingestion of data from sources not currently handled by Backstage. The Backstage community has built a lot of Providers over the years, but they may require tweaks to fit your specific use-case. For example, Roadie has rebuilt the GitHub provider to use webhook-based ingestion because the size of Catalog we habitually deal with break the GitHub rate limits

You can also modify processors for existing Kinds. For example to extend the list of allowed relationships between Kinds you need to tweak those processors.

You can also create wholly new processors to define new business logic or processes for manipulating and validating that data when you create a new Kind. Going back to the Value Stream example, now you can differentiate Value Stream from Domain and allow the Kinds to deviate usefully from one another. Maybe they each need different allowed relationships, or they’ll build their entities differently: the choice is yours.

3. Data

In the out-of-the-box OSS Backstage model, the data for the system model comes from yaml files. This follows the GitOps model, where changes are made in git-tracked repositories and then ingested by other systems (in this case, the Backstage Catalog).

That means if you want to change or update your model you need to change all those files. That in turn means that opening PRs against every repos which contain a relevant yaml file. This is often a large undertaking, adding significant friction. That’s why most high-volume users of OSS Backstage have built API- and database-based mechanisms to do mass updates. Roadie has two: the Decorator UI and APIs to do a variety of different update patterns (idempotent updates to sync data from a source of truth into Backstage, or just pushing in whole entities via the Roadie Entities API).

Levers to pull when extending the model

Below are some common methods for extending the Backstage data model:

1. Custom Annotations

Difficulty: Trivial

• Why: If you need to add metadata specific to your organization (like security labels, compliance levels, etc.), you can define custom annotations.
• How: Annotations are added as key-value pairs within the metadata.annotations field in your YAML definitions. These annotations can be used to enhance search functionality, create custom views, or provide additional context.
• Example: Adding security-level: high as an annotation for services that handle sensitive data allows you to quickly filter and prioritize compliance and monitoring for these services.

References:

• Backstage Annotations Documentation: Documentation on creating custom annotations to extend metadata.

  apiVersion: backstage.io/v1alpha1
  kind: Component
  metadata:
    name: fraud-detection-model
    description: "AI model for fraud detection"
    annotations:
      security-level: high
  
  ...
  

2. Custom Types

Difficulty: Easy

• Why: In cases where the existing entity types (Component, API, etc.) do not fit your specific resources, you can create custom entities.
• How: Define a new entity type in any valid catalog-info.yaml. This simple involves adding a new type to the spec.type in the YAML file.
• Example: Suppose you have machine learning models as a core resource in your project. You could define a new model type.

References:

• Roadie Kinds and Types documentation talks a lot about how to use Types without introducing problems

  apiVersion: backstage.io/v1alpha1
  kind: Component
  metadata:
    name: fraud-detection-model
    description: "Machine learning model for fraud detection"
    annotations:
      security-level: high
  spec:
  	type: model
    version: "1.0"
    trainingDataset: "transactions-v1"
    accuracy: "95%"
  

3. Modifying Existing Kinds to Add Custom Relations

Difficulty: Normal

• Why: Relationships between entities help you capture dependencies, ownership, and team structures within your catalog. If your use case involves additional relationship types, custom relations can improve representation.
• How: Modify the relevant processor for a given Kind to enable new types of relationships to be built for that kind. Then define relations within the spec.relations section of the YAML file.
• Example: Suppose you want to track models associated with data sources. You could create a custom relation usesDataFrom, linking ML models to the Resource entities that document data sources they rely on.

References:

• Roadie Kinds and Types Documentation: Provides practical examples of defining and extending Kinds.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: fraud-detection-model
  description: "Machine learning model for fraud detection"
  annotations:
    security-level: high
spec:
	type: model
  version: "1.0"
  trainingDataset: "transactions-v1"
  accuracy: "95%"
  relations:
  - type: usesDataFrom
    targetRef: resource:exampleorg/some-data-source
    target:
      kind: resource
      namespace: exampleorg
      name: some-data-source

4. Creating Entirely New Custom Kinds

Difficulty: Normal / Hard

• Why: When the System Model cannot adequately encapsulate how you build software or the relationships between various parts of your organisation, you will need to build a custom Kind.
• How: Write a new processor for that Kind and define a custom schema for that Kind. This ensures all entities adhere to required fields, valid types, and constraints, providing an additional layer of validation. Then add new catalog-info.yaml files for the new Kind to relevant resources, or modify existing catalog-info.yaml files.
• Example: For the MLModel entity, you could create a new Kind to represent that in your System model. Using that new Kind you could then model relationships as version, trainingDate, and accuracy.

References:

• Backstage JSON Schema Documentation: Explains how to define and enforce custom schemas.

Conclusion

Backstage is an extremely flexible framework for modelling software and once the building blocks and options are understood it’s simple enough to fully customise the model.

Useful links:

• Backstage System Model: official docs and a good starter diagram for how entities in the Catalog interact.
• Backstage Entities: official docs on the lifecycle of entities
• Backstage Relationships: official docs on how relationships work inside Backstage
• Modelling software in Backstage: Roadie blog from 2021 about how to model software in Backstage using the core system model. This still represents a great primer on the out-of-the-box system model and how you could use it.

According to Gartner, IDPs are reported as the most frequently piloted technology in their 2022 - 2024 Technology Adoption Roadmap Survey, and 75% of organizations with platform engineering teams will provide IDPs by 2026.

With this growth, 3 main categories of Internal Developer Portal products have emerged:

• Open-source IDPs are exactly what you think. The code is open-source and you host and maintain it yourself.
• Proprietary IDPs are typically software as a service startups funded by venture capital. You purchase access to them just like you purchase access to PagerDuty or GitHub Enterprise.
• Hybrid IDPs are a combination of open-source and proprietary. They’re built on an open-source foundation, but come with commercial support and proprietary features.

In this article, I’m going to dive into each category one by one, and learn the pros and cons of each.

I’ll also explain why I believe the Hybrid model brings the best of both worlds. The community, extensibility and lack of vendor lock-on of the open-source model, alongside the low cost, ease of use, and fast deployment of the proprietary model.

Open-source

While technically not an IDP, and technically not the only open-source IDP, you can’t talk about open-source IDPs without the conversation focussing on Backstage.

Backstage was open-sourced in early 2020 by Spotify. It was a rewrite of their internal IDP that they’d been using for years. It’s not an IDP because it’s actually a set of TypeScript libraries that developers can combine together to build an IDP, but it has still captured a huge amount of attention in the IDP space.

When Backstage launched, it was often compared to Hygieia from Capital One, which predates it by almost 4 years. Even after Backstage appeared, a third contender emerged with a strong start. Ride-hailing company Lyft threw their hat into the ring when they released Clutch a few months after Backstage was open-sourced. Clutch describes itself as "an extensible platform for infrastructure management”.

in 2024, Clutch popularity seems to have stalled, Hygieia is deprecated, and the reality is that Backstage dominates the open-source IDP market. Although not a perfect metric by any means, GitHub stars show the difference in popularity of each tool.

Benefits of the open-source model for IDPs

In many ways the open-source model is perfect for building an Internal Developer Portal. IDPs are a window into all of the tools that developers use. They’re the venerable “single pane of glass”.

Developers use a lot of tools though, and the developer tool landscape is thousands of tools strong and growing. Even the Cloud Native Landscape alone has more than 1,000 tools in it’s ecosystem.

How can a single IDP integrate with and plug-in to all these tools? Well, by creating an open-source community who will help with the effort.

Huge community

This is exactly what Backstage has accomplished. The Backstage community is huge and engaged.

• Backstage was the top end-user contributed CNCF project of 2023, with more than 4,000 contributions from end user companies.

Benefits of the open-source model for IDPs

In many ways the open-source model is perfect for building an Internal Developer Portal. IDPs are a window into all of the tools that developers use. They’re the venerable “single pane of glass”.

Developers use a lot of tools though, and the developer tool landscape is thousands of tools strong and growing. Even the Cloud Native Landscape alone has more than 1,000 tools in it’s ecosystem.

How can a single IDP integrate with and plug-in to all these tools? Well, by creating an open-source community who will help with the effort.

Huge community

This is exactly what Backstage has accomplished. The Backstage community is huge and engaged.

• Backstage was the top end-user contributed CNCF project of 2023, with more than 4,000 contributions from end user companies.
• There are 17,000 people in the Backstage Discord channel, with dozens and dozens of questions being asked or answered every day.
• Backstage has a deep partner ecosystem. Whether you’re a Fortune 50 enterprise or a Series C startup, there’s a partner who can help you deploy and use Backstage.
• Backstage has official support from leading DevTools companies. Companies like Snyk, PagerDuty, Dynatrace and even AWS have released officially supported plugins for Backstage.

This vibrant community means that you can always get help with Backstage. The other benefit is the extensibility and the number of integrations available.

Huge number of integrations

The result of this large community is that Backstage is installed and adopted at thousands of organizations. These organizations use a diverse set of engineering tools, and many adopters have built and open-sourced Backstage plugins for these tools.

The Backstage plugins directory has more than 200 plugins available, so you can be fairly confident that there are plugins available for the tools you use.

Highly extensible

Underlying all these plugins is a framework which is designed from the ground up to be maximally extensible. The docs say:

Backstage is a single-page application composed of a set of plugins.

Our goal for the plugin ecosystem is that the definition of a plugin is flexible enough to allow you to expose pretty much any kind of infrastructure or software development tool as a plugin in Backstage.

Practically, this means you can customize and extend your Backstage install to your specific requirements. If you need to build your catalog in Gerrit rather than GitHub, then you can swap out the GitHub integrations and replace them. The same concept applies to most parts of Backstage.

By coupling this technical extensibility with a permissive Apache 2.0 license and a mature governance model, adopters can make wholesale changes to Backstage if they like.

The result is that many organizations have customized Backstage heavily. Take a look at this screenshot of Sunrise, the Backstage-based IDP that Zalando created, as an example.

This completely custom UI shows the progress of changes as they flow through CICD on their way to production. Very little of what you see here comes out of the box with Backstage. Instead, it’s wired together from the basic building blocks of Backstage plugins.

Lack of vendor lock-in

Because Backstage is open-sourced and backed by large end user companies and the CNCF, adopters can feel comfortable that it will be around for a long time, and will continue to meet their needs into the future.

The IDP market is still very early. The majority of the players have been founded in the past 5 years. We will likely see some consolidation of this market as players get acquired and wound down. Any customers of these companies will be forced to migrate to a different solution.

We’ve seen this play out in the CICD market already. There was a time when Travis CI was a dominant provider of continuous integration. In 2019, Travis CI was acquired by private-equity firm Idera. One month later, layoffs began. In early 2020, Travis CI stopped providing free support for open-source products, and in late 2020 the pricing changes began.

Companies can avoid this fate in the IDP market by building on an open-source project instead of a commercial vendor with it’s own data model and lock-in.

Drawbacks of the open-source model of IDPs

It’s expensive to build

Backstage is a larger undertaking than most people realize before they get into it. To quote Gartner in their Innovation Insight for Internal Developer Portals report:

Gartner inquiries indicate that Backstage implementations may require substantial effort in standing up the service.

In order to replicate something approaching the Sunrise portal mentioned above, companies should expect to allocate 2 to 5 engineers to the project for a number of years.

Don’t believe me? Zalando said it themselves in this presentation they did at the Autodesk Developer Productivity Summit.

All that I’ve showed you was done through the contributions of many teams, but the core team was three engineers and a engineering manager.

That team has been working on Sunrise since 2020.

And that’s where my team is focussed on, actually like, the discovery part of our application landscape, and where we actually decided to, a few years ago in the beginning of the Backstage era in 2020 when Backstage was released, to actually onboard ourselves into Backstage.

Let’s do the math. Assume an engineer or engineering manager costs $250k per year, therefore the team of 4 costs $1 million per year. 4 years have passed since 2020, so they’ve spent $4 million in total on their core Backstage team. Add in the “contributions of many teams” and we could easily be talking about a $6 million outlay.

Now you’re probably saying “yeah but we’re not as big as Zalando”, but trust me, even small companies will spend a half a million dollars pretty easily.

It requires uncommon skills

Backstage is typically owned and managed by platform teams or DevOps teams. These teams are probably skilled in YAML and kubernetes-native languages like Go.

Backstage is written in TypeScript. The backend runs on NodeJS, and the frontend is written in React.

Organizations who are considering self-hosted Backstage should consider whether or not they have the skills to develop in these languages. As mentioned above, Backstage is a set of libraries that users combine together to make a developer portal. That means you need to write TypeScript to make use of it. You need to understand frontend development, HTML and CSS to customize it. It may not make sene make sense to build up these skills in a Platform and Infrastructure organization because they will not be easily transferrable to other projects.

The second set of uncommon skills that a Backstage deployment needs are project management and developer relations. Once it’s live, you need to evangelize it. You have to go out to the rest of the organization and teach them about it and how to use it. Are your platform developers really going to want to do this work?

It’s got a long time to value

The typical Backstage rollout has a few phases. It starts with an initial deployment, which typically takes a few months to get a basic Backstage instance into production. Some parts of Backstage will work straight out of the box at this point. You’ll be able to use the scaffolder to create new software projects from predefined templates, for example.

Other parts of Backstage require an adoption phase. The software catalog is usually one of them. In order to have a complete software catalog, teams must first put their software into it.

In 2023, at BackstageCon North America, I spoke to the experiences of some Backstage adopters who had attempted this feat. I first explained that 60% of the Backstage adopters I had interviewed were attempting to populate their catalog by writing YAML files that contain metadata about their services (these are called catalog-info.yaml files).

I then shared some quotes from adopters who had followed this path

The catalog never really worked for us. We struggled with adoption and rate limiting. Half the info in the catalog was wrong when they first pushed adoption. 2,000 engineers, 2 years experience

The thing we struggled with the most was getting teams to create the YAML file. We spent 6 months actively encouraging teams to register their components with Backstage. After 1 year, only 30 or 40% of the active repos were there. 120 engineers, 1 year experience

Without a lot of product development and project management, this catalog population process takes a long time. There are many Backstage adopters who are at less than 50% catalog completeness after two years with the tool. Even Spotify admit that Backstage adoption rate often stagnates at 10%. You may have dreams of your software catalog becoming a single pane of glass for all of the software development in your organization, but it simply cannot fulfill its purpose if the software is not in there.

Proprietary

At the other end of the spectrum, there are a number of proprietary developer portals like Cortex and Port.

The concept of proprietary developer portals is largely the same as Backstage. They have similar features and target a similar user in the engineering organization. The main differences are in their extensibility. Users can’t edit the code of a proprietary developer portal, and there won’t be a large community who are producing open-source plugins.

Benefits of the proprietary model for IDPs

While a proprietary developer portal may never be as extensible as Backstage, they certainly do have some benefits.

Quicker to get started

Proprietary IDPs will mostly just work out of the box. You don’t have to plan sprints and do research and write code to get some value. They come with admin interfaces where you can set some properties, connect up your tools, and starting using the product straight away. This is likely a better experience for platform engineers who don’t have the time or expertise to deploy and customize Backstage from scratch.

When you do need to configure something, the documentation provided by a proprietary developer portal will likely be more comprehensive and up to date. They’re simply more motivated and able to produce high quality documentation than an open-source community. This makes setup faster and less confusing, and helps to prevent misconfigurations.

User-friendly interface

Proprietary IDPs bring their own user interface (UI). They’re developed from the ground up with design systems in place to help to ensure consistency across integrations and pages. They also have designers and UI engineers working to implement design practices in a coherent way.

While the Backstage core team have done a lot to put the tool in a good place, the reality is that Backstage’s default interface leaves a lot to be desired.

This reality is brought on by the combination of overly familiar Material UI libraries and community contributed code that can be hit-or-miss in terms of how it looks and feels.

Backstages plugins can be contributed by anyone, so there’s no guarantee that they will look good, or look consistent. It’s very likely that different plugins will work completely differently even though they are installed in the same Backstage instance.

More guidance

Proprietary IDPs are more opinionated in how they should be used. When you work with a vendor you should get access to solutions engineering and customer success resources who can help to share lessons from other customers.

Working with the open source can sometimes feel like being handed a box of car parts and asked to build a car. Sure you can figure it out eventually, but you’re going to make mistakes along the way, and it probably won’t be the most efficient path.

Having a partner to work alongside and share lessons is worth a lot, and helps to cut the time to value for your IDP project.

Drawbacks of the proprietary model for IDPs

Vendor lock-in

Proprietary IDPs have their own in-house APIs, their own data model, and their own way of doing things. Many of the options on the market are early stage startups. There’s a good chance some of them will disappear over the next few years. If you’re a customer, your IDP will go with them.

When they do survive, you’ll likely be forced to choose between paying up for price hikes, or completing an expensive migration project to a different solution.

Limited extensibility

It’s not possible to change the code of a proprietary IDP. If there’s something about the way it works that doesn’t suit the needs of your company, your best hope is to contact support and hope that they have the desire and the roadmap space to change it.

Most development organizations have some homegrown tools that have sprung up over the years internally. These tools only exist inside the company they were created, and no IDP vendor will have an out-of-the box integration for them. For these tools, you need to create your own plugins if you wish them to be part of your single pane of glass IDP. Some proprietary IDPs do support custom plugin development, but you’ll need to do so using only the concepts and libraries that the vendor provides, and the plugins you create will not be transferrable to any other solution.

Proprietary IDPs have fewer integrations than the massive Backstage community. Even the most well funded proprietary vendor has 48 plugins listed on their integrations page, less than a quarter of the number that are available for Backstage. No matter how big this vendor grows their team, they won’t be able to compete with the 500+ companies who have contributed to Backstage.

Limited speed of execution

The tech landscape is constantly shifting and evolving. Before serverless we had Kubernetes, before Kubernetes we had virtualization, before virtualization we had bare metal. In between and around these major platform shifts we had and continue to have a massive number of different types of serverless, containerization and virtualization options.

A developer portal is supposed to wrap all of these shifts and technologies in order to give the best possible coverage over your internal engineering landscape. Proprietary IDPs may struggle to capture all of these technology shifts into the future, leading to gaps in the portal that platform teams put in front of their internal users.

Hybrid

Last, but not least, we have the Hybrid model. This model takes an open-source foundation with hundreds of plugins and integrations available, and makes it available in an out-of-the-box format which is easy to get started with.

This is what Roadie is. It’s the only standalone IDP based on Backstage, and it offers the best of both worlds - extensibility and speed of execution.

Huge number of integrations

The vast majority of plugins that have been created for Backstage can work on Roadie. Our integrations library contains 70+ plugins and integrations today, and we’re adding more all the time, in line with customer requests. Once a new open-source plugin is created, it takes us about a day to make it available in Roadie.

The only reason we wouldn’t add a Backstage plugin are that it’s of such low quality that it doesn’t work or provide any value.

This means that Roadie customers can effectively choose from 200+ plugins and integrations that the Backstage community has created and use them on our platform. In fact, we’ve created some of the most popular plugins ourselves and open-sourced them.

Highly extensible

Roadie strives to support as much customization of our IDP as possible. In addition to theming and branding, we support a custom data model, custom Backstage plugins, custom proxies, layouts, scaffolder actions and on and on. You name it, we can probably let you customize it.

This means that you can customize Roadie to meet the specifics of your organization, improving adoption and facilitating ease of use.

No vendor lock-in

Roadie is API compatible with Backstage. The same software metadata schema that works on Backstage will work on Roadie. Custom frontend plugins written for Backstage will also work on Roadie. We’re just supporting and extending the same APIs.

This has two benefits for customers.

1. They can migrate off self-hosted Backstage onto Roadie very easily. Numerous organizations have already done this. Simply connect us to your source code management tool and we’ll automatically ingest any software metadata files you’ve created.
2. They can migrate back to self-hosted Backstage very easily. In fact, our Terms of Service dictate that we will give you your data back within 30 days of termination.

User friendly interface

Roadie’s interface is vastly improved compared to the barebones Backstage experience. We’ve worked with our users to streamline how people use the product. This means better out of the box defaults and easier customization.

Inexpensive

In many cases, Roadie is 5 times cheaper than self-hosting Backstage.

People tend not to realize this up front, but the most highly developed Backstage deployments in the world have cost millions of dollars. We talk to companies all the time who have built a team of 5+ engineers around Backstage and run those teams for 3+ years. The most expensive Backstage deployment we’ve seen cost $7.5 million dollars over 3 years.

Roadie doesn’t require such an extensive team to be built around it. We take away all of the operations and deployment effort, and much of the customer support that has to happen. We also give you out-of-the box features like Scorecards and Role-Based-Access-Control that you would otherwise have to build yourself to realize the potential of your IDP.

Lastly, we charge based on usage. You only pay for the engineers who write code which is tracked in the catalog. Everyone else gets to log in for free. This means you can ramp gradually throughout your org, rather than paying for the developement of the entire IDP up front.

Requires few skills

Roadie doesn’t require any TypeScript skills. You just drag and drop plugins where you want them, and configure the product in our administration panels. It’s dead simple. This is perfect for platform teams who are trained on cloud-native technologies. We even have no-code plugins that you can use to display lists or charts inside the UI without writing any code.

If you do need to make something completely custom and you don’t want to write a Backstge plugin yourself, we can sort you out with professional services to suit your needs.

Short time to value

We’ve built a ton of features into Roadie to help shorten the time to value. For example, we help teams build out their catalog quickly and we see customers reaching a high level of catalog completeness within 4 months.

We also provide customer success and guidance to help organizations effect change with their IDP. We take the lessons we’ve learned over the years and use them to guide you to success in as short a time as possible.

Conclusion

We’ve designed Roadie from the ground up with flexibility and easy adoption in mind. We believe engineering organizations should not waste their resources on undifferentiated heavy lifting projects like deploying an open-source IDP. At the same time, they shouldn’t be locked into a proprietary data model or limited in terms of what they can build or the integrations that are available.

It turns out that you can have your cake and eat it too. An Internal Developer Portal can be both easy to use and supported by a massive and growing open-source community. This means that you can focus on unlocking productivity in your engineering organization while knowing you have a rock-solid and flexible foundation beneath.

--

I’ve also written about the difference between IDPs and the IDP space in general over on The New Stack

Image by MSTORK from Pixabay

🚨👮 RBAC with Fine-Grained Control

Earlier this year we launched our Role-based Access Control (RBAC) feature to help our customers have greater control over who does what within Roadie.

That started off at a fairly coarse-grain, with the ability to restrict basic CRUD actions within the application to a subset of users.

The ultimate goal was always to have much tighter and more specific controls than that though, so we built a new layer for our RBAC controls.

Now, with our RBAC plugin you can:

• Create and surface custom permissions for your Custom Plugins
• Create custom permissions policies to group those permissions together, either using our built-in permissions or Custom Plugin permissions you've created
• Attach ALLOW or DENY options to those permissions policies so that you can more flexibily articulate the policy you'd like to create
• Target annotations and other entity metadata to attach permissions

That means you'll be able to:

• Target specific catalog items
• Block scaffolder templates that you only want certain users to be able to see or run

🏛️ New Backstage Backend (and we're now on v1.30)

We've made the jump to the new Backend now that it's officially stable. It wasn't a trivial thing though, and there are a few potholes that we hit along the way. To help others making the transition we even wrote a blog about it.

We also took the opportunity to upgrade to v1.30 of the core project, so that's fun.

🎨 Notifications have come to Roadie

A set of Notifications plugins landed with Backstage v1.28 and they've recently been integrated into Roadie. There's still some way to go for Notifications to mature in the OSS project (for example, individual plugins are responsible for emitting notifications and there currently isn't a way to really manage those - they're all just transparently sent to users 😅) but it's moving fast. There should be some notification management in the next release for example.

We'll be working in the next few months on integrating Notifications into various plugins that we manage, including Tech Insights. Exciting stuff!

💈 Scaling, scaling, scaling

Roadie customers have pushed the limits of Backstage lately.

Multiple Roadie customers now have:

• Catalogs with 200k+ Component and Resource entities
• Catlaogs with 10k+ users and groups entities
• Tech Insights facts with millions of data points captured

That presented some major scaling challenges for our infrastructure and the way we have architected our version of Backstage.

After a lot of work to identify bottlenecks, scale cloud resources where required, and signficant engineering time introducing performance improvements, we're now able to comfortably handle even the largest Backstage installations in the world.

🧘‍♂️ Custom Columns v2 & Fact Tables(in beta)

Software Catalogs tend to be tables. They're the most intuitive mechanism for displaying large quantities of information.

Making those tables super-flexible is top-of-mind for Roadie at the moment. So much so that we have not one but two features in beta-testing to help introduce much more flexibility into our tables.

The first is Custom Columns v2. The first cut of Custom Columns allows you to take arbitrary metadata from each Catalog entity and add it to your Catalog tab as a new column. The values were rendered as strings though, so we're adding a whole bunch more types to allow easier comprehension of the information. Think: numbers, colourful ranges, links and Catalog entities, all accessible and filterable from within a Custom Catalog Tab.

The second is Fact Tables, which allow you to display Fact data that you've ingested for your entities from Tech Insights Data Sources (either built-in ones that we provide or custom ones you've created). You can filter it by team and display multiple Data Sources side-by-side.

Soon the two worlds will collide 🌚 🌝, with facts accessible inside Custom Columns. That means you'll be able to have a Catalog tab with data from any configured source alongside existing Catalog data.

🔌 Plugins & Integrations roundup

• Wiz plugin: we made a new plugin! The new Wiz security frontend plugin now surfaces Wiz data inside Backstage. It's available both inside Roadie and as an OSS plugin. Wiz certification for this plugin is ongoing but should happen soon. We'll be working on a Wiz Tech Insights data source after that. Watch this space 👩‍🚀🚀
• LaunchDarkly plugin: we also made a LaunchDarkly plugin for managing feature flags and surfacing configuration inside your Backstage instance.
• Shortcut plugin: we made a Shortcut plugin. It's not quite as widespread as Jira (that's a pretty high bar...) but Shortcut is gaining some traction as a nicer way to engage with software development tickets and workflows. We use Shortcut internaly and we're big fans. Check it out.
• Okta & Jira plugins: we've refreshed our Okta and Jira plugins to make them more seamlessly work with the new Backstage backend. Enjoy.
• AWS account creation: last but by no not least, on the topic of catalog data sources this time rather than plugins - our AWS provider has been modified to ingest EKS clusters and containers auto-magically. This is the first step in allowing a one-hit configuration of both AWS and the Kubernetes plugin (we're currently modifying the K8s plugin to ingest these resources and take the config from AWS).

Driving adoption of Backstage—or any internal developer portal—in an organization usually hinges on organization-wide initiatives. These initiatives attempt to coordinate the adoption of specific parts of the product. But behavior-change initiatives can be tricky to navigate and succeed with.

This post outlines a variety of proven strategies for fostering Backstage adoption. The strategies you select should depend on your goals for Backstage and the current structure and realities inside your organization.

The strategies highlighted here are based on experiences from Roadie customers who’ve already implemented them successfully. Determining whether they’re the right fit for your context is down to you.

Defining Successful Backstage Adoption

First, lets consider what successful Backstage adoption looks like for your organization. How will you measure this success? Do you envision:

• Daily usage: 80% of engineers logging into Backstage each day?
• Catalog completeness: 90% of your software fully cataloged?
• Dependence: 90% of teams reporting Backstage as essential for their workflow?

Most likely you'll have a combination of goals: Some teams may use scaffolder templates only occasionally, but this could save months of engineering time. Some might use Backstage solely for API documentation, which provides immense daily value on its own. And you may want to automate software maturity reporting with TechInsights so senior management has real-time insights into metrics like software security.

Each goal has intermediate milestones. For example:

1. Every team uses the scaffolder at least once per quarter.
2. 50% of new software is templated by the scaffolder.
3. 90% of new production software is templated by the scaffolder.

And each of these steps requires a distinct strategy to move forward. Broadly, these can be split into:

• Land and Expand: Start small, growing adoption progressively across teams.
• Expand and Land: Launch with widespread value, aiming for rapid, organization-wide onboarding.

Below, we’ll explore tactics within each approach. You don’t have to pick just one—mixing strategies can often yield the best results.

1. Land and Expand

This approach usually involves piloting Backstage with a few teams or a specific area of the organization before expanding it. It’s especially useful for organizations with a more varied and independant engineering culture, or one lacking top-down support.

Benefits of “Land and Expand”

• Learning and iteration: Working closely with a small group allows you to gather insights, refine processes, and tailor adoption strategies for a broader rollout.
• Inspire others: Early adopters create a model that other teams can learn from.
• Build organic momentum: Introducing Backstage “aha” moments, such as the scaffolder, can drive excitement and encourage others to adopt.

Proven Tactics for “Land and Expand”

The tactics that work best depend on your organization’s circumstances. Here are some approaches that have worked well in real-world applications:

1. Automate multi-step processes with Scaffolder templates

Start by streamlining time-consuming tasks. Even a handful of templates can deliver significant and measurable savings that are appreciated by both engineers and leadership.

Example: If you’re setting up a team to build a customer communications pipeline, a few clicks in Backstage can provision AWS dev/prod accounts, a secure VPC, a sample server with CI jobs, and internal docs, all in under 20 minutes.

2. Get content into the catalog early

Starting with an initial set of entities can provide immediate value and help teams see the utility of Backstage as a source of information.

Example: Import users and teams from an identity platform like Okta to build a visual map of team structures with minimal setup. Teams can instantly use Backstage to answer questions like “Who is the product manager for the Customer Success team?”

3. Promote API documentation as a go-to resource

Make Backstage the primary source for looking up API documentation by encouraging teams to add their APIs early. This can encourage wider catalog usage.

4. Develop custom plugins for organization-wide needs

Create your own custom plugins that address common pain points across the organization. This can create a compelling reason for users to engage with Backstage regularly.

Example: Lunar Bank built a plugin for RabbitMQ dead letter queues, which drove high initial usage and catalog engagement.

Expand and Land

This approach aims for rapid onboarding of multiple teams by delivering value across the entire organization. It generally requires top-down support or a dedicated platform team to do the initial heavy lifting.

Key Tactics for “Expand and Land”

Prerequisites - Tracking usage and integration

Having data on team-level adoption can help track progress in adoption in several key areas.

• Catalog: Adding repositories to the catalog can help measure catalog adoption and provide comparative progress data for teams. i.e. The Identity team has 30 repositories with CI pipelines and only 5 entities in the catalog. Roadie comes ready populated with repositories in your catalog, a graph of catalog completeness and ready made Scorecards to track it in Tech Insights.

• Scaffolder: Data on Scaffolder usage is available already in the database as standard, though you’ll have to extract it and analyse it.

• TechDocs and Plugins: Usage of various plugins can be assessed via tracking like Google Analytics which can easily be integrated into Backstage early on.

Roadie provides usage analytics like Catalog completeness out of the box to its customers as its such a common requirement to understand a portion of the ROI equation.

1. Set clear mandates

Organizational mandates can drive adoption effectively if they’re tied to broader goals, have backing from senior leadership and are checked up on at a regular cadence or certain date.

Example: “Each team must have its production services cataloged with a PagerDuty annotation by September 21st to facilitate incident impact mapping.”

2. Regular check-ins and visualizations

Regular team check-ins or public dashboards showing migration progress can improve prioritization and motivate teams. Twilio explained how they do this at the Autodesk Developer Productivity Summit.

3. Celebrate success

Highlight the progress of individual teams to foster collaboration and enthusiasm. Regular shout-outs for API additions or new scaffolder templates can build positive momentum.

4. Integrate Backstage into new hire onboarding

Familiarize new hires with Backstage from day one. Add onboarding docs to TechDocs, making Backstage the go-to place for engineers starting out.

5. Pre-populate the catalog

Consider using scripts or providers like the AWS provider to populate the catalog with essential data, reducing the manual workload.

Conclusion

There are many potential strategies to achieving your incremental Backstage adoption goals via organization wide initiatives. Planning what to try and at what point is not an easy choice. The ideas above are not an exhaustive list but hopefully can give you some fresh approaches in your own adoption journey.

Image by Dirk (Beeki®) Schumacher from Pixabay

Developer Experience (DX) refers to the tools, systems, and culture that impact how developers work. It reflects how efficiently developers can do their best work without frustration or impediment, and as such, DX is stronglyassociatedwithhigh-performing teams that deliver impactful, value-creating software. When building software, the highest cost is almost always engineering time, so it make sense to optimize for developer efficiency and productivity.

The benefits of improved DevEx: https://newsletter.getdx.com/p/impact-of-developer-experience

As organizations grow, developer experience often suffers due to increased friction, formal processes, and fragmented knowledge, which hampers productivity and collaboration. Informal communication channels that once worked well in small teams become ineffective, and developers struggle to find information or navigate complex systems - a sort of software development Dunbar’s Number . Maintaining a strong DX requires proactive strategies and consistent adaptation to evolving team sizes, ensuring that workflows remain efficient and developers stay engaged and motivated.

How to Improve Developer Experience with Backstage

How can large software organizations deliver a great DX? Improving DX typically involves addressing core challenges that hinder efficiency as organizations scale, many of which can be resolved by using an Internal Developer Portal (IDP) like Backstage:

Discoverability of Services and Information

As organizations grow, developers often struggle to find services, APIs, and documentation. A developer might spend hours searching for documentation or rebuilding a service simply because they can't locate what they need.

How Backstage Solves This:

The Backstage Service Catalog centralizes services, APIs, documentation, and ownership, making it easy for developers to find what they need, see dependencies, and know who to contact - eliminating wasted time. TechDocs, a core feature of Backstage, makes internal documentation easily accessible and up-to-date by storing it alongside the code itself. This centralization reduces friction and boosts productivity by providing a single interface for all documentation needs.

A typical Backstage software catalog

Quality and Speed of Releases

In software delivery, pace and consistency are critical. Fragmented workflows and inconsistent CI/CD pipelines can lead to uncertainty and delays, slowing down releases and increasing errors.

How Backstage Solves This:

Backstage templates standardize releases by automating workflows and enforcing best practices. By integrating with tools like GitHub, Backstage ensures a consistent, transparent software delivery process, reducing friction and increasing developer confidence.

Self-Service Capabilities and Workflow Simplification

In larger organizations, developers often depend on other teams for tools and approval, which can cause delays and frustration. Self-service tooling helps developers maintain momentum without waiting for approvals.

How Backstage Solves This:

Backstage enables developers to self-provision resources, deploy services, and configure environments through integrations with cloud providers like AWS or Kubernetes - removing bottlenecks and keeping developers focused on coding.

A collection of software templates that automate common workflows - everything from creating new customer environments to opening a new PR to add software into a Backstage software catalog.

Governance, Standards Adherence, and Complexity Management

As organizations grow, maintaining governance and ensuring adherence to standards becomes challenging. Over time, undocumented services and lack of enforced policies lead to vulnerabilities, with leadership often lacking visibility into best practice adherence.

How Backstage Solves This:

Backstage’s Tech Insights plugin enforces governance by setting up custom checks to monitor security, testing, and compliance, helping teams ensure their services meet internal standards and guidelines. Developers use Tech Insights to address gaps, while leadership tracks governance adherence, providing a unified view of service health and managing complexity as the organization scales.

Tech Insights scorecard showing adherence of the organization to Backstage component best practices over time

How to Track and Measure DX

Implementing an IDP like Backstage is an important first step to improving developer experience, and helps address some of the more obvious impediments to DX. The crucial next step is understanding how to track DX and measure it. This is especially true as your organization grows, so do its complexities. Continuous feedback from tools that track DX are critical to ensure that you’re always ahead of potential bottlenecks or compliance gaps. Measuring DX can take many forms. There are two main approaches to consider depending on your organization's needs and existing tools.

Approach 1: Aggregate DX Metrics in a Third-Party Tool

One approach is to use specialized third-party tools to collect DX metrics and then surface those insights within Backstage. Tools like getdx.com and open-source DORA plugins are specifically designed to aggregate developer productivity data. They track metrics like cycle time, lead time for changes, and deployment frequency, and then present the results in a dashboard format.

OpenDORA courtesy of https://nl.devoteam.com/expert-view/introducing-opendora-team-performance-observability-for-your-organization/

For example, you could integrate a DX Plugin that aggregates productivity metrics from across your CI/CD systems and other development tools. These metrics could then be displayed in Backstage, where developers can easily see how well they’re performing against industry benchmarks. Metrics like DORA (Deployment Frequency, Lead Time for Changes, Mean Time To Recovery, and Change Failure Rate) are especially valuable for tracking operational efficiency and developer productivity.

This approach uses specialized tools for deeper analytics while keeping insights in Backstage, minimizing context-switching.

Approach 2: Aggregate Metrics Directly in Backstage with Tech Insights

Alternatively, you can collect and track DX metrics natively within Backstage using the Tech Insights plugin. Tech Insights enables you to define and track custom checks and metrics, giving you full control over what data you collect and how it's presented. This approach allows you to integrate metrics from your existing systems (like GitHub or monitoring tools) and aggregate them in Backstage, providing a centralized view of DX without relying on third-party dashboards.

Using Roadie’s Tech Insights plugin, you can create scorecards that track various metrics related to DX. For instance, you could monitor the percentage of services adhering to security policies, the number of projects using automated testing, or DORA metrics - all from within Backstage itself. Scorecards can also be viewed for each individual team - so with this data, those teams can quickly identify areas of improvement, measure changes, and identify focus areas to optimize DX.

This approach centralizes everything in Backstage, allowing for more tailored metrics and scorecards based on your organization’s needs while reducing context-switching for developers.

Tech Insights scorecards on a Team page in Backstage

The right DX tracking approach depends on your organization's goals, tools, and the complexity of your developer workflows. Whether you aggregate metrics in a third-party tool and display them in Backstage or centralize everything within Backstage using Tech Insights, the key is making DX measurable and visible to all stakeholders. A strong developer experience is about tracking, iterating, and improving workflows while providing transparency and actionable insights.

Image by GrumpyBeere from Pixabay

Migrating to Backstage’s new backend system is more than just an upgrade - it’s a step toward a more scalable, efficient, and streamlined platform. The new backend introduces a modern architecture that simplifies plugin integration, reduces complexity, and improves overall performance. This is a significant shift from the monthly updates and incremental improvements you may be used to. By adopting this new architecture, you’ll gain better control over plugin dependencies, reduce overhead, and set the stage for easier future development.

This migration is particularly exciting because it marks the culmination of a long journey towards a more modular and efficient backend. Originally introduced as an experimental feature, the new backend has matured into a fully supported system that greatly enhances how plugins interact with core services. It’s a big leap forward from the previous architecture, and while the transition may require some effort, the long-term benefits—such as improved scalability, clearer separation of concerns, and better dependency management—make it a worthwhile investment for teams looking to future-proof their Backstage setup.

Overview

In this guide, we’ll focus on helping you migrate your existing Backstage backend with minimal disruption. We’ll walk through key steps, such as backing up your current system, creating a bridge for your existing plugins, and gradually integrating them into the new architecture. By the end of this article, you’ll have a clear understanding of how to approach the migration while maintaining your current functionality, and how to unlock the benefits of Backstage’s new backend.

We’ll focus on performing a minimal migration that gets your main backend (index.ts in packages/backend) up and running in the new system. At this stage, we won't migrate all plugins, as that can be a more time-consuming process. Instead, we’ll leverage a temporary bridge function to convert existing plugins into the new system using a transitional environment, allowing you to start benefiting from the new architecture right away.

Preparation for Migration

Before diving into the migration process, it's essential to ensure that your new system will function as expected post-migration. Proper preparation can significantly reduce the risk of issues arising during or after the transition.

Establish a Testing Plan

1. Manual Testing: Develop a comprehensive plan for manually testing your application with the new backend. This should include detailed scenarios that mimic real-world usage of your application. Identify critical paths and functionality that must work seamlessly in the new environment.
2. Integration Tests: In addition to manual testing, it's crucial to have a robust set of integration tests. These tests should cover all functionalities, particularly those related to any custom modifications you've implemented in your existing system. Ensure that your tests validate the expected behavior of all integrated components.

Pay special attention to any custom features or modifications you’ve made. These areas are often the most prone to issues during migration, so thorough testing is essential. And of course it goes without saying, but make sure all your existing tests don’t fail!

By laying the groundwork with a well-defined testing plan—comprising both manual and automated integration tests—you can increase your confidence in the migration process. This preparation will help ensure that your new backend system operates smoothly and consistently after the migration is complete.

Migration Guide from Backstage

Before proceeding with this migration guide, it's essential to familiarize yourself with the resources available from Backstage. The official migration guide on the Backstage website provides valuable insights and best practices for transitioning your backend system. While there may be some overlap with this guide, repetition can reinforce key concepts and ensure that you don't miss critical steps.

Take the time to read through the Backstage migration guide before continuing with this document. Doing so will provide you with a solid framework and context that will aid in your migration efforts, ultimately leading to a smoother transition.

What’s the Plan for Rollout?

To ensure a seamless transition to the new backend, it’s crucial to carefully plan and prepare your rollout strategy. Here are some key considerations to guide your approach:

1. Deployment Strategy:
   • New Image Creation: Consider creating a new Docker image for the updated backend. This allows you to maintain clear versioning and ensures that your deployment environment matches your development and testing environments.
   • Blue-Green Deployment: If feasible, implement a blue-green deployment strategy. This involves running two identical environments (blue and green) where one is live while the other is idle. You can switch traffic between them to minimize downtime and facilitate rollback if necessary.
2. Monitoring and Metrics:
   • Set Up Monitoring Tools: Implement monitoring solutions to track performance metrics, error rates, and resource usage. Tools like Prometheus, Grafana, or your existing APM solutions can provide valuable insights during and after the rollout.
   • Health Checks: Ensure that health checks are in place to quickly identify any issues with the new backend as it goes live.

Migrating Your index.ts to the New Backstage Backend System

In this section, I’ll walk you through the process of migrating your Backstage backend to the new backend system. We’ll focus on updating your index.ts file, which serves as the main entry point for your backend. This migration will allow you to start leveraging the streamlined architecture and dependency injection provided by the new system.

Step 1: Backup Your Existing index.ts

Before starting any migration, it’s essential to create a backup of your current index.ts file. This way, you can reference it or roll back if needed. Let’s save this backup as index.backup.ts.

Additionally, to avoid issues with type checking while you’re in the middle of the migration, add @ts-nocheck at the top of your backup file.

// index.backup.ts
// @ts-nocheck

Step 2: Set Up the New index.ts

Now, create a new index.ts file, which will be the entry point for your new backend system. For this initial step, you can keep it as minimal as possible. The goal is to have a working backend skeleton, which we will build upon later.

Here’s an example of a minimal setup:

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

backend.start();

At this point, your new backend doesn’t do much—it's just an empty shell. But it’s important to verify that the basic setup works before adding any of your legacy plugins.

What’s Next?

In the next part of the migration, we’ll create a temporary legacy environment for your existing plugins and gradually integrate them into the new backend system. This approach allows for a smooth transition, letting you keep the old plugins running while starting to take advantage of the new architecture.

Creating a temporary plugin environment

To ease the migration process, Backstage provides a handy bridge function, makeLegacyPlugin, which helps create a temporary environment compatible with the old backend system. This allows your legacy plugins to continue working while you transition to the new backend architecture. The function ensures all required dependencies are injected into the plugin, simulating the old system’s behavior.

Here's an example of using makeLegacyPlugin:

const legacyPlugin = makeLegacyPlugin(
  {
    logger: coreServices.logger,
    cache: coreServices.cache,
    database: coreServices.database,
    config: coreServices.rootConfig,
    reader: coreServices.urlReader,
    discovery: coreServices.discovery,
    tokenManager: coreServices.tokenManager,
    permissions: coreServices.permissions,
    scheduler: coreServices.scheduler,
    events: eventsServiceRef,
    eventBroker: eventBrokerService,
    auth: coreServices.auth,
    httpAuth: coreServices.httpAuth,
    userInfo: coreServices.userInfo,
    pluginName: coreServices.pluginMetadata,
    identity: coreServices.identity
  },
  {
    logger: log => loggerToWinstonLogger(log),
    cache: cache => cacheToPluginCacheManager(cache),
  },
);

In this setup, your makeLegacyPlugin function acts as a temporary replacement for the older makeCreateEnv function. If makeLegacyPlugin returns the same dependencies as makeCreateEnv, you're set for a smooth transition.

You can also provide type conversion functions within the second parameter to handle any type discrepancies between old and new service structures. For example, converting a logger or cache to the format expected by the new backend system.

Caveats and Considerations

One important note is that all of these dependencies will be instantiated for every plugin that uses the legacyPlugin bridge. This can become problematic, particularly if any dependencies involve heavy operations (e.g., opening database connections). While this bridge is a useful short-term solution, it's highly recommended to fully migrate your backend plugins to the new system as soon as possible.

Using the legacyPlugin

You can add your legacy plugins to the backend using legacyPlugin like this:

backend.add(legacyPlugin('todo', import('./plugins/todo')));
...
backend.start();

By focusing on migrating only your index.ts file initially, you minimize disruption while keeping the migration manageable. This strategy allows you to stay on top of the process while taking advantage of the new backend architecture in stages, ensuring a smooth transition.

Adding new backend system plugins

You can directly add your plugins that are already migrated to the new backend system into your index.ts file.

backend.add(legacyPlugin('todo', import('./plugins/todo')));
...
backend.add(import('@roadiehq/foo-bar')); // a backend plugin in the new system
...
backend.start();

Overriding Core Services

Backstage offers a robust architecture that includes a set of core services added to the application by default. When a plugin or service references a core service, it automatically receives the appropriate instance.

Custom Implementations

If you had custom implementations of certain services in your old system, you will need to override them in the new backend. This is crucial for maintaining functionality and ensuring that your custom logic remains intact.

Recommended Approach

To keep your Backstage directory structure organized and maintainable, I recommend creating a separate package for these service overrides. This approach promotes clarity and separation of concerns within your project.

• Package Naming Convention: In the upstream OSS version of Backstage, this package is commonly referred to as backend-defaults. Adopting this convention in your project will help standardize your codebase and make it easier for others to understand your structure.

Implementation Steps

1. Create the backend-defaults Package:
   • Set up a new package in your Backstage repository specifically for service overrides.
2. Implement Overrides:
   • Define the necessary overrides for the core services that require customization. Ensure that these overrides integrate seamlessly with the existing Backstage architecture.
3. Update Your Application:
   • Modify your application’s configuration to reference the new backend-defaults package, ensuring that your custom implementations are used where needed.

The current core services that backstage provides are the following:

// backstage/backstage/packages/backend-defaults/src/entrypoints
auth
cache 
database
discovery
httpAuth
httpRouter
lifecycle
logger
permissions
rootConfig
rootHealth
rootHttpRouter
rootLifecycle
rootLogger
scheduler
urlReader
userInfo

You can import the core services from the respective path like this:

import {
  RootHttpRouterConfigureContext,
  rootHttpRouterServiceFactory
} from '@backstage/backend-defaults/rootHttpRouter'

Creating Your backend-defaults Package

To effectively manage your service overrides in Backstage, you can create a backend-defaults package using the Backstage CLI. This package will serve as a centralized location for your custom service implementations.

Step-by-Step Instructions

1. Run the Backstage CLI Command: Execute the following command to create a new package:

   npx backstage-cli new
   

2. Select Package Type: When prompted, choose the option for creating a Node library:

   • Node Library: This will allow you to export shared functionality for backend plugins and modules.

3. Set the Package ID: Add the package ID as backend-defaults when prompted. This step ensures that your package is correctly identified within your Backstage setup.

4. Package Creation: Upon completion, this command will generate a new package in your packages directory named backend-defaults.

5. Organizing Your Overrides: To keep your package structured and maintainable, I recommend creating a folder within the backend-defaults package called services. This folder will house your overrides or any new services you implement.

   mkdir packages/backend-defaults/services
   

The following is an implementation of an override of the coreServices.database service. This is a potential fix for cases where your createEnv() and pluginID strings did not match so in your old system the database name does not match the API path.

In the new backend system it will always be the case that your plugin’s ID determines the name of its database and the path it will be attached to. This “fix” is needed because the current upstream Backstage doesn't provide a way to override the database name for these rare edge cases.

import {
  coreServices,
  createServiceFactory,
} from '@backstage/backend-plugin-api';
import { ConfigReader } from '@backstage/config';
import { DatabaseManager } from '@backstage/backend-defaults/database';

export const databaseServiceFactory = createServiceFactory({
  service: coreServices.database,
  deps: {
    config: coreServices.rootConfig,
    lifecycle: coreServices.lifecycle,
    pluginMetadata: coreServices.pluginMetadata,
  },
  async createRootContext({ config }) {
    return config.getOptional('backend.database')
      ? DatabaseManager.fromConfig(config)
      : DatabaseManager.fromConfig(
          new ConfigReader({
            backend: {
              database: { client: 'better-sqlite3', connection: ':memory:' },
            },
          }),
        );
  },
  async factory({ pluginMetadata, lifecycle }, databaseManager) {
    const pluginId = pluginMetadata.getId();
    let databaseName;
    switch (pluginId) {
      case 'foo-bar':
        databaseName = 'foo_bar';
        break;
      case 'tech-insights':
        databaseName = 'tech_insights';
        break;
      default:
        databaseName = pluginId;
    }
    return databaseManager.forPlugin(databaseName, {
      pluginMetadata,
      lifecycle,
    });
  },
});

Configuring the httpRouter service

The backend system comes with its default configured httpRouterService. This is good for basic use cases. It contains multiple middleware and is responsible for the default configuration of the express router.

If the default configuration is not enough, or you made customizations on your router in the old system you can provide your configurations like this:

// index.ts
import {
  RootHttpRouterConfigureContext,
  rootHttpRouterServiceFactory,
} from '@backstage/backend-defaults/rootHttpRouter';
...
backend.add(
  rootHttpRouterServiceFactory({
    configure: (context: RootHttpRouterConfigureContext) => {
      const { app, config, logger, routes, applyDefaults } = context;
			// register your custom middlewares

			applyDefaults() // apply the default middlewares from the backstage core service
    },
  }),
);

Backstage comes with a default request logger. You cannot turn it off and can not be overridden. If you want to replace it (or any of the default middleware) and use your own request logger you have only one choice: don’t apply the default middleware, and instead use each individual middleware as needed, and your custom implementations.

// index.ts
...
backend.add(
  rootHttpRouterServiceFactory({
    configure: (context: RootHttpRouterConfigureContext) => {
      const { app, config, logger, routes, applyDefaults } = context;

      const backstageMiddlewares = BackstageMiddlewareFactory.create({
        config,
        logger,
      });

			// we leave out the backstageMiddlewares.logging() and use our own logger.
      app.use(myCustomRequestLoggingHandler());

			// add rest of the default middlewares
      app.use(backstageMiddlewares.helmet());
      app.use(backstageMiddlewares.cors());
      app.use(backstageMiddlewares.compression());
      app.use(routes);
      app.use(backstageMiddlewares.notFound());
      app.use(backstageMiddlewares.error());
    },
  }),
);

Healthcheck

If you want to override the default healthcheck you can easily do it by attaching a new endpoint for it.

You can create a backend plugin for your healthcheck and then register it in your index.ts file.

Use the backstage-cli to create a new plugin.

// run and select   backend-plugin - A new backend plugin 
npx backstage-cli new

This will create a new backend plugin inside your project under the plugins folder. It will be called the ID that you provided in the prompt.

// plugins/healthcheck-backend
export const healthCheck = createBackendPlugin({
  pluginId: 'healthcheck',

  register(env) {
    env.registerInit({
      deps: {
        rootHttpRouter: coreServices.rootHttpRouter
      },
      init: async ({ rootHttpRouter, rootLifecycle }) => {
        rootHttpRouter.use('/healthcheck', (_, res) => {
          res.json({ status: 'ok' });
        });
      },
    });
  },
});

// packages/backend/src/index.ts
backend.add(import('healthcheck-backend'));

Using the lifecycle hooks

The new backed system provides a core service to hook into the different lifecycles of the process. There is a plugin scoped and a root scoped lifecycle and rootLifeCycle service respectively.

In the old system you might have hooked into the service start promise. In the example below we call the function runOnStartup

  // packages/backend/src/index.backup.ts

  const service = createServiceBuilder(module)
  service
    .start().then((_server: Server) => {
      logger.info(`Startup finished`, {
        uptime: process.uptime(),
      });

      runOnStartup({
        config
      });
    })
    .catch(err => {
      logger.error(
        'The http server threw an unexpected error',
        isNativeError(err) ? err.message : `unknown error raised: ${err}`,
      );
      process.exit(1);
    });

The same functionality can be achieved with the lifecycle hooks in the new backend system.

Create a module for your functionality. The example demonstrates how to add a lifecycle hook to the tech-insights plugin:

export const techInsightsModuleCalculateNew = createBackendModule({
  pluginId: 'tech-insights',
  moduleId: 'calculate-new',
  register(reg) {
    reg.registerInit({
      deps: {
        config: coreServices.rootConfig,
        discovery: coreServices.discovery,
        featureFlagStore: featureFlagStoreServiceRef,
        lifecycle: coreServices.rootLifecycle,
      },
      async init({ config, discovery, featureFlagStore, lifecycle }) {
        const onStartup = () => {
          ...
        };
        lifecycle.addStartupHook(onStartup); // Add the function to be run at startup
      },
    });
  },
});

Testing the New Backend

Testing your migrated backend is a critical component of any upgrade process. Ensuring that your application functions as expected after migration can prevent a host of issues down the line.

Setting Up a Development/Test Environment

It is strongly recommend to establishing a dedicated dev/test environment. This allows you to deploy your new version and conduct tests with minimal traffic interference. Since we are re-architecting the entire Express application and modifying how plugins are integrated, validating that your plugins are accessible is absolutely paramount.

Implementing Health Check Tests

A good starting point for testing is to implement basic health check tests using your existing end-to-end (e2e) or integration testing framework. These tests should verify that each plugin is mounted correctly to its expected path. Here’s an example of how you can do this with Playwright:

import { test, expect } from '@playwright/test';

test.describe('/healthcheck', () => {
  test.describe('GET', () => {
    test('The healthcheck endpoint is configured', async ({ request }) => {
      const result = await request.get(`/healthcheck`);
      expect(result.status()).toBe(200);
    });
  });
});

Reviewing End-to-End Test Coverage

Before migrating, take a moment to review your end-to-end test coverage. Understanding your current testing landscape will help identify gaps, especially given the extensive changes involved in this migration. The more coverage you have over critical services, the better equipped you will be to catch issues early.

Pay special attention to:

• Custom Implementations: Review any middleware, metrics, or logging mechanisms that may impact the migration.
• Company-Specific Plugins: Ensure that any frontend/backend plugins unique to your organization are covered.
• Startup Hooks: Verify that these are functioning correctly post-migration.
• Legacy Functions: Check the usage of custom legacy functions to avoid conflicts with plugin IDs and database names.

Manual Testing

Finally, don't underestimate the importance of manual testing. Take the time to navigate through your application before completing the merge. The migration touches a broad surface area, making it challenging to cover every scenario with automated tests. If you've managed to create comprehensive automated tests—kudos to you! But a thorough manual check can help catch any edge cases that might otherwise be missed.

Conclusion

By following these testing practices—establishing a robust test environment, implementing health checks, reviewing your test coverage, and conducting manual tests—you can significantly reduce the risks associated with your backend migration. This proactive approach will help ensure a smooth transition and a reliable application post-upgrade.

Quirks Encountered in the New Backend System

As we transitioned to the new backend system, we encountered several quirks that are important to highlight, particularly regarding database naming conventions and plugin path correlations.

Database Name and Plugin Path Correlations

In the updated architecture, database creation is directly tied to the plugin ID. While this design should streamline the process, it can lead to issues if there has been a lack of consistency in the previous backend, specifically within the createEnv function.

const createEnv = makeCreateEnv(config);
const todoEnv = useHotMemoize(module, () => createEnv('todo'));

In the legacy system, the database name was simply derived from the parameter passed to the createEnv function. The new backend would generate the databases using the plugin ID or the string you pass to the legacyBackend bridge function and it would use this same string as the API path. This is an issue if your parameter to the createEnv function and the path you attached your plugin to were not the same.

In the new system, the plugin ID takes precedence—it dictates both the database name and the API path.

Key Considerations

• Consistency is Crucial: Ensure that you consistently use kebab-case for plugin identifiers across both the legacy and new systems to avoid unintended database creations.
• No Official Override: Currently, there is no configuration option to override this behavior, making it essential to align naming conventions proactively.

Catching Errors in Plugin Configuration

It's crucial to ensure that any misconfigurations or missing settings are promptly identified. One common issue is that if a plugin configuration is not available, the application may still start up without any visible indications of the problem. This can lead to missed error messages and difficult-to-diagnose issues later on.

Effective Monitoring Strategies

To improve your visibility into potential errors during startup I recommend an approach to run your backend service with output filtering that focuses on error and warning messages. Here’s how you can do this:

1. Start Your Application: Launch your backend application as you normally would.

2. Use Grep for Real-Time Monitoring: Pipe the output of your application to grep to filter for key terms like "error" and "warn". This can be done in a Unix-like terminal using the following command:

   yarn start-backend | grep -E "error|warn"
   
   

   This command allows you to see real-time log messages that could indicate issues with your plugin configuration.

By integrating these practices into your development workflow, you'll significantly enhance your ability to catch and respond to configuration errors on time.

Conclusion

Understanding these quirks is vital for a smooth transition to the new backend system. By ensuring consistent naming conventions, you can mitigate potential issues related to database and plugin management. Catching errors and warnings early can save you a lot of time in the upgrade process. Stay vigilant to avoid complications that could arise from discrepancies in your configurations.

Image by GenerativeStockAI from Pixabay

This article explains many of the customization options available in Roadie today. From look-and-feel customization like theming, to deep data model flexibility, custom scaffolder actions, and completely custom plugins, Roadie has the flexibility you need.

Backstage customizability

Backstage is not a Developer Portal. It’s actually a framework for building developer portals. You can think of it as being more like a Software Development Kit for building your own developer portal. Each company is expected to mix & match and extend the libraries that the Backstage community make available, so that they end up with a completely customized and unique IDP.

This fact is evident from day one of using Backstage. To get started, you don’t download and run a Docker container like you might expect. Instead, you run a command line tool to scaffold your own Backstage instance. Once that’s done, you write your own code to customize it.

-> npx @backstage/create-app@latest
? Enter a name for the app [required]: acme-corp-idp

Creating the app....

Another example of this flexibility is evident in the plugin architecture. The Backstage community has created hundreds of open-source plugins that can be installed into a Backstage-based IDP in order to provide visibility into the many many tools that might be used. Backstage is designed in a way that makes these plugins easy to install and configure.

From authentication hook points to source code management tool integrations to self-service scaffolder templates and UI components…. each and every part of Backstage can be ripped out and replaced and customized to your needs and desires. This is part of the philosophy of Backstage.

Roadie customizability

Despite being built on Backstage, Roadie is a complete, out-of-the-box, developer portal. We take away all of the work of building and maintaining your IDP, so that you can focus on getting value from the tool.

We believe that most organizations shouldn’t need to build a team of 4 or 5 engineers around Backstage. You should want most of your engineering efforts focussed on initiatives that deliver direct customer value, rather than building internal tools from scratch.

This is one of the reasons Roadie is delivered as Software as a Service. We want you to show up on day one and just start using it.

At the same time, we also want to retain the philosophy of Backstage. You need to be able to customize Roadie just as much as you would customize Backstage. You should to be able to connect your own authentication providers, integrate your homegrown tools, and mix and match the plugins that make sense to you. For this reason, we strive to make Roadie as customizable and extensible as we possibly can.

We believe that by combining Roadie’s ease of use with the flexibility and scope of the Backstage ecosystem, Roadie offers the best of both worlds, and is the best IDP on the market.

This post explores many of the major ways you can customize Roadie to meet your needs. Here are your options.

Theming and UI

Your IDP should look and feel familiar to your users, and it should focus their workflows in order to improve efficiency.

Logos and branding

Roadie let’s you replace the logos and branding across the site so that you can expose an IDP that matches your brand and themes.

This image shows the dark-mode version of a theme for a fictional company called BeautyBox.

Sidebar

Roadie’s sidebar configuration lets you cut the navigation down to focus it on the most important use cases in your company. If you don’t need self-service automation capabilities in your IDP then you can easily remove them to streamline things.

We also support reordering of sidebar sections, custom plugins in the sidebar, and deep links to important documentation.

Catalog customization

The Roadie catalog offers control over how the catalog is collected in the first place, and how it is represented to end users who need to consume information in the catalog.

Data model

Many engineering organizations have custom terminologies they use to describe their particular software development life cycle. If your company uses Domain Driven Design, you might want top level concepts like “Value Streams” in your catalog. Other companies might want to support groups of people called “Tribes”. Probably every company will want a list of “Products”.

Roadie supports a flexible and customizable data model that lets you rename existing Kinds and create your own types of entities.

The screenshot below shows:

1. Backstage’s Group kind renamed to “Teams”,
2. A custom entity type called “Products” has been created,
3. Unused kinds like Locations and Domains have been hidden from users.

Table columns

Different IDP users have different jobs, and that means they need different views. Roadie’s tables can be customized on a per user basis, so that everyone can streamline the UI to match the workflows they need.

Users can turn columns on and off, resize columns and change the density of catalog tables. The best part is that these settings are persisted so you can easily dive back into your workflow.

The launch announcement covered the benefits of Roadie’s catalog UI in more depth.

Entity providers

Custom entity providers are a way to provide entities into the catalog from external systems and existing data sources. They’re a critical integration point that helps ensure that your catalog will automatically stay up to date with external systems.

Roadie makes it easy to use custom entity providers via the Roadie Agent. The agent is a library that you can dump entities into in order to have them appear in the catalog.

Here’s a simple example where we use the Roadie Agent to dump an array of hardcoded entity metadata.

const { RoadieAgent, createRoadieAgentEntityProvider } = require('@roadiehq/roadie-agent');

const fakePayload = {
  type: 'full',
  entities: [
    {
      entity: {
        // Standard Backstage entity metadata omitted for brevity
      },
    },
  ],
};

const myEntityProviderHandler = async (emit) => {
  await emit(fakePayload);
}

RoadieAgent.fromConfig()
  .addEntityProvider(
    createRoadieAgentEntityProvider({
      name: 'testprovider',
      handler: myEntityProviderHandler
    }),
  )
  .start();

Auto-discovery settings

Any software metadata in YAML files should be auto-discovered by Roadie so that engineers don’t need to remember to register it via the UI.

Roadie supports custom auto-discovery settings, including glob matching.

Group and User ingestion

Roadie needs an organization chart in order to ensure that ownership of software can correctly be assigned to teams (aka. Groups), and to ensure that scorecard data is correctly rolled up through the org chart.

The best way to make sure the org chart is up-to-date and correct is to automatically sync it from a HR tool. To facilitate this, Roadie supports built in integrations for Microsoft Graph (also known as Microsoft Entry ID, or Azure Active Directory) and Okta. Other solutions can easily push Users and Groups into the catalog via our API.

Plugins

The real power of the Backstage ecosystem comes from its plugins. Roadie supports more than 70 Backstage plugins out of the box at the time of writing (September 2024). It typically takes us about a day to integrate a new open source plugin once it’s created by the community.

Plugin configuration

Most plugins have configuration options that Roadie admins need to be able to configure to suit their particular setup. This is easily done via admin panels we have built into the product.

Take the Argo CD plugin for example. Roadie supports two completely separate ways of integrating with Argo CD:

1. The app locator method allows you to dynamically search and identify Argo CD registered applications from multiple Argo CD instances.
2. The proxy method is used to construct links to individual Argo CD applications.

You can choose the best one for your needs, and even configure minute details like Namespace and Resource allowlists and blocklists.

The APIs of your Argo CD instances are probably not exposed on the public internet, but it’s still easy for Roadie to access them securely. We provide an open-source broker that runs in your infrastructure and makes an outbound connection to Roadie.

For other plugins, like the Kubernetes plugin and EKS, we support AWS Role Sharing as the preferred configuration method. Whatever your connectivity needs, we probably have a solution for it.

Of course Roadie’s responsibilities don’t begin and end with plugin configuration. We’re also:

1. Vetting plugins for quality issues as we integrate them,
2. Scanning plugins for licensing issues (which your legal team will be happy about),
3. Documenting plugins so people know how to use them,
4. Updating plugins as their authors release new features.

Frontend plugins

If there isn’t a community created plugin that suits your needs, or you need a plugin for a homegrown tool that only exists in your company, Roadie can support that too.

Users can write native Backstage plugins that run on Roadie and securely connect back to your private internal APIs via the broker.

The plugin development workflow supports the ability to locally host custom plugins and run them right on Roadie during development. This basically means that you can make a change to your plugin code in your IDE, and simply refresh Roadie to see your changes live.

We also support running production and development versions of the same custom plugin in Roadie at the same time, so you can iterate on your plugins without disrupting your users.

Productionizing a plugin is as simple as running the Roadie CLI against it to package it up and push it to Roadie.

roadie plugin:build --location /my-custom-plugin-folder/ --host https://static-assets.roadie.so/<my-tenant>/myCustomPlugin

The full range of Backstage frontend APIs are available to your custom plugins. You can query the catalog, check Role Based Access Control permissions, and even push analytics to see how people are using your plugin.

Proxies

Plugins frequently make use of proxies to upgrade requests with token authentication before forwarding them on to a backend service to retrieve data. Roadie users can create their own proxies inside the application. These proxies can be used to connect to third-party SaaS APIs like PagerDuty, or private internal APIs.

Here’s a user created proxy for SonarCloud.

Entity Pages

Entity pages are the pages in the catalog which pull together information on a particular piece of software, a team, or an infrastructure resource.

UI layouts

Adding plugins to Roadie interfaces is a simple matter of picking them off a list. This process is the same for both community created plugins (including plugins created by Roadie, plugins created by Spotify, and official plugins from companies like PagerDuty and Snyk) and for custom plugins you create yourself.

Once you’ve chosen your plugin, simply drag-and-drop it on the page, and resize it to suit your needs.

Different types of entities need different layouts. The Backstage Lighthouse plugin makes sense for a website, but not for a Kubernetes cluster. Roadie organizes the plugins in this way so that it’s easy to create custom views without too much setup. Our built in role based access control ensures that admins have the power to set up layouts, without overloading regular users with too many options. Team pages are customizable in the same way, they just typically have different cards and widgets.

Props on plugins

Sometimes individual cards will have their own editable properties. Roadie supports that too. For example, this card displays a list of recent GitHub Actions CI runs and their status. But how many recent CI runs should be displayed, and what layout should be used? Well, you can set that with the editable props.

This screenshot shows the edit mode of the Dynatrace Backstage plugin.

Metadata cards

We’re aware that many platform teams wish to display custom information in Roadie, but don’t necessarily have the TypeScript skills on hand to create custom plugins from scratch. To make this easier, Roadie includes no-code cards that can display information from custom metadata on the Entity.

Imagine we have a Backstage entity with some custom properties in its metadata.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: sample-service
spec:
  type: service
  owner: group:roadiehq/engineering
  lifecycle: production
  custom:
    engineeringManager: user:dtuite
    productManager: user:samnixon87

How would we display this in the catalog? On Roadie, it’s as simple as adding the EntityMetadataCard and configuring it in a couple of clicks. The card even auto-links the Engineering Manager and Product Manager to their User pages in the catalog.

Home page

Of course it’s not just pages in the software catalog that can be customized. Roadie users can also customize the Home page plugin to display key information that’s relevant to them. This is the place to display your open pull requests, community news, your calendar, and other useful info that’s tailored to the logged in user’s experience.

Scaffolder

Roadie scaffolder supports the full range of customization that you would expect from the Backstage scaffolder.

Actions

Custom scaffolder actions are a way to execute homegrown CLI’s or arbitrary code as part of scaffolder templates. These are executed within your own infrastructure, which provides an added benefit in that it makes it easy to send internal network requests.

Users can use the Roadie Agent (the same library used in the Custom Entity Providers section above) to register custom scaffolder actions with Roadie.

RoadieAgent.fromConfig(config)
  .addScaffolderAction(
    createRoadieAgentScaffolderAction({
      name: 'custom-action', // The name of the action as defined in Roadie
      handler: async (ctx) => {
        try {
          fs.writeFileSync(
            `${ctx.workspacePath}/test.txt`,
            'new file with new contents',
          ); 
          // Writing a new file into the shared workspace
        } catch (err) {
          console.error(err);  // Local logging on the Roadie Agent process
        }

        let count = 0;
        while (count < 5) {  // Additional other actions that is wanted to be taken. This time looping for 5 seconds
          await new Promise((resolve) => setTimeout(resolve, 1000));
          count++;
          await ctx.log(`hello world`); // Sending a log message to be displayed to the end user
        } 
      },
    }),
  )
  // Add a second custom scaffolder action
  // .addScaffolderAction(...) 
  .start();

These can then be used in your templates like this:

steps:

  # This step executes on Roadie
  - id: fetchTemplate
    name: Fetch file
    action: fetch:template
    input:
      url: ./skeleton

  # This sends the workspace context to a self-hosted scaffolder runner
  # and executes the custom action we defined above. The payload passes
  # values to the self-hosted scaffolder runner and logs are written
  # back to Roadie.
  - id: invokeCustomAction
    name: Invoke a custom self-hosted action
    action: roadie:agent
    input:
      action: custom-action
      shareWorkspace: true
      payload:
        name: someValue

Field extensions

Custom field extensions give you the ability to build your own inputs and UI components for the scaffolder. You write them using the normal Backstage API, and push them into Roadie using the custom plugins pipeline.

Once registered, they can be used in Scaffolder templates by matching the name of the ui:field prop.

parameters:
  - title: Fill in some steps
    required:
      - name
    properties:
      name:
        title: Name
        type: string
        description: My custom name for the component
        ui:field: MyCustomInput

Authentication & Authorization

Single-Sign-On (SSO) setups

Roadie provides native support for any SSO provider you can imagine. Okta, Microsoft Entra ID, Google, AWS, Ping Identity…. you name it, we’re probably already using it in production.

Setting this up is a simple matter of sending our docs to your IT team, or whoever controls your SSO setup. We will then work directly with that team to get your authentication provider configured.

Once set up, your IT team can grant and revoke access to Roadie without being blocked by us. They have full control over the process.

Roles

Customers who have purchased our role-based access control add-on can define custom roles which can then be assigned to groups of users.

In this example, you can create a role which can only read the catalog. It wouldn’t be able to execute scaffolder templates or register anything in the catalog.

Roles can also be assigned via your Identity Provider like Okta or Microsoft Entra ID. This is especially useful when your IT team wishes to manage access and roles in a single place.

Permissions/policies (coming soon)

When role-based access control gets really powerful is with the ability to define custom permissions which can then be combined in roles. For example, perhaps you want to hide Component entities which are tagged with sensitive from everyone except the security team.

We’re working on this ability at Roadie and expect it to roll out in 2024.

Tech Insights (Scorecards)

Custom data sources

Data Sources are recurring jobs which gather data that can then be used to write checks against.

For example, Roadie comes with a built-in GitHub Settings Data Source which records facts about each repository in the catalog, like whether or not branch protection is turned on, or whether or not force pushes are allowed.

Users can also create their own Data Sources from scratch. At the time of writing, 7 different types of Data Sources are supported. Here are some examples:

1. HTTP Data Sources hit third-party APIs and pick values from the JSON response
2. Component Repository File Data Sources inspect files in the repository associated with each Component and record values from them.
3. Push Based Data Sources receive webhook events and store them for processing.

Creating Data Sources is a simple matter of filling out a few fields. You don’t need to write any TypeScript or YAML. Data Sources support advanced features like metadata variables, JSONata processing, inline testing, and error handling.

Custom checks

Checks inspect a value created by a Data Source and give it a pass or fail for each entity it applies to.

This check ensures there is a README.md in the repository associated with each piece of software.

Advanced features like boolean logic and live testing of the check are supported. Checks can link to documentation when they fail, or even link to a scaffolder template that can help the owner of a service to fix the failure.

Custom scorecards

Users can create their own scorecards that apply to the software in the catalog in order to communicate best practices and expectations to teams.

Each scorecard is a user defined collection of checks that shows up in reporting, in the catalog, and on team pages. Scorecard data is rolled-up through the catalog so that you can dip in at any level of the org chart and see aggregated data from below that point.

And there’s more coming

We’re constantly adding to the ways you can customize Roadie. If you want to stay up to date with the latest customizations, subscribe to our newsletter to hear what’s coming.

Image by GrumpyBeere from Pixabay

Cloud computing - the double-edged sword

Managing cloud costs is fast becoming a strategic priority. While cloud service providers (CSPs) like Google Cloud, Microsoft Azure, and Amazon Web Services offer incredible flexibility and scalability, they can quickly lead to ballooning costs if left unchecked. We’ve all seenthecloudbilling horror stories, but even for companies that have a good grip on their cloud costs, it’s easy to get into a situation where cost growth begins to outstrip associated revenue growth.

Such a position can undermine profitability, growth, and operational flexibility, ultimately putting the organization on an unsustainable trajectory. Beyond simple expense, runaway costs can mask inefficiencies in architecture, resource allocation, and service usage, creating a vicious cycle of wasted consumption. Left unmanaged, this can undermine even the most successful products, ultimately affecting their long term viability.

A notable case in point is Spotify, who encountered this scenario early on which forced the company to rethink its approach to managing cloud costs. Spotify realized that to get a handle on this, they needed to empower their engineers - the people closest to the actual cloud usage - to own the costs and be responsible for optimizing them. Rather than a top-down approach mandating a cost reduction, Spotify opted to make the engineers themselves responsible for their own cloud costs, an excellent example of the phenomenon of shifting cloud costs left.

Shifting cost awareness left: a cultural change

Shifting cost awareness left is a concept in effective cloud cost management that is gaining traction. Essentially a form of embedding cost considerations earlier in the engineering process, not as an afterthought for finance teams, that’s exactly what Spotify did, according to James Governor at RedMonk: “Spotify engineering teams are used to having a lot of autonomy, so the company couldn’t just introduce new cost guardrails as a top down concern. Therefore the Cost team tried to foster a culture where optimization would be fun.”

Shifting cost awareness left isn’t just about giving engineers tools, it’s a cultural shift where engineers take ownership of the financial impact of their work and are invested in cost optimization. When engineers are empowered with real-time cost insights, they become agents of change. Instead of waiting for an end-of-month cloud bill to identify costly inefficiencies, engineers can make informed decisions in real-time. This shifts cost management left from a reactive process driven by finance to a proactive, engineering-led initiative.

There’s a philosophical value here that goes beyond dollars and cents. By making cost awareness a core part of the development lifecycle, companies can drive a sense of ownership and even healthy competition among teams. Engineers begin to actively look for ways to optimize their cloud usage, often competing with each other to drive down costs. This culture of cost ownership not only saves money but also leads to better architecture and more efficient systems overall.

According to Janisa Anandamohan, Spotify Senior Product Manager, Cost Engineering:

We know engineers are natural optimizers when it comes to reliability, security, performance, et cetera. And now we’re telling them, hey, add costs into the mix. And they were super excited about that. We had many teams that were just able to tweak their services and data pipelines and to make them more efficient. And we know efficiency doesn’t just help cost, but helps reliability and performance as well. So we were getting double, triple wins along the way.

Spotify’s solution: Cost Insights

Spotify’s approach to shifting managing cloud costs left took the form of a cost optimization and visualization plugin called Cost Insights, on their own internal version of Backstage. The concept is simple: provide engineers with the tools to visualize and manage the costs associated with the services they build, all within the internal platform they are already using - their internal developer portal. By tying cloud costs to specific entities in the Backstage catalog, teams are empowered to take control of their spending and optimize resources more effectively.

This approach worked well for Spotify, partly due to their internal engineering culture, but mostly as a result of the significant engineering effort invested into getting Cost Insights integrated into their developer platform. While they have since open-sourced a pared-down version of the Cost Insights plugin back to the community, most organizations attempting to replicate their success will find it challenging, even when using their plugin. This is largely because of the technical complexity required, and because the current Spotify Cost Insights plugin, like many of the other aspects of the Backstage framework, is far from an out-of-the-box solution.

Just how much work would it take an engineering team, even working with the open-sourced Cost Insights plugin to implement a working solution? It’s a significant lift; here’s what they’d have to do:

1. Cloud billing integration: Set up a mechanism such as an API to pull cost data. This involves not only access configuration but also understanding the data schema of your cloud provider, which can be complex and time-consuming.
2. Backend development: Develop a backend to fetch, process, and store the billing data. This step requires designing a database schema that can handle potentially large amounts of data efficiently and setting up a server to run this service.
3. Cost Insights API implementation: Implement the API endpoints required by the Cost Insights plugin. This includes endpoints for fetching cost data, projecting future costs based on current trends, and breaking down costs by services, projects, or departments.
4. Frontend integration: Integrate the Cost Insights plugin into your Backstage instance. This may involve customizing the plugin to fit into your organization’s Backstage environment and ensuring it interacts correctly with your newly developed backend.
5. Testing and validation: Thoroughly test the plugin with real data to ensure accuracy. Validate the cost projections and insights with your finance team to ensure they align with actual expenditures.
6. Maintenance and updates: Continuously update the backend and frontend as cloud providers change their APIs or pricing models, and as new features or fixes become available in the Cost Insights plugin.

For a small team of three to four engineers, this could take anything from several months to a year to fully implement. This is a big lift for most organizations, which means that for many, this complexity keeps the solution remains out of reach. While the concept of cost transparency and shifting cost awareness left resonates, the time and effort required to implement and maintain a working Cost Insights tool deters widespread adoption.

As if it wasn’t hard enough, all of the work above assumes an organization is using only a single CSP in their stack. In reality, any organization that is using multiple CSPs (say, AWS and GCP) faces a thorny additional hurdle: the lack of data homogenization and normalization from CSPs. Each CSP often presents cost data in a different format, making it extremely challenging for organizations to consolidate and interpret this data in a unified manner if they’re using multiple CSPs.

Fortunately, the recent introduction of the FOCUS (FinOps Open Cost and Usage Specification) standard has changed the landscape for the better. Developed as a collaborative effort by the FinOps Foundation, the FOCUS standard aims to normalize the cost and usage data across different CSPs, providing a common format that makes it easier for organizations to integrate and analyze their cloud spending. This standardization is a crucial enabler, simplifying the data integration process and reducing the overhead associated with translating disparate data formats.

Introducing: Roadie’s Cost Insights Plugin

The adoption of the FOCUS standard significantly simplifies the integration of cost data across various cloud platforms. However, the complexity and effort of setting up Cost Insights is still high - which is where we at Roadie saw an opportunity to help. We recognized that the idea of empowering engineers to manage cloud costs was spot on, but the solution needed to be simpler, more accessible, and ready to use out of the box. As such, we’re in the process of refining and enhancing the existing Cost Insights plugin, making it significantly easier to deploy and use right out of the box.

Our enhanced Cost Insights plugin builds on Spotify’s version, but aims to reduces the setup friction, allowing engineering teams to access powerful cloud cost insights immediately - no custom backend development required. This streamlined approach means engineers can spend more time optimizing their services and less time managing the infrastructure for cost tracking.

With Roadie’s plugin, teams can quickly see which services are driving up their cloud costs, how those costs have trended over time, and what actions can be taken to reduce unnecessary spending. The key here is not just providing data but making it accessible and actionable, so engineers can immediately use it to make decisions.

Roadie’s Cost Insights plugin takes full advantage of the FOCUS standard, ensuring that the data we present is both accurate and comparable across providers. By eliminating discrepancies in how cost data is reported, the plugin enables engineers to gain a clear understanding of their cloud usage without needing to reconcile different data formats.

The ability to assign costs to specific entities within a Backstage catalog is what we’re most excited by. In traditional cloud billing tools, costs are often presented at a very high level, making it hard to drill down and see what’s truly driving the expenses. By associating costs with individual services and the teams responsible for them, the plugin makes cloud costs real and relatable for developers. When a team sees exactly how much their service is costing, they can no longer ignore it or avoid responsibility - it becomes part of their job to optimize those costs.

The Roadie Cost Insights dashboard displaying cloud cost by product

How it works

Our Cost Insights plugin, currently in internal beta, integrates seamlessly with a Roadie-managed Backstage instance to provide teams with an out-of-the-box solution for tracking and managing cloud costs. Key features include:

• Easy setup: No complicated setup required - simply connect to your cloud service provider to Roadie Cost Insights via your cloud administration settings or via a secure broker, and allow the Roadie Cost Insights compatibility layer to take care of all the data modeling and translation.
• Project and group-based cost tracking: Track cloud costs at the entity, team, or product level, and drill down into specific projects or groups for more detailed insights.
• Trend visualization: View cloud cost trends over time, breaking down data by dimensions like products, services, and regions to identify patterns and anomalies.
• FOCUS standard integration: The plugin leverages customer-provided FOCUS data, ensuring consistency and like-for-like comparability across cloud providers.
• Actionable insights: Dashboards help engineers take ownership of their services, and immediate action by optimizing those services or reducing over-provisioned resources.

For example, a team might notice that the costs of a particular product are rising faster than expected. With Roadie’s plugin, they can drill down into the cost breakdowns, identify high-cost services, and take corrective action - such as optimizing resource allocations or reducing usage.

Roadie Cost Insights also supports multiple regions, meaning a DevOps team managing multiple cloud services in different regions could use Roadie’s Cost Insights to track cloud costs by region. For instance, they could spot that their compute resources in one region are significantly more expensive compared to others. Using the plugin’s trend visualization and breakdown capabilities, the team can identify the services or instances driving up the cost and adjust their architecture to either reduce or move resources to more cost-effective regions.

Roadie Cost Insights Architecture - note that the user configuration overhead is limited to installing the Cost Insights client on the cloud service provider infrastructure (or the use of a broker client) while Roadie takes care of all the backend setup.

The future of Roadie’s Cost Insights Plugin

We believe that by simplifying cost management and putting it directly in the hands of engineers, companies can not only save money but also drive innovation, efficiency, and alignment across their teams.

Cloud cost management is no longer just a financial issue—it’s an engineering one. By providing real-time insights into cloud spending and shifting cost awareness left, Roadie’s Cost Insights plugin helps companies empower their engineering teams to take ownership of their cloud usage. This leads to smarter decisions, lower costs, and more efficient systems.

While Cost Insights is still currently in an internal beta, if you’re interested in learning more or becoming an early design partner, get in touch with us today. We’d love to work together to bring the future of cloud cost management to your team.

Title image by Brian Penny from Pixabay

Without a complete catalog, an IDP cannot fulfill it's primary purpose: improving developer productivity. It cannot drive improved discoverability if the software that teams are trying to discover is not in there. It cannot help to measure the standardization or security posture of software it does not know about.

This article is a comprehensive guide to catalog completeness in Backstage. Roadie is built on top of Backstage, and so all of the same lessons apply. We cover why it's important, how to measure it, and how to achieve it.

First, let's learn why you want to build a complete catalog in the first place.

What is a software catalog?

A software catalog is a centralized registry that tracks all the software assets, services, APIs, and resources within an organization. Think of it as a searchable directory that answers fundamental questions: What software do we have? Who owns it? Where does it run? How do systems connect to each other?

Unlike a traditional service catalog (which typically focuses on IT services available to end users), a software catalog is developer-focused. It tracks the technical components that engineering teams build and maintain: microservices, libraries, data pipelines, APIs, and infrastructure resources. A service catalog might list "Email Service" as something employees can request; a software catalog tracks the underlying email-gateway microservice, its API endpoints, the team that owns it, and its dependencies.

Modern software catalogs go beyond simple inventory management. They serve as the foundation for improving software discoverability across development teams, enabling compliance monitoring at scale, and providing secure management of software assets. Platforms like Roadie and Backstage combine software catalog functionality with additional capabilities like scaffolding templates, technical documentation, and scorecards that measure software maturity.

Why building a complete catalog is important

The primary reason companies deploy an IDP is because they want to make developers more efficient. A big part of this is helping teams answer basic questions about the software around them. They want to make it easy to answer questions like "do we have a geocoding service?", "which team owns the checkout service?" and "where is the API spec for the users service?". These questions can only be answered by the IDP if the geocoding service, checkout service, and users service are registered there in the first place.

A complete catalog is essential when using an IDP to measure the maturity of the software that teams are producing. Applying scorecards to software in the catalog of the IDP can be a great way to determine the security posture of each production service. But scorecards can only apply to software which is actually in the catalog.

The software catalog is also the fundamental unit of navigation in the IDP. The lack of a software catalog is the reason that wikis like Confluence or Notion cannot solve the discoverability problem that IDPs solve. In a wiki, content is organized into pages, which are intentionally unstructured and flexible. In an IDP, content is organized by Component (think "piece of software") and is structured so that the same information is available for each Component.

How software catalogs improve discoverability and compliance

A well-implemented software catalog addresses two persistent challenges that enterprise development teams face: finding existing software assets and ensuring those assets meet organizational standards.

Discoverability across development teams

When organizations scale beyond a handful of engineering teams, tribal knowledge breaks down. New engineers don't know what services already exist. Teams duplicate functionality because they can't find existing solutions. The software catalog solves this by providing a single, searchable interface where developers can discover services, APIs, libraries, and resources across the entire organization.

Effective discoverability requires more than a simple list. The catalog needs to track relationships between components (which services depend on which APIs), ownership information (who to contact when something breaks), and metadata like documentation links, deployment environments, and health status. Internal developer portals like Roadie provide customizable features for organizing software services and APIs, making it easy for teams to find what they need.

Compliance monitoring and access control

For mid-sized tech companies and enterprises, compliance monitoring at scale is difficult without a complete software catalog. You can't measure what you don't track. A software catalog enables automated compliance checks: Does every production service have an owner? Are all APIs documented? Do all services meet security baseline requirements?

Roadie's Tech Insights feature builds on the software catalog to provide scorecards that measure software against organizational standards. Teams can track metrics like test coverage, dependency freshness, and security compliance across hundreds of services. This transforms compliance from a manual audit process into continuous, automated monitoring.

Access control also benefits from catalog completeness. When every service has a defined owner and clear metadata, organizations can implement fine-grained permissions for who can view, modify, or deploy each component. This is particularly valuable for managing software assets securely across distributed teams.

How to measure catalog completeness

Having a complete catalog doesn't necessarily mean that every piece of software is in the catalog. Most organizations have their share of abandoned code. It may have been created during hackathons or for brief experiments that never saw production usage. Having all of this stuff in the catalog can create clutter and noise.

There may also be parts of the organization for whom it doesn't make sense to be in the catalog. We work with companies which have large hardware divisions who write code for embedded devices. They don't necessarily follow the DevOps lifecycle and thus are sometimes intentionally omitted.

Production software is the most important stuff to have in the catalog. It's the software that most engineers work with most of the time. Production software will have the most frequently referenced APIs and docs and it's much more important to have an understanding of the maturity of the software that runs in production environments, since it has the most attack vectors.

For this reason, we frequently see customers create a list of software which is deployed to production by referencing ArgoCD or some other deployment tool. They then compare this against software in the catalog, frequently by accessing the Roadie APIs or by using our CSV export functions.

The next best bet is to compare number of pieces of software (aka. Components) in the catalog against the number of active, unarchived repositories in your source code solution. This will never give you a perfect answer, because "pieces of software" don't necessarily map perfectly one-to-one to repositories (monorepos etc), but it's a good start.

How Roadie Helps

At Roadie we give every customer a chart which shows the percentage of their active (non-archived and received a commit in the past 12 months) repositories against the number of Components in the catalog.

Roadie customers can achieve high catalog completeness

Achieving a high level of catalog completeness is definitely possible.

The majority of established Roadie customers are happy with their level of catalog completeness and we have many customers who have a catalog completeness level (measured as a percentage of active repositories which are registered in the catalog) which is above 80%.

Here's a chart showing catalog completeness for Uplight who onboarded in September 2023. Four months later they were at 88% catalog completeness, with more than 600 components in their catalog.

Here's another Roadie customer who increased their catalog completeness from 40% to 90% over a period of four months. They now have more than 750 components registered.

There are countless more examples like this amongst Roadie customers. The goal of this post is to help all companies achieve the same results.

Strategies for building a complete catalog

There are multiple strategies that can be used to build a complete catalog. The strategy that is right for your company can depend on factors like the size of the company and the enthusiasm of the developers to have an IDP.

You may also want to run multiple strategies in parallel, or start with one in order to get the bulk of software into the catalog, and switch to another to get closer to full catalog completeness.

Think about expanding catalog completeness like the layers of an onion. Start with the most enthusiastic early adopters and get them onboard. Then use them as an example as you expand out to the rest of the organization.

We recommend you import users (aka. employees) and teams (called Groups in Backstage nomenclature) into the catalog before creating any software components. You will ideally want to assign ownership of each component to a team as you import it. This task is much easier if all of your teams already exist in the catalog.

If you don't know which strategy to choose…

Experiment! You don't need to roll out to the whole organization in one go. Pick some friendly teams, run one strategy on each team, and analyze the results. Did you end up with software in the catalog? What is the feedback from each team? Where did they get stuck? Take this feedback and use it to improve the process before expanding out to more teams.

Strategy 1: Centralized automated

This strategy involves connecting to a chokepoint in the organization in order to push software metadata into the catalog programatically. It does not require anyone to write catalog-info.yaml files.

Frequently, the initial chokepoint will not have all of the information required to build a useful catalog. When this happens, the software metadata must be enriched after the fact.

Depending on your tech stack and permission settings, following options may be viable:

1. An ArgoCD instance which does deployments to the production environment. It has knowledge of the most critical software in the org (software that goes to production) and can thus can be a good starting point for the catalog.
2. A Helm chart which is used by a large percentage of the deployable software in the organization.
3. A centrally owned CI job or build tool which can be written by the centralized team and applied to software which is owned by other teams in the organization. For example, Lunar Bank have talked about how they populate their catalog from their build tool called shuttle.
4. A legacy software catalog, developer portal or spreadsheet.

Pros

1. The software catalog can quite quickly be bootstrapped to a highly complete state.

Cons

1. The ownership link between software and teams is not immediately established.
2. Companies with no source of truth or a heavily fractured deployment ecosystem may not be able to implement this strategy.
3. The product teams will be less well educated on the value of the IDP when the process is finished.

Tactics to prioritize in order to succeed with this strategy

1. Use custom entity providers
2. Make catalog presence a requirement for deployment

Strategy 2: Centralized manual

This strategy tries to avoid asking the individual product teams to do work. Instead, the centralized team takes it upon themselves to populate the catalog. They may do this in collaboration with the product teams, but they likely won't ask them to write any YAML.

Pros

1. This strategy is likely to be faster than the distributed manual strategy, especially for companies which don't have thousands of microservices.

Cons

1. The product teams will be less well educated on the value of the IDP when the process is finished.
2. It may be difficult for the centralized team to gather enough data about each individual software component.
3. It becomes the centralized teams job to manually keep the catalog up to date.

Tactics to prioritize in order to succeed with this strategy

1. Store software metadata in a single repository
2. Open scripted pull requests
3. Customize the catalog nomenclature

Strategy 3: Distributed manual

This strategy involves asking all of the individual product teams to register the software that they own in the catalog.

Typically, the central team who own the IDP will meet with and educate the product teams on the value of the IDP, either on an individual basis, or in larger groups. The central team will provide tools and materials to the product teams in order to teach them what they need to do to get their software into the catalog (typically create a catalog-info.yaml file), and give them clear steps to take in order to achieve catalog completeness.

Pros

1. Product teams are more likely to feel a sense of ownership over their software in the catalog because they put it there in the first place.
2. Product teams have an opportunity to learn about features of the IDP as they are registering their software. They may then choose to use features in the IDP such as technical documentation.

Cons

1. This strategy requires a lot of work from the central team in order to yield high catalog completeness. It will take time. They will need to educate and continually follow up with product teams throughout the company.

Tactics to prioritize in order to succeed with this strategy

1. Give teams a scaffolder template to register software
2. Share catalog completeness numbers publicly
3. Tie catalog completeness to a wider initiative
4. Write custom plugins to create value

Tactics for building a complete catalog

The tactics you need to use will depend on the strategy you are implementing. This is a full list of all the tactics we know. Please refer to the strategies above in order to know which tactics to choose.

Some tactics will apply regardless of the strategy that is chosen. They are:

1. Customize the catalog taxonomy
2. Move onboarding docs into Roadie
3. Present on the value of Roadie to people managers
4. Tie catalog completeness to a wider initiative

Each of the tactics mostly fall into one of the following categories:

1. Reduce friction for developers who want to get their software into the catalog.
2. Create incentive for developers to put their software into the catalog.
3. Educate developers on how to use the catalog.

Give teams a scaffolder template to register their software

Category: Reduce friction

This involves writing a scaffolder template that teams can use inside Backstage in order to create a catalog-info.yaml file in their repositories.

The scaffolder template will ask the user to fill out some information about their software, typically by picking from pre-defined values, and will open a pull request against a repository when finished. Once the user reviews and merges the pull request, auto-discovery will pick up the catalog-info.yaml file and populate the catalog.

Benefits

1. Individual developers don't need to understand the (long) Backstage YAML API spec in detail.
2. The owners of the IDP can use the template to constrain the "type", "lifecycle" and other properties that is assigned to each software component. This puts guardrails in place that will help create more consistency in the catalog. Catalog consistency is important, and will help you avoid problems in future.
3. The form can integrate with external APIs to pull in sensible options. For example, in the screenshot above, the "Component Owner" is a selection of all the engineering teams in the company. The user doesn't need to type an exact string.

How Roadie helps

Roadie provides a starting point for a software registration template in the getting-started repo. Customers can fork it into their own GitHub org, edit it to meet their needs and import it into their own Backstage instance.

Archive unused repositories

Category: Reduce friction

If our measure of catalog completeness is:

then we can increase catalog completeness by archiving old repositories that nobody is using.

It may sound somewhat silly, but every organization has abandoned hackathon projects and old test repos that are unused and simply causing clutter. By archiving them, we clean up our source code management tool while improving catalog completeness.

Believe it or not, Roadie has one customer who increased catalog completeness from 45% to 75% just by archiving repositories.

How Roadie helps

Our catalog implementation ensures that Components are removed from the catalog when the repo they reference is archived.

Customize the catalog nomenclature

Category: Reduce friction

By mapping the Kinds of entity that show up in the catalog to familiar concepts in your company, you can create instant recognition for developers who land in the catalog.

For example, if your company has a concept of "Valuestream" then make this front and center in the catalog so that users instantly understand what they're looking at.

Benefits

1. Users can orient themselves quickly and get value rapidly.

How Roadie helps

Roadie customers can use our admin interfaces to customize and rename the core catalog concepts, and create completely new ones.

Write custom plugins to create value

Category: Create incentive

Custom plugins provide value for product teams by giving them faster ways to perform bespoke workflows inside the catalog. By creating custom plugins, a central team can incentivize developers to add their software into the component.

For example, Lunar Bank have custom plugins for dealing with dead letters in RabbitMQ. These plugins are regularly useful for developers. This causes them to visit the catalog to use the plugin.

Benefits

1. Custom plugins are quite simple to produce and can quickly create value for software engineers.
2. Custom plugins unlock tailored value for engineers to help them do things more quickly or more easily than they otherwise could. They sometimes even allow them to perform a task that they cannot do at all outside of Roadie or Backstage.

How Roadie Helps

Roadie provides an interface for registering and managing custom plugins. It also facilitates live reloading of custom plugins, the ability to run multiple versions of a plugin side by side, and a scaffolder template to bootstrap a custom plugin monorepo. Custom plugins on Roadie can securely connect back to a private network to load data from internal APIs. Check out our docs to learn more.

Move onboarding docs into Roadie

Category: Educate

Engineer onboarding docs are typically used to help a new engineer to set up their environment and get to productivity quickly. Expedia Group, Spotify and other Backstage adopters have successfully used TechDocs and scaffolder templates to speed up engineer onboarding and help new engineers become familiar with the IDP on day one. The exact same tactic can be deployed on Roadie.

Expedia Group put 850+ engineers through their Backstage based bootcamp in 2022. They discuss it in their case study on the Backstage website.

Benefits

1. Newly onboarded engineers start using Roadie on day one. They get used to it and understand how to come back.
2. Engineers learn how to use a scaffolder template to create a new service during onboarding. This service is added to the catalog, and they learn how the catalog works.

How Roadie Helps

Roadie supports standalone TechDocs that are not tied to a particular software component in the catalog. These are perfect for onboarding docs.

Present on the value of Roadie to people managers

Category: Educate

Roadie provides specific value to managers, directors and VPs that is different than the value that developers might care about. By educating managers on the value they will receive, you can encourage them to work with their teams to get their software into the catalog.

For example, did you know that frequent Backstage users at Spotify are 5% more likely to be at the company one year later. Retention is important for managers, so they need to know about this.

How Roadie Helps

1. Roadie provides Scorecards which can help managers ensure that their teams are producing mature and high quality software. This feature is not available for open-source Backstage.
2. Roadie gives customers value calculators to help them estimate the dollar value they can expect to receive.

Open scripted pull requests

Category: Reduce friction

Opening an automated pull request containing a catalog-info.yaml file is a good way to ease the burden on developers who want to get their software into the catalog. All they need to do is edit the pull request a little bit, review it and merge it.

Keep in mind that your script will need permissions to open a pull request against a majority of repositories in your source code management tool. Depending on your security model, this may not be possible.

This method can work especially well in companies that operate out of large monorepos. A monorepo setup allows the generation of a single pull request that can add many catalog-info.yaml files in one go. It can also be reviewed and merged by a single person with elevated permissions.

While this is a tempting option to quickly build a catalog with YAML files, we have seen customers experience issues with catalog correctness when they use this method. Some teams may blindly merge the pull request without validating the information that it contains. This tactic should be executed alongside an education program to teach teams what to do. Go slowly and experiment.

How Roadie Helps

Roadie provides a token authenticated API which the centralized team can use to tell which repositories are already in the catalog. A simple script can consume this to open a PR against the repos which are not already accounted for.

Our solutions engineering team can work with you to write a simple script that will open a pull request containing a YAML file into each repository.

Make catalog presence a requirement for deployment

Category: Create incentive

By making catalog presence mandatory for deployment, platform teams can be confident that they have all of the important software in the catalog.

In the early days of Backstage at Spotify, teams could not SSH into their machines unless their services were in the catalog. The catalog owner was referenced to determine who was and was not allowed to access the machine.

This tactic works best when the IDP is orchestrating a new greenfield platform that other teams are migrating onto. Adding the catalog-info.yaml file can be one simple step in what is likely a series of steps that teams have to do to migrate. Outside of this situation, it can be politically problematic to block deployments due to a missing YAML file.

Automate catalog collection with custom entity providers

Category: Reduce friction

If developers find writing YAML files tedious, potentially the best thing to do is to make them optional. Backstage's custom entity providers allow adopters to programmatically shovel software entries into the catalog. Custom entity providers are a great way to connect Backstage to an existing source of truth for catalog data, such as a legacy IDP, an ArgoCD instance, a kubernetes cluster, or a CICD tool.

How Roadie helps

Roadie gives customers the roadie-agent. This wraps the custom entity provider concept with a secure connection to Roadie so that customers can easily dump software metadata into the agent and have it appear in the catalog.

Roadie has an Entity Provider API. Simply push an array of software metadata to this endpoint and it will appear in the catalog. To update the metadata, simply push again.

Frequently the programatic source of truth will have some but not all of the metadata that the catalog needs. For example, it may be missing the name of the team who owns the software. Roadie allows users to decorate software in the catalog with extra metadata within the UI. This ensures that the catalog can become complete and rich over time.

Share catalog completeness numbers publicly

Category: Create incentive

One great way to get people bought into the idea of building a complete catalog is to make it a group effort. By transparently reporting on the completeness and "health" of the catalog, a shared sense of ownership over the goal can be created.

Twilio explained how they do this in their catalog at the Autodesk Developer Productivity Summit.

How Roadie Helps

Roadie gives all Tech Insights customers an out-of-the-box measurement of catalog completeness.

We also report on important aspects of catalog richness, like the percentage of pieces of software in the catalog which have an owner assigned. Customers can use these building blocks to measure other attributes of the health of their catalog.

Tie catalog completeness to a wider initiative

Category: Create incentive

Companies usually want a complete catalog so that they can accomplish some wider engineering goal. By promoting catalog completeness in service of this wider goal, teams can better understand why it is important to be in the catalog, and why they should help.

For example, we recently worked with a customer who needed a complete and correct list of all software that had access to their users personally identifiable information (PII). By attaching to this company wide goal and leveraging the influence of the company CTO, the company was able to rapidly label hundreds of software components in Roadie with their PII status.

How Roadie Helps

We run regular customer success meetings with customers to help them identify wider engineering initiatives where Roadie can help accomplish the goal more quickly or more easily. We will then work with teams in order to project manage the delivery of a solution.

Do a whiteboarding session

I did some eduction with a team at [company] where we brainstormed the services they wanted to cover and their relationships, this was a whiteboard exercise and I then created the PR's and had them review.

When Paddle started with Backstage, they realized that they didn't have an existing source of truth they could lean on to populate their catalog, and they would have to collate it manually.

They started with a whiteboarding exercise so they could iterate quickly. Meeting with each team in small groups, Ioannis Georgoulas (Director of SRE), led the process. He first spent time brainstorming the services that the groups wanted to catalog, and defining their boundaries and the relationships between them. This information was all collected in a simple document to start.

Once he had a solid understanding of the service map, Ioannis opened pull requests against each repository with the catalog-info.yaml file that was needed. All the teams had to do was review and merge it. Because they had participated in the process to gather this information, they were already bought in and could understand the value of it.

How Roadie Helps

We provide frequent customer success calls with every customer through the initial stages of implementation. We'll partner with your implementors to run these whiteboarding sessions and gather the information you need to be successful.

Choosing the right internal developer portal for your software catalog

For organizations evaluating internal developer portals, the software catalog capabilities should be a primary consideration. The portal you choose will determine how effectively you can organize and track software assets, manage services and APIs, and implement automated workflows for your teams.

Key features to evaluate

When selecting an internal developer portal, consider these software catalog capabilities:

Customizable organization: The ability to define custom entity types, relationships, and metadata fields that match your organization's terminology. A portal that forces you into rigid categories won't capture the nuances of your software landscape.

Automated workflows: Look for scaffolder templates that let teams create new services with pre-configured catalog entries, reducing manual work and ensuring consistency. This is particularly valuable for mid-sized tech companies that need to balance speed with standardization.

Discoverability features: Search, filtering, and relationship visualization help developers find existing services before building new ones. The portal should make it easy to answer "do we already have something that does X?"

Compliance and security integration: Built-in scorecards or the ability to integrate with security scanning tools helps maintain governance across your software portfolio without manual audits.

Access control: Role-based permissions that respect your organizational structure, so teams can manage their own catalog entries while maintaining appropriate visibility boundaries.

Roadie provides all of these capabilities as a managed Backstage solution, removing the operational burden of self-hosting while giving teams the customizable features they need for organizing software services and APIs. For organizations that want the flexibility of Backstage without dedicating engineering resources to infrastructure, this combination of comprehensive software catalog features and managed operations makes Roadie a strong option for improving discoverability and compliance monitoring in enterprise environments.

Key takeaways/Conclusion

Building a high level of catalog completeness in Backstage need not be intimidating. By carefully choosing the strategy that will work at your company, expanding outwards from the most eager adopters first, and communicating widely as you go, you can reach a high level of catalog completeness in a short amount of time.

Two topics we have not discussed yet, are catalog richness and completeness. These areas go hand in hand with completeness, and work together to ensure that your IDP has the answers that developers need, when they need them.

We'll be covering richness and completeness in another article. If you want to be among the first to read it, make sure to subscribe to the newsletter below.

Making Backstage Easier for New Users

Imagine opening Backstage for the first time, searching for a service, and finding... nothing. How do you even search properly? You’d need to understand concepts like entity kinds, types, and ownership.

And if you can't find the entity, how do you add it? This requires knowing your organization’s ingestion patterns: Do you need a YAML file in the code repo? Or do you manually input the URL into the import flow? Without internal support or clear, beginner-friendly getting started documentation, this process can feel like a maze.

Backstage isn't always intuitive, especially for new users without internal support or clear documentation. Most available open source documentation is aimed at implementers, not end users. It’s often generic, overwhelming, and assumes you’re using GitHub as your SCM system. So, what can you do to make Backstage easier for your team?

How to Simplify Backstage for Your End Users

Invest in User-Friendly Documentation

For Backstage to succeed in your organization, internal getting started documentation to help users use Backstage is a must. This documentation should be front and center for new users, which might mean putting it in an existing documentation platform even if your goal is to eventually move all docs into Backstage’s TechDocs. You can use the Homepage plugin to highlight certain top level info for new users including a preview card for your getting started docs in TechDocs.

Your getting started docs should open with a clear intro to what Backstage is and how it helps. Include simple examples of its core features, remembering that many new users won’t even know what an internal developer portal is meant to do.

As well as have a intro page in your internal documentation explaining it you could also publish a blog post introducing IDPs to your engineers. i.e. Sample content for an intro to IDPs and Backstage

### Internal Developer Portals
An Internal Developer Portal (IDP) is an application which is designed to give our developers easy access to information and commonly used workflows. Its goal is to reduce context switching and toil for developers by providing a “single pane of glass” that helps to improve productivity, reduce duplication of effort, and foster a more cohesive and efficient development environment.

An IDP is a place to find information about the software we build and use, the teams who build that software, and the API specs, code repositories and documentation associated with that software. It also typically provides self-service automation scripts for common tasks like creating applications or making changes to infrastructure, and scorecards to help ensure that software is adhering to best practices.

### What is Backstage
Backstage is an open-source platform for building developer portals. It was originally created by Spotify to manage and streamline their complex microservices architecture and was released for public use in 2020. 

**Key features of Backstage include**:
- Software Catalog: A centralized listing of all services, providing an overview and allowing easy management and discovery.
- Software Templates: Streamlined processes for setting up new projects and services with pre-defined templates.
- Plugins: Backstage is extensible with plugins to integrate with various tools and services used in AstraZeneca.
- TechDocs: A documentation site generator that converts Markdown files into a browsable documentation site.

Backstage aims to improve developer productivity and collaboration by providing a single, cohesive interface for all development-related activities.

Identify key user journeys and craft your getting started content around them:

• Why would a software engineer or product manager initially visit Backstage?
• What are they trying to accomplish?
• What problems could Backstage help with?

Talk to teams who are just onboarding or align these journeys with company-wide initiatives. If, for example, you see the Scaffolder as a high-value feature, start with docs explaining how to use it and how to modify templates.

You could create these docs in TechDocs and link them from wherever engineers currently go for org-level documentation or just create them in existing documentation systems like Confluence.

Ensure they are searchable by including the right keywords in titles and descriptions, whether hosted in Backstage or elsewhere.

Lastly, publicise these docs as much as you can - get a few blog posts out on any internal news distribution channels, ping organization wide channels with the link and a short teaser description (we’ll be writing a more detailed post about internal marketing very soon with a bunch of resources for you to use).

Create a Dedicated Support Channel

Establish a clear support channel—like a Slack or Teams group—where users can ask Backstage-related questions. This not only helps with adoption but also builds a community where knowledge is shared and best practices emerge.

Pin relevant internal and external docs in these channels to avoid repeated questions. Support channels are also a great place to identify gaps in your documentation, allowing you to improve onboarding and make it as self-service as possible.

You could even nominate Backstage champions—advocates within your organization who can help answer questions and lighten the load on your Platform Engineering/DevOps teams.

At Roadie, support channels with our customers have been essential to successful Backstage rollouts. While documentation is the first line of defense, having a place for follow-up questions is key to encouraging usage and reducing friction for busy engineers.

By streamlining these two areas—user-friendly getting started documentation and a dedicated support channel—you’ll make engaging with Backstage a smoother experience for your team and boost adoption.

At Roadie we don’t claim to be experts in writing software standards - we’re done it ourselves, both for Roadie and before as part of other startups, scaleups and large companies, but we don’t claim to be world-beaters at it. What we can claim is that we’ve seen many, many companies go through the journey to create standards and then apply them and we have seen what works.

Why standardise?

Defining and adopting engineering standards is essential for organizations as they scale.

Teams involved in Platform initiatives normally find themselves as the first intrepid explorers in this territory for larger organizations.

Building tooling for multiple different teams and departments requires consistency and a coherent set of practices. Only then can teams coordinate, share and build scalable, maintainable, and secure software together.

Without standardization, answering basic questions becomes impossible and progress is painfully slow, a problem that often hits home at times of peak stress. How can you know which teams operate publicly facing software that has critical vulnerabilities if only half of the teams are using a dependency scanning tool? How can you roll out a new security requirement when engineering teams are each using one of 5 different security tools?

Defining Engineering Standards

Engineering standards are formal guidelines that outline how code should be written, how systems should be designed, and how processes should be executed. These standards ensure that all engineers are working toward the same quality benchmarks.

That means you need consensus between teams about what exactly should be in those standards. In order to do that, even before you decide what the standards are that you’d like to focus on, it’s good to have a plan for how you can agree together on what they should be.

Strategies for Agreeing on Engineering Standards

Getting a team to agree on a set of engineering standards can be challenging but is crucial for their success. Here are some strategies to help facilitate agreement:

1. Start with your own SDLC - if you have one - and/or Industry Best Practices:
   • Software Development Lifecycle and Production Readiness documents often effectively contain a lot of standards recommendations. They’re extremely useful as an input into a formal set of software standards and the two should be synced closely together.
   • Use industry standards like OWASP for security or WCAG for accessibility as a baseline. This helps reduce subjective debate by relying on well-known benchmarks. For example, it’s hard to argue that secure logging isn’t important when it shows up prominently in the OWASP table.
2. Collaborate across teams and functions:
   • Involve engineers, product managers, and operations teams in defining the standards.
   • Run workshops where everyone can voice their opinions, then converge on a decision.
3. Appoint a champion:
   1. Usually this is a member of engineering leadership who is responsible for driving this process and the eventual rollout.
   2. Occasionally this can be a group though, like an Architecture guild.
4. Gradually implement:
   • Start small, either:
     • A minimal set of standards and build from there as the team gets more comfortable. For example, you can start by enforcing code formatting standards and then gradually add performance or security checks.
     • A full set of standards with only a few initial checks so that the team can get comfortable with the whole suite of standards
   • These strategies allow teams to give feedback and for the standards to evolve before they are fully enforced
5. Regularly review and update:
   • An engineering standards document should never stand still.
   • Once the standards are set, encoded them into your systems.
   • Hold regular reviews (e.g., fortnightly or monthly) to review progress of each team
   • Create a regular cadence of review for the standards themselves, based on team feedback and new technology trends and recommit. Once a year is often enough here.
   • As part of these reviews, use data to demonstrate the value of each standard (e.g., reduced production errors, improved system uptime).

Common Standards

To make this concept concrete, let’s consider some common standards teams might define (nb: this is by definition a non-exhaustive list):

• Logging and Monitoring: Log levels, message formats, error tracking, use of a centralized tool, use of alerts.
• Security: Authentication, authorization, encryption, secure coding practices, and dependency vulnerability management.
• Performance: Response times, load management, and scalability.
• Reliability: Redundancy strategies, failure handling, backup and recovery.
• Code Quality: Style guides, review processes, formatting rules, and readability standards.
• Documentation: API documentation, code comments, and README files.
• Testing: Code coverage, test automation, and test environment standards.
• Version Control: Branching strategies, commit message guidelines, and pull request processes.
• Deployment: Continuous Integration/Continuous Deployment (CI/CD) pipelines, rollback procedures, and environment configurations.
• Accessibility: WCAG 2.2 guidelines, color contrast, text-to-speech, keyboard navigation

You don't want to over-elaborate at this point. It is important to end up with ~8-10 different areas to focus on.

Nice vocabulary to use when setting standards

• Must. Used to define mandatory items. i.e. A service must use a logger
• Should. Used to define items which are reasonably expected to exist. If a services choses not to adopt this standard, the expectation is that they justify why not.
• May /Could / Will. Used to define items which are more aspirational or for services that are consider mature.

An example: AcmeCorp.com

Let’s imaginee AcmeCorp.com are a well-known platform selling books, clothes, food, laptops, paddling pools and power tools around the world. They’re an anything store, if you will.

Availability and reliability are key to their business, so they spend a lot of time thinking about how to measure and improve that for the software they build.

Previously, teams would simply assert that their service was reliable, performance, secure etc, but aside from anecdotally looking at the past weeks/months/years to validate that assertion, it was hard to prove or disprove. It was also hard to compare across services.

To help that, a cross-functional group at AcmeCorp.com have agreed a series of standards that they believe will ensure their service stand up to considerable load during peak periods, and that if outages or incidents do happen that they’ll recover quickly.

| Area | Standard | | ---------- | ---------- | | Monitoring | Health checks for critical components must be defined and an ideal state determined. | | | Service state must be constantly observed and recorded and dashboards should be created to show this data. | | | Monitoring should have metrics that describe how effective a service is. These metrics are available and easily viewable on a dashboard. | | | Events should be exported and/or sampled and collected in addition to other metrics | | Availability | Service availability must be determined programmatically. | | | Expected and unexpected behavior for a given service must be defined in tests and alerts. | | | Basic SLIs should be defined and used to calculate SLO targets. This should include the number of number of good events / total number of events are being recorded | | | SLOs should be actively measured, calculated, and displayed in a dashboard | | | Error budgets may established and a policy outlining what to do when service runs out of budget is established. | | | SLOs (and error budget policy where appropriate) should be documented in a prominent location where teams and stakeholders can easily review. | | Logging | Logs must show startup, shutdown, and errors. | | | Logs must have have a rotation and retention policy defined. | | | Logs from all hosts must be sent to a centralized location. | | | Logging pipeline must be resilient to transient failures and should be fully recoverable when ingestion returns to a healthy state. | | Alerting | Basic health checks must be attached to alerts when failing. | | | A dashboard must display all alerts currently firing. | | | The body of any alert must contain information that is be needed to diagnose and fix the problem. | | | An official on-call rotation for high-priority alerts must be configured and activated. | | | High-priority alerts should tuned such that they don't fire outside of business hours unless necessary. If resolution of an issue can wait until business hours, it should not page the on- call engineer. | | | High-priority alerts should be triggered only for urgent, actionable issues that require a human's intervention. | | Scalability | Operating manuals for service scaling must be up to date and consumable by newly onboarded or tangentially-familiar engineers. | | | Service must handle unexpected increases in load without manual effort, up to a known threshold. | | | Unexpected increases and decreases in load must be handled automatically. | | | Unexpected increases in load above a known threshold may be handled automatically. | | | Owners of a service may run regular scaling exercises to test scaling assumptions. | | | Service may be able to deprioritize features and load when needed. | | Resiliency and Recovery | Run books must exist that outlines steps of recovering from loss of capacity. | | | Owners should have conducted testing on outages to validate recovery run books and quantify performance degradation. | | | Owners should demonstrate manual recovery is possible with minimal performance degradation (within established threshold) | | | Owners may demonstrate automatically recovery is possible with minimal performance degradation (within established threshold) |

Breaking Standards Down Into Scorecards and Checks

Once the standards are defined, they need to be actionable, measurable, and concisely group so that teams can understand them. This is where checks and scorecards come into play.

Scorecards

Scorecards allow teams to measure in only a few data points how well they are adhering to their engineering standards across a project or organization.

Scorecards should flow naturally from your standards and be fairly simple to define. Name them things that align with those standards and are comprehensible as a bucket of actionable checks against those standards.

For example:

• Security is a good, simple, easy to understand name. If you wanted to create levels for your scorecards to have some sense of progression, you could say Security - Level 1
• Secure Coding Standards might be a good option if you wanted to go to a more granular level with your scorecards.

Try and end up with ~10.

Checks

For each scorecard/standard, you need to break it down into one or more specific checks. A "check" is a verifiable condition that can be automated or manually enforced.

Just like Scorecards, Checks should be named things that are comprehensible but crucially should also be actionable.

For example:

• Node version should be >18 is a clear true/false statement about what the expectation is for a given service that uses Node.js. It is also clear what needs to be done in order to pass that check.
• Similarly CODEOWNERS should be enabled draws an even more direct line to what needs to be done for a given service to pass a check.

At Roadie we use our Tech Insights plugin to build these checks - the backend for which we also open sourced.

Whether you’re using Roadie, OSS or hand-rolling these checks, it’s important to have an idea of what a computationally enforceable check would look like for each of your standards.

Automating and visualising standards

Last but not least, you need some way to repeatedly check standards are being adhered to.

This usually comes in two forms:

• You can programatically check to see whether documentation exists. This is often in the form of a runbook at a given path, i.e. /docs/runbooks/recovery
• A third party tool is used to capture data that can then be interrogated programmatically. For example, this can be something as simple as an SLO existing in Datadog for a given service.

Both data sources can be used to confirm at scale whether or not a given service is correctly adhering to a given standard.

Teams no longer have to assert compliance, they simply need to ensure that the evidence is currently surfaced to prove that they comply.

Many scorecarding solutions do exactly this for you, without the need for teams to individual wire up different systems. For example, Roadie customers use Tech Insights plugin to provide standardised, automated checks across their entire software catalog with minimal or no intervention from individual teams.

Returning to AcmeCorp

Using the example of AcmeCorp.com again, let’s take one of their areas are turn it into a Scorecard with a series of checks. They use Datadog for their dashboards and Sentry for their logging so they can both provide sources of truth for their checks.

| Scorecard | Underlying Standard | Checks | Data source | | ---------- | ---------- | ---------- | ---------- | | Monitoring | Health checks for critical components must be defined and an ideal state determined. | Service has >1 health checks defined | Repo file that contains healthcheck test results | | | Health checks for critical components must be defined and an ideal state determined. | Healthcheck test results return current status codes | Repo file that contains healthcheck test results | | | Service state must be constantly observed and recorded and dashboards should be created to show this data. | Service has a Datadog dashboard to record service health | Datadog | | | Monitoring should have metrics that describe how effective a service is. These metrics are available and easily viewable on a dashboard. | >1 service metric is defined in Datadog | Datadog | | | Monitoring should have metrics that describe how effective a service is. These metrics are available and easily viewable on a dashboard. | Datadog metric monitor is configured | Datadog | | | Events should be exported and/or sampled and collected in addition to other metrics | >1 event has been sent to Sentry in the last day | Sentry |

Making a fix easy to implement

The final stage of implementing engineering standards is to make adherence as simple and easy as possible. If a check or scorecard failure is hard to achieve then teams will take longer to resolve it, the rate at which they resolve it will be lower, and standards will ultimately suffer.

Ask yourself:

• How many steps does it take to bring a service into compliance for a given standard? How can that be reduced?
• How can common or shared systems be leverage to help multiple teams into compliance? Using the AcmeCorp example above, a templated Dashboard for Datadog could help all teams skip design and production steps when setting up a Monitoring dashboard.
• How can the overall cycle time from error to resolution be reduced? Can common fixes be added to shared repositories or How To guides be written to help teams?

Where possible, implement quick fix options

At Roadie we use the Backstage scaffolder to automate many of the fixes for our scorecards. To take a simple example, one of our engineering standards is that branch protection must be enabled on any repository we create. If a service is linked to a repository without branch protection it fails that check. To resolve it, we have a 3-second Scaffolder template that can modify the GitHub settings associated with the repository. The only thing the team needs to do is look at the check and click a button.

Conclusion

Defining engineering standards is critical for ensuring that software systems are built and maintained with quality, security, and scalability in mind. Breaking these standards down into checks and scorecards allows teams to monitor compliance and ensure continuous improvement. By following a collaborative approach to defining standards and making them measurable, teams can streamline their development processes and produce more reliable software.

Whatever you are focusing on, the key to success lies in making standards actionable, measurable, and adaptable.

Plugins like the Catalog Graph Plugin allow powerful visualisation of these relationship graphs.

Similarly, relationships can become a core piece of information displayed on entity Overview pages.

Relationships can be added in one of two ways - via a YAML file update in an SCM repository, or in a third party source like GitHub Teams or Okta. A YAML update looks something like this:

kind: Component
metadata:
	name: identity-backend
spec:
	...
+	dependsOn:
+		- component:default/users-backend
+		- resource:aws/identity-backend-ec2

Restrictive Relationships

There are a very limited and restrictive set of allowed relationships between different Kinds in Backstage by default. Knowing what is allowed requires looking up the Backstage documentation each time to check the schemas.

For instance, a Domain cannot “depend on” anything in the default Backstage relationship model. However, maybe you have a domain like Shipping that depends on another like Identity. The Identity domain exposes APIs for customer addresses that the Shipping domain breaks without access to. You might even want to directly model the hard dependency on specific external API’s inside the Identity domain so that its easy to identify issues and notify the right people in charge of the Shipping domain when something breaks in those APIs.

Confusingly, many of the default relationships terms that can be used in the YAML spec definition don not map directly to the actual relationship in the catalog. For instance if you want to add a partOf relationship for a Domain to another Domain, you would need to use subdomainOf . Adding it as an explicit partOf field would not work unless you add that in the extended processor as we’ll see shortly. Each Kind has its own differing semantics for each relationship type, which needs to be referenced and checked to ensure a relationship is correctly added.

Expanding available relationships

Luckily its an easy engineering task to add a processor that can handle more expansive relationships and even new terms for those relationships such as manages/managedBy for Users. This processor would run in addition to the BuiltinKindsEntityProcessor that does the default relationship processing for the catalog and would look something like this:

import { CatalogProcessor, CatalogProcessorEmit, processingResult } from '@backstage/plugin-catalog-node';
import { Entity, getCompoundEntityRef, parseEntityRef, RELATION_DEPENDENCY_OF, RELATION_DEPENDS_ON } from '@backstage/catalog-model';
import { LocationSpec } from '@backstage/plugin-catalog-common';
import { RELATIONSHIP_MANAGED_BY, RELATIONSHIP_MANAGES } from '../constants';

export class ExtendedRelationshipsProcessor implements CatalogProcessor {
  getProcessorName(): string {
    return 'ExtendedRelationshipsProcessor';
  }

  postProcessEntity(
    entity: Entity,
    _location: LocationSpec,
    emit: CatalogProcessorEmit,
  ): Promise<Entity> {
    const selfRef = getCompoundEntityRef(entity);
    function doEmit(
      targets: string | string[] | undefined | null,
      context: { defaultKind?: string; defaultNamespace: string },
      outgoingRelation: string,
      incomingRelation: string,
    ): void {
      if (!targets) {
        return;
      }
      for (const target of [targets].flat()) {
        const targetRef = parseEntityRef(target, context);
        emit(
          processingResult.relation({
            source: selfRef,
            type: outgoingRelation,
            target: {
              kind: targetRef.kind,
              namespace: targetRef.namespace,
              name: targetRef.name,
            },
          }),
        );
        emit(
          processingResult.relation({
            source: {
              kind: targetRef.kind,
              namespace: targetRef.namespace,
              name: targetRef.name,
            },
            type: incomingRelation,
            target: selfRef,
          }),
        );
      }
    }

    if (entity.kind === 'Domain') {
      doEmit(
        entity.spec?.dependsOn as string[],
        { defaultKind: 'System', defaultNamespace: selfRef.namespace },
        RELATION_DEPENDS_ON,
        RELATION_DEPENDENCY_OF,
      );
      doEmit(
        entity.spec?.dependencyOf as string[],
        { defaultNamespace: selfRef.namespace },
        RELATION_DEPENDENCY_OF,
        RELATION_DEPENDS_ON,
      );
    }

    if (entity.kind === 'User') {
      doEmit(
        entity.spec?.managedBy as string,
        { defaultKind: 'User', defaultNamespace: selfRef.namespace },
        RELATIONSHIP_MANAGED_BY,
        RELATIONSHIP_MANAGES,
      );
      doEmit(
        entity.spec?.manages as string[],
        { defaultKind: 'User', defaultNamespace: selfRef.namespace },
        RELATIONSHIP_MANAGES,
        RELATIONSHIP_MANAGED_BY,
      );
    }

    if (entity.kind === 'Group') {
      doEmit(
        entity.spec?.managedBy as string,
        { defaultKind: 'User', defaultNamespace: selfRef.namespace },
        RELATIONSHIP_MANAGED_BY,
        RELATIONSHIP_MANAGES,
      );
    }

    return Promise.resolve(entity);
  }
}

Questions to consider

There is a rational that restricting the available relationships for each kind in the catalog prevents misuse of relationships by users adding things to the catalog. However it is worth bearing in mind that relationships in the catalog are mostly just syntactic sugar over the same link between two entities. There is nothing different about dependsOn and partOf in Backstage except the perceived meaning of these terms.

Educating users

Easily accessible documentation on standards and definitions is the best way to educate contributing users to your Backstage catalog. Backstage has its own schema docs and descriptions on existing relationships but you can also host internal documentation in a more compact format that is easier to reference quickly.

What we did to fix it

At Roadie, we’ve added expanded relationships for all kinds in the catalog to make it easier for people to correctly add relationships using the terms they understand. We’ve documented them in a compact format and are working on a UI based editor for these relationships so that users do not have to manually edit each YAML file and go through a PR process to build up a comprehensive mapping of dependencies in an organization.

We've also added a new card that can show all relationships for an entity, regardless of what kind of relationship it is in list format.

Contact us via Discord on on this website's chat to find out more about what Roadie can help with or to talk to our engineers about this topic or other issues you might be having adopting Backstage.

Core to Backstage is its Catalog of entities. The Catalog provides a database of software components, resources, libraries, and other kinds of software items. It provides client code and an API backend to retrieve items in the catalog, along with the software interfaces required to populate the entity catalog. Its model is flexible, customizable, and powerful.

However, with great power comes great responsibility. Without experience, it's easy to develop anti-patterns in Backstage catalog usage. These anti-patterns can then turn into major performance issues at scale. This in turn leads affects trust and usage of the product as a whole.

At Roadie, we provide an out-of-the-box version of Backstage for our customers. We have come across many of the ways in which non-optimal Catalog client usage can affect performance of the application as a whole. We have seen these performance issues result in lagging page loads and (in extreme cases) causing page loads to fail in Backstage.

By applying the patterns explained in this post, you could see a huge improvement in Catalog response time. In some cases, you may even see Catalog queries perform 48x faster!

Architecture of the Entity catalog

The entity catalog in Backstage is made up of three components. A Catalog client, the Catalog backend, and the Catalog database. When Backstage starts up for the first time, it will have a Catalog database and a catalog backend. When you visit the Backstage application in your browser, it will make use of the Catalog client to retrieve data from the Catalog backend. The Catalog backend in turn retrieves the requested catalog items from the Catalog database.

Using the Backstage Catalog Client

Soon after deploying Backstage in an organization, users will want to customize it.

Frequent customizations we come across include loading entities into the Catalog from an in-house platform or visualizing data from an internal system in the Backstage UI. Customization is normal and is a sign that Backstage is adding value for teams.

When developers write extensions to Backstage, it's likely they will come across the need to interact with the Catalog. There are two ways they can do this:

1. via a Frontend Backstage extension
2. via a Backend Backstage extension

To make use of the Catalog client in a frontend Backstage extension, you are likely to be using the useApi hook, along with a useAsync function.

import { catalogApiRef } from '@backstage/plugin-catalog-react';
import { useApi } from '@backstage/core-plugin-api';
import { stringifyEntityRef } from '@backstage/catalog-model';
import useAsync from 'react-use/lib/useAsync';

export const CustomReactComponent = () => {
  const catalogApi = useApi(catalogApiRef);
  const {
    value: entities,
  } = useAsync(async () => {
    const response = await catalogApi.getEntities();
    return response.items || [];
  }, [catalogApi]);

  return (<>{entities.map(stringifyEntityRef).join('\n')}</>)
}

In a backend Backstage extension, you are likely to be constructing the Catalog client using the discovery client. The discovery client is a helper that allows plugins to discover the API location of other clients.

Generally, if you are writing a backend plugin, like a new REST API or a Catalog processor, you will have access to the discovery client. Depending on your particular situation, you may have access to the discovery client in a different way.

import { CatalogClient } from '@backstage/catalog-client';
import { DiscoveryApi } from '@backstage/core-plugin-api';

export const getAllEntities = async (discovery: DiscoveryApi) => { 
  const catalogApi = new CatalogClient({
    discoveryApi: discovery,
  });

  const response = await catalogApi.getEntities();
  return response.items;
}

You will notice that once you have an instance of the CatalogApi, it is used in the same way in either the frontend or a backend extension.

(await catalogClient.getEntities()).items;

Check the Backstage docs for a more comprehensive explanation of the full Catalog interface.

How big can a Backstage Catalog get?

When thinking about Catalog size, it’s useful to think about two things:

• How big is each individual entity?
• How many entities do you have?

A typical Entity

A typical Backstage Entity looks like this:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: artist-web
  description: The place to be, for great artists
  labels:
    example.com/custom: custom_label_value
  annotations:
    example.com/service-discovery: artistweb
    circleci.com/project-slug: github/example-org/artist-website
  tags:
    - java
  links:
    - url: https://admin.example-org.com
      title: Admin Dashboard
      icon: dashboard
      type: admin-dashboard
spec:
  type: website
  lifecycle: production
  owner: artist-relations-team
  system: public-websites

It describes a website called the artist-web. It has a few basic Backstage entity properties, and some annotations, tags, and links. It's encoded in YAML here, but in Backstage's database, it is stored as plain text in JSON format.

Uncompressed, this entity definition is about half a kilobyte. Therefore, a Catalog containing about 20,000 similarly sized entities would add up to about 10 megabytes of data uncompressed. That's a pretty big chunk of data to be sending over the wire.

However, we haven't seen anything yet…

The Backstage Catalog model defines an API Kind. These are used to document the endpoints that services make available in Backstage. API entities often contain an embedded OpenAPI doc.

For example:

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: artist-api
  description: Retrieve artist details
spec:
  type: openapi
  lifecycle: production
  owner: artist-relations-team
  system: artist-engagement-portal
  # The embedded OpenAPI spec is in the defiition
  definition: |
    openapi: "3.0.0"
    info:
      version: 1.0.0
      title: Artist API
      license:
        name: MIT
    servers:
      - url: http://artist.spotify.net/v1
    paths:
      /artists:
        get:
          summary: List all artists
    ...

At Roadie, we have seen multiple customers with API kind entities in their Catalog with embedded OpenAPI docs as large as 1 megabyte in size. It's easy for even a small-sized engineering organization to have 50 such APIs documented in Backstage. Unoptimized, that's 50MB+ of data that's being transferred every time we query the full Catalog.

This Catalog size is important because poor use of the Catalog APIs can cause huge database queries and API response sizes to result, which will cause both unwarranted traffic across the network and unwanted wasted time transferring, encoding, and decoding that data.

How to Make Good Use of the Entity Catalog

With all this said, we wanted to run through some good practices that are going to help with improving the Backstage experience. The examples we use in the tables below are measured in the browser on a production Catalog that has 14k entities.

Unoptimized, we're looking at 2.16 seconds and 59.5 MB of data. That's our starting point. Now each experiment we do below is going to improve that data.

Only query the entities fields that you need

By default, when retrieving entities from the Backstage Catalog, Backstage will return the whole entity for each item listed. As mentioned above, an entity might be as large as 1 megabyte. As such, limiting fields requested to the ones that are strictly required can help a lot. For example, you might have code like the following that is requesting every entity in the Catalog and then converting the result into an array of entity references.

(await catalogClient.getEntities()).items.map(stringifyEntityRef);

If you look under the hood, you'll find that the stringifyEntityRef function only makes use of the kind, name, and namespace. As such, we can cut down on the amount of data transferred across the network by limiting the fields that are requested.

(await catalogClient.getEntities({
  fields: ['kind', 'metadata.name', 'metadata.namespace']
})).items.map(stringifyEntityRef);

| Operation | Time | Size | | --- | --- | --- | | catalogClient.getEntities() | 2.16 seconds | 59.5 Mb | | catalogClient.getEntities({ fields: ['kind', 'metadata.name', 'metadata.namespace'] }) | 0.767 ms | 1.2 Mb |

Make use of the filter option to retrieve only entities that you need

We have seen a pattern develop whereby the Catalog client is used to retrieve all of the entities in the Catalog, and then the list is filtered client side.

(await catalogClient.getEntities()).items.filter((entity) => entity.kind === 'Group');

It is more efficient to send a filter to the catalog client so that filtering is done either in the Backstage backend or the Backstage database.

(await catalogClient.getEntities({ filter: { kind: ['Group'] } })).items

| Operation | Time | Size | | --- | --- | --- | | catalogClient.getEntities() | 2.16 seconds | 59.5 MB | | await catalogClient.getEntities({ filter: { kind: ['Component'] } }) | 69 milliseconds | 2.5 MB |

Avoid retrieving all entities in order to count entities

A common pattern we see in Backstage is for developers to download the whole contents of the Catalog in order to count the entities in that Catalog. The following code will cause Backstage to query the database for every entity, then the client will need to decode the JSON it retrieves in order to count the number of entities.

(await catalogApi.getEntities({})).items.length

There is a far more performant way to do this, using the query API. The following requests a limit of 1 entity to be returned, and also requests that the uid field from the entity is the only item that is returned for that entity. The query API always returns the total count of entities for that query. As such it gives us what we need with out downloading the whole Catalog to the client.

(await catalogClient.queryEntities({
  fields: ['metadata.uid'],
  limit: 1,
}).totalItems;

The change suggested here is going to save work for the Backstage database, the Backstage backend, and the Backstage frontend.

| Operation | Time | Size | | --- | --- | --- | | catalogClient.getEntities() | 2.16 seconds | 59.5 MB | | catalogClient.queryEntities({ fields: ['metadata.uid'], limit: 1 }) | 45 milliseconds | 0.5 Kb |

Enable Gzip Compression

When not using the Catalog Client, we recommend using gzip encoding to reduce the amount of data transferred. This is crucial because requests for large amounts of data directly from the Backstage APIs can be massive. Enabling compression significantly decreases the data volume sent to the client. You can achieve this by including the Accept-Encoding header with your client requests.

| Operation | Size | | --- | --- | | curl https://backstage-server/api/catalog/entities | 59.5 MB | | curl https://backstage-server/api/catalog/entities -H 'Accept-Encoding: gzip' | 6.7 MB |

Keep Backstage up to date

The first, and perhaps the most important thing to consider is to keep Backstage up to date. If you are using Roadie, you are already using a very recent version of Backstage. However, if you are managing Backstage yourself, you may have fallen behind. Backstage releases new versions at least once a month, and these versions often contain very valuable performance improvements to the Catalog.

For example, in version 1.6.7 of the Catalog client library, there was an optimization. Previously, the Catalog client would sort all entities before returning them to the caller. This is a nice, helpful utility until there are thousands of entities to sort. Often, it is not necessary or optimal to receive a sorted list of entities.

Collaborate on the OSS Backstage core project

As part of research for this document, we spoke with the core maintainers of Backstage, and there are some great ideas about how to continue to improve the performance. For example, it has been discussed that by default, the getEntities function should be replaced by an iterator object. That iterator would be used to page over the list of entities rather than retrieving the whole list.

As such, keeping up to date with Backstage releases will allow you to benefit from these performance improvements.

Conclusion

This article is illustrative of some of the performance gains that can be achieved, and your mileage may vary. We have not delved into the performance implications of these changes on backend memory and database query performance. However, we can say that these changes can greatly improve these items too. It's difficult to quantify; however, at Roadie, we were seeing huge memory spikes and large garbage collections occurring in Backstage when the whole Catalog is queried. This is possibly due to the physical sizes of the entity Catalogs and the serialization and deserialization that occurs between the client, backend, and database.

We have shown that making use of some good patterns can result in a much improved load times for users. We have shown some examples where timings are reduced from multiple seconds to sub-second. We have also shown that the sizes sent across the wire can be greatly reduced from multiple megabytes to tens of kilobytes.

A well-managed and optimized internal developer portal can make your software engineers more efficient and empower them with the information they need. When load times are reduced from multiple seconds to sub 1 second, developers enjoy a fast, responsive experience that means they’re more likely to use Backstage and find what they need.

For many, this manual process can feel like a series of hurdles—switching contexts, waiting for pull request reviews, and depending on other team members or systems.

This friction increases significantly when you scale across hundreds or thousands of repositories. For example, changing a group name might require chasing after numerous teams to raise and merge PRs, turning what should be a simple update into a months-long process.

Having to edit YAML makes keeping your catalog up-to-date a challenge, let alone enriching it with additional data or plugins like PagerDuty.

How long does it take to add a PagerDuty plugin?

Lets use the PagerDuty plugin as an example. Suppose you want to add PagerDuty monitoring to a Backstage entity representing a backend service. All you need to do is add an annotation to the YAML file with the relevant service ID.

The quickest way to edit the YAML file might seem simple: open GitHub, click the pencil icon on the About card, and start editing.

Manual Process in GitHub

Once inside GitHub’s web editor, you can make the change, commit it to a new branch, and open a pull request.

Sounds straightforward, right? But here’s where it gets tricky. To add the PagerDuty plugin, you need the correct service ID, which requires logging into PagerDuty, searching for the service that matches your entity, and then copying the ID from the browser’s address bar (since it’s not easily accessible in the UI).

In GitHub’s case, we can edit this immediately, commit it to a new branch and open a pull request rather quickly all in the web editor.

But to find the correct Service ID we must first go to PagerDuty. Hopefully we can log in and access the available services, and then search for the correct one using the names in the Component YAML.

Now, after locating the ID (in the address bar) and adding it to the YAML file, you still need to commit the changes, submit a PR, and wait for it to be reviewed and merged. Depending on the team’s review cadence, this could take days or longer.

Multiply this by hundreds of teams and thousands of repositories, and the process becomes incredibly slow and frustrating. Minor updates can take weeks or months to implement at scale.

A Better Way: Entity Updates with Roadie

At Roadie we’ve built UI based tools that allow you to set things like PagerDuty annotations with a few clicks. Instead of seeing a missing annotation card you’ll be able to select from a list of PagerDuty services and update the entity immediately, all from the same page in the UI.

Bulk updates via UI

But what if you need to update multiple entities at once? Roadie also provides a bulk editor that allows you to make updates across many entities using a simple table format. This dramatically reduces the manual effort required to keep your catalog accurate and up-to-date.

Behind the Scenes: How It Works

Our UI-based editors are powered by a backend feature we call Fragments. Fragments are partial entity data stored in a database table and merged into the existing entities in the catalog by a processor. These fragments can be updated through our UI or via API.

Leverage external APIs to make things easier

The custom UI we’ve seen earlier in this post also leverages external API’s to provide a dropdown of available options.

In this example we’ve used PagerDuty’s API to fetch a list of services using the API token already added to Backstage to make your PagerDuty plugin work. This call is made via the proxy which is set up to point to PagerDuty using your API Token on the backend.

Script bulk changes via API

However, sometimes bulk editing of metadata in the catalog is best scripted, and in this scenario, Roadie users make use of our Fragments API to make changes to entities, such as changing the ownership of a group’s entities when the group gets absorbed into another group, or updating any relationship references to a user entity when that individual leaves and someone else takes their role.

We’re not the only ones who’ve done something like this - companies like Twilio and others have built similar fragment functionality so that they can script updates to there catalog via API and free up platform teams to help keep the catalog data rich and accurate.

Reach out to us on Discord or our website’s chat messenger if you want to hear more about this and other improvements Roadie has made to the Backstage experience.

These are called “Kinds” and they are meant to be combined with a “Type” that signifies more detailed sub categories within these generic buckets. For instance, you could have a Component of type library or a Component of type website. Types are completely up to you to define, which can lead to its own set of problems, as we will see.

When someone wants to add something to your catalog, they will need to decide first what kind that something belongs in, and then the type. Most of the time these questions arise sitting in front of a blank YAML file in your code editor, without any helpful context to hand. This can lead to several problems, as we shall see, even for those who understand the Backstage YAML approach and schema.

The common problems

Which Type?

How does this decision get made? A conscientious user could first go to the existing catalog and try to find patterns already used for a similar thing, and then decide whether to follow that pattern or not.

However it might also be the case that the user just tries to use their own common sense to choose the kind and type. This often leads to problems particularly with types as you can end up with multiple versions of the same type - i.e. website, Website, site, web-server, webServer, web-app etc.. This can even be intentional when the existing type doesn’t follow the formatting of other types so a user decides to introduce a corrected duplicate.

Duplicate types can cause problems when you want to search for things in the catalog by type as you have to select all variants, both in the UI and via API. The Backstage search might not work as expected or might miss results. And the cognitive load of seeing a messy catalog also has an impact in perceived reliability of the data, which could mean users look at it less and don’t see it as a source of truth anymore.

Which Kind?

Types are not the only potential problem area. Kinds may not neatly map to the top level terminology or groupings used inside your organization. Maybe you are using Value Streams, which are groupings of Domains as an organizational concept and way of grouping teams. Backstage has no ValueStream kind, only the Domain kind. Domains are not the same as a Value Stream - in fact Value Streams usually comprise of multiple domains. If you want to model Value Streams with the default options you would have to use a kind of Domain with a qualifying type of value-stream, which might feel unintuitive or even stop users adding it at all.

The solutions

So how can we address these problems?

Problem 1: Kinds don't map to top level concepts in your organization causing confusion

Custom Kinds

It is possible to create custom kinds in Backstage that better align to your organization.

Creating a new kind is a relatively simple engineering task involving registering a new processor and validation for that kind that emits any desired relationship mappings.

import { CatalogProcessor, CatalogProcessorEmit, processingResult } from '@backstage/plugin-catalog-node';
import { ProductEntityV1 } from './ProductKind';
import { Entity, entityKindSchemaValidator, getCompoundEntityRef, parseEntityRef, ...} from '@backstage/catalog-model';
import { LocationSpec } from '@backstage/plugin-catalog-common';
import productSchema from './Product.roadie.v1.schema.json';

// Creates a validator using the JSON AJV schema imported above. 
const validateProductEntity = (entity: Entity) => 
	entityKindSchemaValidator(productSchema)(entity) === entity

// Processors will run against every entity in the catalog
export class ProductKindProcessor implements CatalogProcessor {

  getProcessorName(): string {
    return 'ProductKindProcessor';
  }

  postProcessEntity(
    entity: Entity,
    _location: LocationSpec,
    emit: CatalogProcessorEmit,
  ): Promise<Entity> {
    const selfRef = getCompoundEntityRef(entity);

    // Function for triggering relationships to be processed
    function doEmit(
      targets: string | string[] | undefined,
      context: { defaultKind?: string; defaultNamespace: string },
      outgoingRelation: string,
      incomingRelation: string,
    ): void {
      if (!targets) {
        return;
      }
      for (const target of [targets].flat()) {
        const targetRef = parseEntityRef(target, context);
        emit(
          processingResult.relation({
            source: selfRef,
            type: outgoingRelation,
            target: {
              kind: targetRef.kind,
              namespace: targetRef.namespace,
              name: targetRef.name,
            },
          }),
        );
        emit(
          processingResult.relation({
            source: {
              kind: targetRef.kind,
              namespace: targetRef.namespace,
              name: targetRef.name,
            },
            type: incomingRelation,
            target: selfRef,
          }),
        );
      }
    }

		// Adding relationships
    if (entity.kind === 'Product') {
      const product = entity as ProductEntityV1;
      doEmit(
        product.spec.owner,
        { defaultKind: 'Group', defaultNamespace: selfRef.namespace },
        RELATION_OWNED_BY,
        RELATION_OWNER_OF,
      );
      doEmit(
        product.spec.system,
        { defaultKind: 'System', defaultNamespace: selfRef.namespace },
        RELATION_PART_OF,
        RELATION_HAS_PART,
      );
      ...
    }
    return Promise.resolve(entity);
  }

	// This is is required so that the catalog will be able to ingest this new Kind
  validateEntityKind(entity: Entity): Promise<boolean> {
    if (entity.kind === 'Product') {
      return Promise.resolve(validateProductEntity(entity));
    }
    return Promise.resolve(false);
  }
}

In addition you will need to add the kind to an allow list in your app-config.yaml file in the root of your Backstage repo like so:

catalog:
  rules:
    - allow:
      - Component
      - Product
      ...

However, you will want to ensure that any new Kinds introduced are stable enough concepts in you organization that they are unlikely to be changed. Deprecating a top-level Kind when you have thousands of YAML files to manually update across hundreds of teams can be a time consuming process.

Documentation

You will also want to document your new kind somewhere internally which could mean duplicating Backstage documentation on catalog schemas so that your organization has a single place to go for reference to all available entity schemas.

Scaffolder

Lastly you can create a Scaffolder template to help people bootstrap a new catalog-info.yaml file that uses a dropdown of available kinds so that its easy to know what options are available. An example template for GitHub can be found here.

Problem 2: Types are prone to misuse and duplication

Types are in some ways a harder problem to solve as the pain point is largely caused by a disconnect between writing YAML manually in a code editor and your Backstage application.

Its worth noting additionally that types have a constrained set of characters, and for instance cannot have separate words. If the format is invalid they will fail to be ingested to the catalog.

Documentation

You can write documentation on catalog entity schemas to try and keep track of agreed types and hope that your contributors will reference it.

Using the CI Pipeline

However a better approach might be using some kind of step in your CI that either forces types to match existing types, tells you if it does or does not, or just tells you what existing types there are.

You can establish existing types in the catalog easily using the /api/catalog/entity-facets?facet=spec.type endpoint, or you could expose a dedicated type validator endpoint in your Backstage backend that also tells you if a type is already in the catalog.

Alternatively you could use a config based approach or allowlist of types. With this approach, new types must be added to the validator (and ideally documentation) via a review process.

The downside of some of these less permissive CI jobs are that they can add friction, which can lead to entities being labelled with types that are not appropriate because the contributor doesn't want to go through the process of requesting a new type even if it is a valid addition.

Alternatively a more permissive CI job could just print out existing types for the kind(s) you are adding via the /api/catalog/entity-facets?facet=spec.type endpoint and leave it to the contributor to make sure they align if it is not a new one.

Using the Scaffolder

You can also create a Scaffolder template to help people bootstrap a new catalog-info.yaml file that uses a dropdown of available Types so that its easy to know what options are available and not duplicate them. Specifically you can use the SelectFieldFromApi parameter widget in your template and point it to the Catalog API endpoint to get a list of available types.

How Roadie can help

At Roadie, we help you overcome these kinds of challenges in a few ways. We take care of adding new kinds to Backstage for you as well as a wide range of relationships or new relationships for those kinds.

CI Validator

Our catalog validator can run on your CI to check that the YAML you are adding or updating is valid (i.e. the type field format will not break the ingestion).

Tech Insights Scorecards

Our Tech Insights feature allows you to track the correctness of YAML files being added to your catalog across the organisation as well as at Group levels, and even fix issues with a link to a pre-filled Scaffolder template.

Scaffolder

Lastly we make available a series of Scaffolder actions that allow non-technical users to add things to the catalog with context provided by API calls that can then populate a list of existing types for you so users won't make a mistake with duplicates.

The Backstage developer portal is an excellent tool for platform teams, as well as engineers, to keep a handle on their software, maintain compliance statuses and spin up new services. Unfortunately the open source Backstage is known for its difficult set up time and overall cumbersome maintainability.

This pain is often made worse when the catalog within a Backstage instance gets large. Below are a few high-level pointers that we have come across during our journey to support bigger engineering organizations. We’ve scaled our installation to support organizations with multiple tens of thousands of entities over the last year. We’ll dig deeper into individual topics, based on interest, on further articles.

Let us know about your optimization problems or questions in either Roadie or Backstage Discord channels!

Handling the Catalog data

We have written more extensively about catalog performance and how to improve that in a separate blog post.

When developing on top of Backstage, you are always building on the foundation of solid catalog data. This makes the CatalogAPI usually the most used API on both back- and frontend of the application. It may be that the entities in the system grow large (looking at you API specs) or that there is just a large quantity of them (looking at you automatically ingested AWS resources). Therefore it is important to retrieve only the actual necessary fields that are displayed to the user and limit the amount of entities being fetched.

The default CatalogClient has the option to retrieve only relevant fields through the API. Do use it. Also make sure to use the pagination if possible and hit the correct endpoints with your catalog client. Retrieving less data is always going to be cheaper than retrieving more data. It really makes a big difference whether you want to JSON parse/stringify the biggest API docs in the world multiple times, along with the rest of the catalog data or you just want to use ineffective string manipulations for sorting purposes.

For some cases we at Roadie have needed to introduce our own endpoints and catalog queries to improve the performance in larger catalogs. These endpoints could be as simple as creating pointed subset queries directly against the database table to identify only needed entities. Or possibly only returning partial entity data with preformatted response shapes. Having the ability to do use-case specific queries for relevant data, and use the better performance usually present in the database layer makes a big difference at times.

Processors vs. Providers

In the early days of Backstage the approach to ingest entities into the catalog was by using Processors to retrieve data from third party sources. This is still a remnant within the product, and is (unfortunately) still used by some integrations. The main purpose of processors nowadays is to enhance the entities, but same caveats on their usage are still present.

CNCF maintainers of the project introduced Providers to Backstage at a later stage. These providers allow more maneuverability to schedule and modify the payloads that you are sending to the catalog. Being able to chunk the ingested entities into smaller buckets, having the ability schedule the intervals with more (or less) granularity and having better visibility to the internals of the catalog is a big benefit when tweaking the catalog ingestion to work optimally.

In many cases the problem may still remain though. The providers may have the need to emit locations or other intermittent data before it is finally stored into the system as a full entity. And in those cases the entity may need to go through the processing pipeline again.

When you encounter issues that may be related to this approach, make sure that your processors are nimble beasts and are definitely not blocking the event loop. Small milliseconds make a difference here. The system is both processing a lot of data and doing it multiple times. User experience may also suffer when processing times get large or some immediately expected entities are clogged up behind a large processing queue.

In Roadie we have taken an even more performant alternative approach for few specific entities that we natively support and ingest. We want more fine-grained control to serve our larger users better and have created an alternative, self-contained processing module to do the processing for some specific use cases.

Scaffolder

By default the Scaffolder within a Backstage project runs in the same process as the rest of the application. This is by design, but there is an escape hatch that can be used to externalize this from the codebase. Backstage is built as a modular monolith and in theory has the possibility to be spread out into multiple services.

There is a fair amount of work to achieve that but the payoff is usually there. The decision to make here is to identify the tradeoffs that your company is willing to sacrifice. Is it ok that only a single scaffolder run is manageable at one time? Does it matter if the rest of the Backstage application begins to show signs of slowness when other processes are running?

The Scaffolder, and larger Tech Insights installations, take a lot of CPU cycles from the underlying hardware which may negatively interfere with the user experience. Blocking the event loop is the big no-no in the Node.js world when it comes to performance. If you are running heavy tasks within the same process that you are using to serve your users, you may encounter bad times. If bad times appear, consider externalizing some of the chunkier pieces of your instance. These may include the Scaffolder, Tech Insights, Search indexing, Cost Insights and the catalog processing loop.

Roadie has extracted the larger, more resource hungry processes into ephemeral standalone processes to avoid eating up the event loop cycles of the main application. In our case these are running in AWS, where we host your Roadie instances, as ECS tasks or Lambda functions, depending on the use case. With the backend system fully out for the Backstage project, it should be much easier to spin out supporting services into their own processes and leave the catalog alone to do what it does best, showing entities to the users in a performant manner.

Perceived Frontend Performance

Of course, performance is relevant only to the users if they are able to feel it. This is present in standard Backstage installations on the frontend layer of the application. Does your catalog load fast? Do you have a ton of frontend plugins installed and your bundle sizes are big? Do you need to rebuild your tech docs every time you navigate to the docs page?

For the frontend resources, there are multiple well-known performance tricks that can be included in the build process and hosting solutions that you are using to serve your frontend app. In the end, the Backstage frontend is a single-page application with all the known benefits and caveats. All the data displayed will need to be retrieved from somewhere before they enter the Javascript runtime to be rendered on the screen.

In most cases getting the data we want to display means API calls. For some that is ok, like getting cheap values from fast endpoints, but for some the roundtrip to the server is not worth it. You can embed relevant information to the index.html that is either served from the backend (in newer Backstage installations) or pre-built during the deployment process. You can also use localstorage to your advantage, in fact Backstage does use this for some of its data, but not necessarily for caching purposes unfortunately.

YAMLs and scaling maintainability

The canonical and recommended approach by the CNCF open source maintainers of Backstage is to use catalog manifest files, usually called catalog-info.yaml within code repositories to store entity data for the catalog. In a large amount of cases this is the wrong approach. You maybe able to keep domain and system entities up to date easily, since they are small in numbers and change rarely. For other kinds/types of entities we have seen with multiple of our customers that maintaining and keeping those YAML files up to date in an engineering organization is difficult.

A better approach to ingest entities in many cases is to automate the process of at least initial entity information from a more robust source of truth. In the end, trusting humans to update a random file in their repository just for the sake of updating it seems unlikely to succeed 100% of the time.

In modern engineering organization there are multiple different good sources to use as the canonical starting point for your entity data. For users and groups you have your Oktas or Azure Entra Ids. For repositories you have your GitHub APIs. For components, APIs and resources you have your running instances, your exposed OpenAPI endpoints and your K8s or cloud provider APIs.

Ingesting the relevant data automatically from these allows you to trust that the relevant information is up to date and mirrors correctly the software that is actually being developed within your organization and what is running in environments.

That is in the end the purpose of a developer portal, a mapping of software that your are providing to your customers, not a mapping of software that you have at some point written into a well-formatted text file.

Rate limits

There are downsides to automating your catalog ingestion as well. Backstage relies heavily on integrations towards third party APIs and this causes some implications on how up to date it can keep the catalog information. Being so reliant on other services and wanting to be the single pane of glass to display that information means that you need to be aware of the limitations of this system.

Backstage by default is a pull-based system which contacts third parties using API tokens or other authentication information and retrieves relevant data. Usually in the form of plugins, this isn’t a massive issue since the actual concurrent user count is relatively small. Even for the bigger clients we don’t usually see high 3 digit morning rushes. The just-in-time nature of retrieving data on runtime to display from third parties on the familiar frontend thus works well.

On the other hand, Backstage also stores data internally. Data that it gathers automatically from third parties and uses to generate insights or enhance entities. These processing loops usually run at a schedule and try to slurp in as much as they can. Herein lies the problem where rate limits are introduced in the system.

Monitoring rate limits against different system is extremely important and helps you identify when you getting close to the edge to make the downstream service angry. Backstage offers a good set of monitoring primitives to expose metrics from your providers. You can for example set up open-telemetry to gather the information you need. Exposing rate limit information either by querying it on a loop periodically or directly embedding it to your fetch client implementations. The former approach gives you the ability to manually tweak your calling schedules to accommodate your integrations, the latter may give you the ability to automatically slow down the calling loop to satisfy the limits.

Let us know if you have encountered any other aspects or approaches that have helped you scale your Backstage instance and work effectively within your organization.

In today's fast-paced tech landscape, the importance of efficient software delivery and operations cannot be overstated. As organizations strive to innovate and deploy new features rapidly, they often encounter a significant hurdle known as the "Day 2 problem." This term refers to the challenges that arise after the initial deployment of software, including maintenance, scalability, and observability. Roadie, leveraging the power of Backstage, is here to address these issues head-on.

Understanding the Day 2 Problem

The Day 2 problem emerges once a service or application is live. Initial deployments, often facilitated by tools like the Backstage scaffolder, can be exciting and successful, but afterwards organizations must ensure that their systems are robust, resilient, and scalable to handle ongoing demands.

Common challenges associated with the Day 2 problem include:

1. Operational Complexity: As the number of services grows, so does the complexity of managing them. Understanding dependencies, configurations, and performance metrics becomes increasingly challenging.
2. Monitoring and Observability: Ensuring that systems are functioning correctly requires effective monitoring tools. Without proper observability, identifying issues and their root causes can be a daunting task.
3. Scalability Issues: As user demand fluctuates, systems must scale efficiently. Inadequate scaling strategies can lead to outages and degraded performance.
4. Technical Debt: Over time, quick fixes and shortcuts can accumulate, leading to a backlog of maintenance tasks that hinder progress.

While we all know of the pitfalls of continued development of a service, we also often fall into them. So how can we avoid these problems and build better software over time?

Roadie and Backstage: A Powerful Duo

Enter Roadie, a platform designed to enhance developer experience by simplifying the management of software infrastructure. Built on Backstage, an open platform for building developer portals, Roadie provides a comprehensive solution to the Day 2 problem.

Tech Insights

At the core of Roadie's solution to the Day 2 problem is the Tech Insights feature. This powerful scorecarding tool equips teams with the necessary capabilities to monitor and improve their systems effectively.

Here’s how it addresses the challenges:

1. Built-In and Custom Data Sources: Roadie provides automated data capture of key systems and the ability to create your own custom data sources. Each of these data sources captures data to evaluate the health and performance of your services. These data sources run regularly, ensuring that any deviations from expected standards are identified quickly, allowing teams to address issues proactively.
2. Checks: Roadie supports the creation of custom checks tailored to specific organizational needs. This flexibility allows teams to define criteria that are most relevant to their applications, enhancing monitoring and compliance.
3. Scorecards: Roadie provides scorecard features that offer a clear visual representation of service standards across multiple checks. These scorecards make it easy for teams to assess standards at a glance, highlighting areas that require attention and improvement.
4. Technical Documentation: When issues are detected, Roadie offers direct links to relevant technical documentation. This ensures that developers have immediate access to the information they need to resolve issues efficiently.
5. Scaffolder Templates: In addition to documentation, Roadie integrates scaffolder templates into checks to guide teams in fixing source code or performing various infrastructure tasks. This feature streamlines the process of implementing changes, reducing the time spent on manual fixes.

Wider Benefits of Using Roadie with Backstage

1. Improved Developer Experience: By providing clear visibility, Roadie enhances the overall developer experience. Teams can focus more on building features rather than troubleshooting issues.
2. Faster Incident Response: With automated checks and scorecards, data piped into partner systems like Incident.io, and clear lines of ownership outline in the Software Catalog, teams can respond to incidents faster, minimizing downtime and improving overall service reliability.
3. Increased Productivity: By streamlining service management and automating fixes through scaffolder templates, Roadie empowers teams to be more productive, allowing them to deliver new features and enhancements more rapidly.
4. Proactive Maintenance: The combination of checks, scorecards, and direct access to documentation helps teams adopt a proactive maintenance approach. This reduces technical debt and ensures that systems remain healthy and efficient.

Conclusion

In a world where rapid deployment is essential, addressing the Day 2 problem is crucial for sustained success. Roadie, in conjunction with Backstage and its scaffolding capabilities, offers a powerful solution to navigate the complexities of service management post-deployment. By leveraging the Tech Insights feature—complete with checks, scorecards, and fix links—Roadie enables teams to focus on what they do best: building and delivering innovative software solutions.

As organizations continue to evolve, leveraging tools like Roadie and Backstage will be key to overcoming Day 2 challenges and ensuring a smooth journey from deployment to operation. Embracing this approach will not only improve operational efficiency but also foster a culture of continuous improvement and innovation.

🚨 A new Product kind

A new kind has made its way into Roadie.

You can now leverage the Product kind in your catalog. It works in much the same way as other business-level kinds like domain and system, which seek to collate and create boundaries around other elements of your architecture.

It's flexible Kind, designed to provide a neat wrapper or intemediary between existing concepts.

You might have:

• Domain > Product > System
• Product > Domain > System
• Domain > System > Product

The relationships and structure of this are up to you. If you have any problems or run into unexpected inability to articulate the model you're after then just let us know on Slack/Teams.

Enjoy. https://roadie.io/docs/getting-started/model-software/

🧘‍♂️ More (and more-flexible) relationships between entities

We have extended the number of permissible relationships between entities to cover different arrangements of entities.

We've also introduced the managedBy and managerOf relationships to provide referential integrity between entities that managed things and the entities themselves. People manage people and people manage services (distinct from ownedBy which is often not accurate at describing responsibilities within a system).

If you want to surface those relationships on an entity page you now can, using the EntityRelationsCard. This allows you to filter the displayed relations and only show the ones that are relevant for your use-case.

More info here about what is now possible. https://roadie.io/docs/catalog/showing-dependencies/

If you require additional flexibility in relationships between entity kinds that is not supported, you can request this to be added via the Roadie Support.

🏛️ Custom columns for your Catalog (in beta)

Ever wanted more control over the columns you have in the catalog? Well now you have it. We expose the setting of default columns and creating custom columns based on entity metadata, for each tab you create in the catalog.

We currently have this in beta, so let us know if you want some custom columns and we'll create them for you.

🎨 Type-based entity layouts

In the past you used to be able to set different entity page layouts for the Component kind based on types, but for all other kinds (Resources, Domains etc) it was limited to one per kind.

This doesn't really make much sense when you think about Resources for example, where an entity that is looking at a kubernetes cluster really needs different information from one looking at a AWS RDS instance.

So we changed it. Type-based layouts are now available for all kinds. https://roadie.io/docs/getting-started/configure-ui/

💈 New sidebar configuration options (in beta)

We've been adding new functionality to the Sidebar for a while in a piecemeal fashion, so we decided it needed a full revamp. You can now:

• Add sidebar items that start with the root pages (i.e. /catalog) resolving a particularly annoying issue where you were blocked from doing so because one of those root pages already existed.
• You can hide the Login to GitHub option entirely for all users.

More info here:

🔌 Plugins & Integrations roundup

• Incident.io plugin: we now expose the Humanitec plugin, so you can view deployment information insider Roadie. Fun.
• AWS account creation: last but by no not least, on the topic of catalog data sources this time rather than plugins - our AWS provider can now create AWS accounts and associated resources auto-magically. We also now ingest AWS tags as part of AWS automatic resource discovery. You can use tags to build relationships between newly ingested AWS resources and other entities in the catalog. Neat.

🚨 Controlling your Catalog with Role-based Access Control (RBAC)

We've been working on this for a while and it is with some fanfare that we announce Role-based Access Control (RBAC) on Roadie. 🤝✨🙌

Transparency of information is at the heart of Roadie (and one of the key philosophical principles of Backstage), but there are often valid reasons for gating access to information and maintaining the principle of least privilege.

For example, let's say you want to onboard Customer Service agents to your catalog so they can quickly find information about a service that may be experiencing some issues. You don't necessarily want to give Customer Service folks the ability to execute a scaffolder run or browse through Tech Insights Scorecards: that's just unnecessary and may even be confusing.

What you need in that situation is fine-grained control over your Roadie instance.

What is RBAC in Roadie?

• A new framework for access control in Roadie
• Every customer will have access to this new framework
• Some features are part of a paid add-on

And how does it work?

• Every part of Roadie is behind a Permission
• Roles are made up of sets of Permissions
• Users have Roles
• Admins can manage roles
• Admins can manage users

🎭 Role management

Several roles are available out of the box. They cover our existing access management setup as well as well as introduce some convenient shortcuts for new user groups that we've seen emerge recently:

• Admin
• Tech Insights Admin
• Maintainer
• Viewer

📚 Roles from a variety of sources, like your identity provider

The new permissions system will be allow you to send us roles from your identity provider as well as define them in the Roadie UI. This will mean the old GitHub Admin group is no longer required

Roles from all sources appear in the Role Management UI so you can understand where a user is inheriting a role from.

🆔 User management

You can then attach users to roles in the User Management UI and update the roles (and therefore permissions) that a user has access to on-the-fly.

As part of the switch-over to the new permissions system we run a background task to map the old roles system to the new one so there's no switching cost for existing customers.

🛃 Custom permissions and roles

The out of the box roles will get you so far, but any organisations need or want a higher degree of control over who can see what in their Catalog, who has the right permissions to trigger Scaffolder actions, and which elements of Tech Insights are or aren't displayed to certain users.

That's where custom permissions policies and roles step in.

The ability to create new roles and attach fine-grained permissions are optional paid extras and will cover use cases like hiding services in the catalog or controlling who can run individual scaffolder templates. We can setup trials/start discussions if you’re interested.

What's next for RBAC?

Individual entity permissions

At the moment, while we allow for a much higher degree of control over the catalog than before, we haven't drilled down to the individual entity level to say You need to have permissions-to-view-X-component` to view X component'. That's next.

That will mean you can:

• Control access to individual components or entities
• Control access to scaffolder templates and actions
• And create your own extremely specific permissions to target these entities

🙌 Backstage 1.26

We've now upgraded everyone to 1.26. This upgrade introduced some significant changes to the authentication system and lays the groundwork for the notification systems that lands fully in the next few versions.

🔌 Plugins & Integrations roundup

• Humanitec plugin: we now expose the Humanitec plugin, so you can view deployment information insider Roadie. Fun.
• Coder plugin: speaking of plugins, we also integrated the new Coder plugin. Tying neatly together Humanitec and Coder, Humanitec recently ran a webinar on how Coder thinks about integrating into Backstage which is worth checking out.
• AWS tag-based relationships: last but not least, on the topic of catalog data sources this time rather than plugins - we now ingest AWS tags as part of AWS automatic resource discovery. You can use tags to build relationships between newly ingested AWS resources and other entities in the catalog. Neat.

🗂️ Customising the Catalog UI

Your Catalog should match how you organise teams and build software. In the past, you've had to use Backstage nomenclature to model those concepts in the Catalog, but no more.

We've introduced Catalog Customisation in the Admin area so you can use your own naming conventions and concepts to define your Catalog using Kinds and Types and simplify the UI for everyone.

Add, edit, and reorder Catalog tabs to your hearts content.

Docs can be found here on how to get started.

💈 Sidebar slimming

In the same vein, we have also opened up support for full customisation of the sidebar.

This was possible in a limited capacity in the past, but you can now modify the whole kit and caboodle. This applies across your instance of Roadie for all users, allowing Admins to slim down the sidebar and reduce information overhead for everyone.

🌅 Surfacing metadata

We've also making customisation easier by introducing a EntityMetadataCard. The new card allows you to pull in any metadata you'd like from a catalog-info.yaml file and surface it on an Entity page. This allows you to pull in rich information on things like ownership or the last time an entity was updated, without the effort of writing a custom Typescript plugin to do the same job.

🌈 New default fonts, padding and colours

You may also have noticed Roadie has had a general facelift. This is part of our efforts to make it easier for users to consume visual information on any given Roadie page. This will be a long-running piece of work for us, but expect Roadie to just look and feel nicer in the weeks and months to come.

🙌 Filters for all!

We've also added Advanced Filters to all Catalog Kinds. This allows you to create a filtered, linkable views to share with colleagues. It has previously been possible to use Advanced Filters for a number of different Catalog Kinds, but now it's available for all.

Happy filtering :)

🔌 Plugin roundup

We've added a new Dynatrace plugin. This is the best plugin for customers of hosted Dynatrace services. You can also access the Dynatrace plugin for self-managed users, which has been renamed Dynatrace for Managed in line with Dynatrace naming conventions.

This is part of our efforts to more closely sync up with plugin maintainers like Dynatrace to make sure Roadie customer feedback reaches them. Their plugins get better, Roadie gets better, you get a better service from both. Win, win, win. 🤝

🤖 Custom Scaffolder Actions.

Templates and the Scaffolder get heavily used by our customers to democratise common tasks like adjusting cloud account budgets or making changes to some Terraform repos.

One of the historic limitations with the Roadie Scaffolder has been that customers were unable to create and use their own custom actions. This added some friction for more advanced Scaffolder users where they were limited by the Scaffolder Actions that we supported by us when they came to write a Template.

No more though! Roadie customers can now create their own self-hosted Custom Scaffolder Actions.

That means:

1. You can now write your own Scaffolder Actions to complete a task or use some custom CLI that Roadie isn't aware of
2. Then register those Actions in Roadie - as many as your heart desires.
3. Context can then be passed back and forth between Roadie and these Actions within any template you write.

Detailed docs can be found here to get started.

⽥ Custom Field Extensions

In the same vein, we have also opened up support for Custom Field Extensions for the Scaffolder.

You can write your own React Components and validator functions to handle the use cases not currently covered by the existing Templates. This allows you to customise a lot of the Scaffolder forms to your hearts content.

🔌 Plugin roundup

• The Dynatrace plugin is now supported on Roadie. You can use it to surface recent problems, error traces, and synthetics results for your services.

• The PagerDuty plugin has had a significant upgrade thanks to the fine work Tiago Barbosa is doing as part of PagerDuty taking maintainership of the plugin earlier this year. We've adopted the new plugin, so you'll see a slick new UI and some additional features as they rollout.

🤖 [Beta] Repositories have come to the Catalog.

We think a lot about catalog completeness at Roadie.

One of the most successful strategies we've come across to aid in getting components into the catalog is to surface what is in the catalog against a list of repositories. The gap between the two help to identify what is not in the catalog. Simple.

To help customers do this as easily as possible we've brought Repositories in from the cold and added them as a core part of the Catalog.

That means:

1. There is a new Repository tab in the Catalog
2. Repositories are auto-discovered for GitHub users. For other SCMs the Roadie API is available to push your repositories in.
3. Quick editing of Repositories inside the Catalog table itself.

📝 Set owners and other properties in the Catalog table UI (no yaml required)

We’ve expanded Decorators again to allow editing of Title, Owner, Type and Lifecycle for Catalog entities from the Catalog table itself.

You can now edit the title, owner, type and lifecycle of entities right in the Catalog table UI. All without yaml. Just click the pencil icon in the Actions section of whatever row you'd like to edit.

⛏️ [Beta] Auto-discover AWS Resources with our new provider

Our customers often want to represent AWS resources in the Catalog and have been using the Roadie API to do just that. There's a simpler way though, so we decided to build it. Behold: the Roadie <> AWS Provider.

It can currently be configured to pull in:

• Lambda functions
• EKS clusters
• S3 buckets
• DynamoDB tables
• EC2 instances
• and RDS DBs

Coming soon: AWS Accounts.

⏫ Upgrade to Backstage 1.23

We’ve upgraded Backstage to 1.23 for all customers:

• A fix to a vulnerability identified by us as part of our annual third-party penetration test. It related to the Scaffolder and didn't actually affect us, but we worked on fixing it with the core Spotify team to keep the rest of the community safe. More info here.
• A tweak to Nunchucks trimblocks that allows for creating control over templating in the Scaffolder, worked on by our own Miklos, one of the core Roadie team.

Full release notes for Backstage 1.23.

🔌 Plugin roundup

• The newEnd of Life plugin is now supported on Roadie. We've bought in both the frontend and the backend for this plugin, so it is able to read from repository files via the "source-location" annotation. It’s nice. We like it.

🤖 The Roadie API is now live for all users on our Growth Plan.

It includes:

1. The Backstage catalog API exposed, via token authentication for Roadie customers.
2. A scaffolder API for testing templates in continuous integration (via dry-run), triggering templates, and listing historical scaffolder runs.
3. A **Tech Insights API** that allows you to create, read, update and delete scorecards, checks and data sources.

At the moment API token generation is limited to Admins. If you are an Admin and need a token, simply navigate to Administration (bottom of the sidebar) and Account (in the tabs across the top). Give your token a name and click Generate Token.

🎨 More Decorators: you can now set owners and other properties in the Roadie UI (no yaml required)

We’ve expanded Decorators to allow setting many more fields on the Entities in your catalog.

• You can now override the owner, lifecycle and tags of Components right in the Roadie UI.
• Groups and all other Entity Kinds have also been expanded so that more properties can be set.

All without yaml. Just use the Decorate Entity feature in the top left corner of each Component page. Simple.

💯 Scorecards in the Catalog

How to surface scorecard information to teams is something we think a lot about at Roadie. A few months ago we added Rollups to help. This month we launched Scorecards in the Catalog to increase visibility and discoverability for teams. This is something with a long history (the original discussion in the Backstage community was way back in September 2020) and it's something we're hyped about.

🙅‍♂️ Tech Insights gets exclusions (in beta) and a new Facts list

We’re currently beta testing the ability to exclude entities from a Tech Insights scorecard and a check. This will give fine-grain control over the checks and scorecards you can create and pinpoint areas of your catalog.

We’ve also added a Facts list under the Data tab in Tech Insights to improve discovery of what you can and can’t do with Tech Insights data sources.

⏫ Upgrade to Backstage 1.21

We’ve upgraded Backstage to 1.21 for all customers. The biggest change since the last update at the end of 2023 is the new scaffolder UI with horizontal paging.

The upgrade to 1.21 also fixes a small but annoying bug that was pointed out by some customers: the scroll position of TechDocs pages now returns to the top when you navigate between different docs.

Our next version bump will be to 1.23 when that lands (hopefully later this month).

🔌 Plugin roundup

• The newPagerDuty plugin is now supported. PagerDuty took over support for the plugin in January, deprecated the old plugin and launched their own version. It’s nice. We like it.
• The Pulumi plugin is now supported. With it, you can bring infrastructure data associated with your Pulumi stack into Backstage.
• The Cost Insights plugin now has beta support. If you're interested reach out to one of the Roadie team on Slack, Discord or Teams.

Tech Insights

Rollups

Rollups aggregate Scorecard and Check data by team and department, up and down your organisational hierarchy, and let you add scorecard information to teams in the catalog. Learn more below in the Tech Insights section.

Here’s an example for a team called “engineering”.

From this, we can see that the engineering team is doing a great job of using PagerDuty correctly, but could do better at Dependabot configuration. If there are other teams reporting to this one in the org chart then their data will be rolled up into this view also.

Add the ScorecardResultForGroup and ScorecardResultsTableForGroup Cards to Group layouts to see results like this.

You can also see this data presented in report format on a single Scorecard, and dive into the data at different levels in the org.

Bug fixes and improvements

This month has been packed with improvements.

• We’ve got a built-in Data Source that scans for errors in CODEOWNERS files.
• We’ve got a built-in Data Source that ensures your branch protection is correct.
• The built-in Snyk Data Source has been updated to use the github.com/project-slug where possible.
• We fixed some rounding errors in our Check calculations.
• We fixed a labelling issue where there were two inputs called Type on the New Check form.
• We fixed a bug where regex comparison results were exporting incorrectly.
• We fixed a bug where selected annotations or labels in the filters of Scorecards couldn’t be deselected.
• Improved the performance of Data Sources which iterate over repos containing hundreds of thousands of files.
• The “is not blank” operator used to incorrectly ask for a value. Now it doesn’t.
• We had mislabeled the “Number” type as “Integer”. This is fixed.
• Markdown is now supported in Check descriptions so you can link to supporting documentation.
• The Proxy input is now a typeahead so it’s easier to find your favorite proxy.
• Scorecard rings now calculate in a more accurate way. A Component used to have to pass all checks on a scorecard to be counted in the ring. Now all checks that are passed will contribute to the score.
• We brought consistency and sanity to the positioning of the Re-run, Recalculate and Refresh buttons on Scorecards, Checks and Data Sources.
• Fixed some bugs which would prevent scorecards from showing up in the catalog in some cases.
• GitHub based Data Sources now filter out archived repos.
• Improved a bunch of help text sections on the New Data Source page.

Catalog

Decorators

Decorators allow you to easily add metadata to the stuff you track in your Roadie Backstage catalog. Check out the blog post for full details and to learn how to use them. One simple use case is to use decorators to add a Team Charter and some links to groups in the catalog.

Bug fixes and improvements

• API specs are now searchable. Start your endpoint searches with a forward slash.
• Renamed “Create…” in the sidebar to Templates. “Create…” was ambiguous.
• We removed Tools from the sidebar and moved its pages into Administration.
• The card used for displaying Links now hides itself from the interface when there are no links.
• Catalog table column visibility is now independently set for each Kind of Entity.
• Catalog tables can now display a links column.
• Entity Titles are now displayed in the catalog table instead of name when possible.
• Admins can now change the sidebar color in the Theme settings.
• We improved the Locations Log and renamed it to Administration → Entity Locations. You can also find it in the tabs of the Import page.

But measuring how well your Catalog is doing is something everyone needs and can help you tune your plan along the way.

What is a correct and complete Catalog?

Backstage is used by organizations large and small, and for different purposes. Thus, there’s no universal definition of “correct” or “complete” for the Catalog. But let me explain what I’m referring to in this context.

With “correct,” I mean the data in individual entities. It’s not enough that a component shows up in the Catalog. It must have all the information required for its type. For example, if a component of type service shows up in the Catalog but doesn’t have PagerDuty and API Docs annotations, it’s not rich enough to meet my definition of “correct.” Most importantly, all the meta-data and annotations in the entity must correspond to the entity for it to be correct.

With “complete,” I am referring to coverage of software assets surfaced in the Catalog. The absolute meaning would refer to having every single component, user, and other kind of entities in the organization reflected in Backstage. However, this is rare. More often, teams may define “complete” within a scope, such as having an entity tracked for every business-critical service, or covering a subset of teams in the Catalog.

Tracking Catalog correctness

Once you have more than a few entities registered in your Catalog, tracking how rich and correct the metadata in each of them is becomes impossible, especially if you’re welcoming dev teams to onboard their own components.

Here’s when you need Tech Insights. Tech Insights lets you check data points across your Catalog entries and summarize the findings in Scorecards.

Tech Insights is available as building blocks in the OSS plugin and as a fully-fledged Scorecards solution as a paid addon in Roadie. The OSS version will provide you the fundamentals so you can implement your own Scorecards solution, while Roadie offers a no-code UI with hundreds of pre-built checks. You can apply the ideas in this article using either of them.

What to check for?

With Tech Insights, the possibilities of what you can check for are pretty extensive. Thus, I recommend designing a schema that defines what constitutes best practices in your Catalog.

For example, for all components I expect a minimum to at least have a name, description, group owner, and a valid type to be set. To overview all of these criteria I need to define several checks in the component metadata:

1. Check that name and description are present: make Tech Insights check the metadata for entities is present for the fields you’re interested in.
2. Check that the component is associated to a GitHub repository: linking a component to GitHub unlocks several features in Backstage, thus, make sure to check that the github.com/project-slugannotation is present. You can also check if it complies with a form using regex.
3. Check that the type of entity is valid: the type field in Backstage can be an arbitrary string, which is prone to human errors. You can check that the types defined in all components do belong to a set of strings that you expect.
4. Check that components are labelled correctly: if you need your components to be tagged with their corresponding region or tier, using labels is the simplest way to go. Thus, you must asses which components are complying with the expected labels.

You can check any other detail from the entity metadata, and decide if the check applies for only a subset of them.

Once you have all the checks you want to apply defined, you can aggregate them in a Scorecard that keeps track of which entities are complying with all the checks.

Go beyond Catalog correctness

Happiness in your Catalog becomes a measurable objective with Tech Insights. However, with Tech Insights you can check more than the validity of entity metadata. For example, you can

• Overview Docker image migrations
• Track on-prem SonarQube metrics
• Or, enforce branch protection.

If you’d like to see what Roadie’s Tech Insights can do for your team, feel free to book a demo!

An Internal Developer Portal is as good as it tackles your teams’ unique challenges. While Roadie comes with dozens of integrations—such as PagerDuty, ArgoCD, and Sentry— your teams most likely rely on custom workflows, private systems, or in-house tools as part of their software development life cycle. Bringing those specific requirements into your Developer Portal can streamline your developer experience significantly.

For example, Lunar Bank built a dead-letter management plugin, while American Airlines centralized their permissions requests through a custom section of their Backstage instance.

Roadie offers tools to simplify the development and deployment of your custom plugins.

Getting Started scaffolder template

Register Roadie’s New Custom Plugin scaffolder template to jump-start your plugin development. The template will ask you a few details about your plugin and then create a new repository with a basic plugin structure and sample code.

Dev previews within your instance

Once you have your custom plugin running on your machine, you can get a live preview right within your Roadie instance. Your instance will automatically be updated with any code changes which will be applied when you refresh the page.

When writing a custom plugin for Roadie, you can use APIs provided as React hooks so you don’t have to deal with async requests or authentication at the plugin level.

You can preview all your plugin’s views within Roadie: pages, widgets, and cards. Furthermore, you can rely on preview entities to help you develop faster.

Deploying custom plugins to Roadie

Using the Roadie CLI, you can build and deploy your Backstage plugins and see them in your instance within a few seconds after you push them upstream. The simplest option for deployment is to let Roadie host your plugin, but you can also deploy your plugin to other services like Netlify or GitHub Pages.

The SOC2 Type 2 certification is a third-party audit that assesses your compliance over a period of time, to ensure your security, availability and confidentiality controls are operating as they should. Back in July 2022 we achieved our first SOC2 Type 2 report, but it was only a 3 month assessment period. This time around we were audited over an entire 12 month period.

A SOC2 Type 2 audit covers all the nitty-gritty details, like how we handle data, control access, respond to incidents, and keep a close eye on things. We're leaving no stone unturned when it comes to security and compliance.

We believe getting a 12 month SOC2 Type 2 certification shows just how committed we are to keeping your sensitive information safe and is a testament to the hard work and dedication of our entire team. We will continue to demonstrate this commitment year on year as we continue to comply with the SOC2 Type 2 standard. We also aim to expand our compliance to additional standards as we grow.

If you are an existing customer and would like to see a copy of our SOC2 Type 2 Report just reach out via any of the usual channels. We would be happy to share it!

How to use Decorators

Using Decorators is simple:

1. Visit an Entity in your Roadie Backstage catalog (e.g. a team, service or system).
2. Click the three dots in the top right corner and click “Decorate entity”.
3. Add links or annotations to the Entity and press Save.

Decorators you create are stored inside Roadie and not written back to the YAML file that backs the Entity.

This means that Entity metadata can come from multiple places for the same Entity. One annotation could come from YAML, and another from Decorators.

You can see where a specific piece of metadata is backed off to by clicking the three dots again and clicking “Inspect entity”. In this case below, the backstage.io/source-location is internal to Backstage and the other items are applied by Roadie Decorators.

Why we’re doing this

The introduction of Decorators is a slight deviation from the Backstage way of doing things, so it’s important that we explain why we’re doing this.

Auto-ingested sources need decoration

Backstage implementations frequently source the hierarchy of Users and Groups from a tool like GitHub Teams, Okta, or a Human Resources application like BambooHR. To support this, Backstage has a number of integrations into these tools.

These integrations will typically stream the hierarchy of Users and Groups into the Backstage catalog.

The problem with automatically ingesting Users and Groups is that Backstage users don’t get a chance to enrich their Group with information like links to Slack channels or a team charter. This leads to dead looking Group pages in Backstage.

Decorators give Backstage users a way to enrich their Group with the information that they want to show-off.

What this is not

We’re not introducing anything brand new in the Backstage ecosystem and we’re not introducing vendor lock-in.

There’s precedent

Backstage itself adds internal annotations to each Entity. These annotations are not written back to the YAML files. They are instead stored inside the Backstage database.

You can see some examples of this internal metadata here:

Roadie is simply piggybacking on this mechanism, to add the ability to store links and annotations.

We’re Backstage API compatible

We’re not introducing any Roadie-specific changes to the Backstage API or the spec for Backstage YAML files. We’re just making it easier to add values to the existing spec.

Entity Decorations are available via the standard Backstage HTTP API that we expose, so you can always write them back to YAML if you wish.

In the future, we will look to support “exporting” Decorators into any YAML file that backs the entity.

Catalog

New catalog page preview

You will shortly see a new catalog page roll out on Roadie. This update affects the main software catalog table, and the filters around it.

This new catalog table brings a number of enhancements:

1. More horizontal space for reading the table. We’ve moved the filters to the top so the table is wider.
2. Per user configurable columns. You can customize the table to show the info that’s important to you. We’ll soon be persisting column choice in your browser so you can pick up where you left off (this is coming imminently).
3. Kind specific columns. Groups used to have an owner column and Users were missing Display Name. We’ve tidied these up and introduced more sensible defaults.
4. New table features. Configurable densities and full screen mode make for slicker presentation. Filter highlights make it easier to find what you’re looking for.
5. Persisted filter choices. If you mostly work with Templates, we’ll keep you on the template page when you navigate away and back.
6. Sharable filter choices. Filters will be in the URL so you can share a link to a specific subset of data.

All this is building up to the ability to bring Tech Insights data front and centre in the catalog. We want to show scorecards in a column so you can drive more action around important migrations and software quality issues. More on this in the coming weeks and months.

Fixed: Disappearing Azure repos entities

We spent 3 weeks tracking down and fixing a tricky bug that would cause Entities discovered from Azure Repos to periodically and temporarily disappear from the catalog.

This is demonstrated by the wigglyness of the entities count before the fix, compared to how flat it is after.

It turns out the Azure APIs don’t return consistent results unless a sort is specified on the queries. Here’s the upstream fix we made to Backstage.

This is a really good example of the value Roadie adds. Are bugs like this how you want your Developer Experience team spending their time?

Tech Insights

Group check data by owner

Check results are now grouped by owner as well as by Component. This makes it easier to track down the team who own the most Components which are failing the check.

We’re currently working to expand this to scorecards, and to aggregate the data up and down the hierarchy of teams, so you can view it at any level.

New tutorials

We added 4 Tech Insights tutorials this month. Learn how to…

1. Find software that doesn’t have a valid CODEOWNERS file.
2. Find software that doesn’t have branch protection enabled.
3. Connect Tech Insights to SonarCloud to collect security hotspot data.
4. Connect Tech Insights to an on-prem SonarQube instance.

Bug fixes and improvements

• We fixed a bug where some Data Sources would fail with an Out of Memory error.
• We now support the YAML content type response when sending HTTP requests in Data Sources.
• Data Sources can now send POST requests for GraphQL APIs and other use cases.
• We rolled out a new version of our broker to patch a security vulnerability.

Scaffolder

Certified label for scaffolder templates

You can now add the Certified label to scaffolder templates to designate them as Platform Team approved and ready for use.

Just add the certified annotation to make this work.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: my-template
  annotations:
    roadie.io/certified: "true"

Bug fixes and improvements

• The scaffolder now supports a task to open a pull request against Azure repos.
• We updated our scaffolder docs page to include more examples and APIs.

With Roadie Tech Insights, software engineering teams can use Scorecards to monitor their software assets in the Backstage catalog and make sure they meet the quality and compliance standards they have set. Scorecards are based on Data Sources and Checks that the user can customize through a UI.

Roadie Tech Insights comes with more than a hundred different types of checks across dozens of data sources like GitHub, Datadog, and Snyk (plus, you can add custom sources). The feature release was covered by DevOps.com, TFiR, and BENZINGA.

Roadie was founded in 2020 to help organizations boost developer productivity through a Developer Portal. The company raised US$3.7 million in a seed funding round led by Boldstart Ventures and Firstminute Capital.

Back in 2021, Roadie released an open-source Tech Insights plugin with primitive APIs so that any Backstage adopter could build their own Scorecards functionality. A few enterprise Backstage adopters like HP and Lunar Bank have implemented their own solutions based on this version of the plugin. However, the open-source Tech Insights requires each team to design its own UI and implement its Data Sources and Checks, on top of managing integrations, security, and databases.

The new fully-fledged version of Tech Insights—available only to Roadie customers—features more than a hundred facts that you can check against across dozens of data sources like GitHub, Snyk, and PagerDuty. For example, Martin Froehlich, vice president of engineering at SumUp, is using Roadie and Tech Insights to “promote and track adoption of supply chain security and code analysis tools like Dependabot, CodeQL, and others, across all of our production service repositories.”

Roadie Tech Insights helps engineering teams build a culture of quality and accountability Using Tech Insights, teams are nudged towards improved software quality over time. Antony Rinaldi, head of architecture and application platform at Baillie Gifford, is using Roadie Tech Insights to “promote and track adoption of security tools with our 250 developers.”

He also added that Baillie Gifford has “created automated checks and scorecards that help us understand which teams have adopted the tools letting us understand how successful our rollout initiative is. We’re excited to expand it to more use cases over the coming months.”

Roadie Tech Insights is a natural development to make the most out of a Software Catalog and keep quality under control.

Most Dev Portals rely on putting metadata files (YAML, Terraform, etc) into the services so they can be updated by the teams that work on them. The more friction you cut from the process of creating these metadata files, the easier it is to convince people to create them.

Backstage’s Scaffolder can make the software onboarding a one-click experience that gives Developers a chance to try out your Dev Portal and easily add their own services to the Catalog.

In this article, I’ll show you how you can write a software template that prompts the user to tell you about their service and opens a PR on their repository. Once that PR is merged, the Catalog will pick up the service automatically (if you have auto-discovery enabled).

Onboard your service with a few clicks

The experience you’re after will let developers onboard their service by filling in a few inputs. The Scaffolder then takes care of generating a catalog-info.yaml file and opening a PR against the service's repository.

In the first section, you’ll ask for basic information about the service. In the second one, you’ll prompt the user to locate their repo and associate it with an owner. And finally, you’ll ask for integration details such as ArgoCD’s app name or PagerDuty integration key.

Once you’ve collected all the information, your software template will generate a catalog-info.yaml file and open a PR against the service’s repository.

Writing a scaffolder template

Software templates in Backstage have two parts: parameters and steps. The parameters define the inputs you want from the user. The steps are actions—like cloning a repo, editing files, creating an AWS secret, or making an HTTP request —that are run one after the other. In this section, you’ll learn more about how both parts can be implemented for an onboard service template.

Defining parameters

Let’s take care first of the parameters. Parameters can be organized into sections in the UI. In this case, you want to have three sections: one for general information, one for the repository, and one for additional details. Here’s what the code that describes the form presented in the last section could look like:

parameters:
    - title: What is your service about?
      required:
        - name
      properties:
        name:
          title: Service name
          type: string
          description: Human readable name. We'll generate a dasherized version from it.
        description:
          title: Service description
          type: string
        owner:
          title: Service Owner
          type: string
          description: Owner of the component
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group
    - title: Where is your codebase?
      required:
        - repoSlug
      properties:
        repoHost:
          type: string
          default: github.com
          ui:widget: hidden
        repoOwner:
          title: Repository owner
          type: string
          default: roadiehq
          enum: ['roadiehq', 'jorgelainfiesta']
        repoSlug:
          title: Repository slug
          type: string
    - title: Integrations (optional)
      properties:
        argoAppName:
          title: Argo CD App Name
          type: string
        pagerdutyKey:
          title: PagerDuty integration key
          type: string

The input that the Scaffolder generates is based on the type of property. In all of the cases in this example, we’re dealing with strings but you can also specify numbers, objects, and arrays. You can also specify a component to be rendered as the input with ui:field. For a comprehensive list of these options check out our Scaffolder documentation.

You’ll want to customize the form according to how you want to register services in your Catalog. For example, I’m hard coding the repository’s host and providing two owner options but you may have more than one host option. You could also use a dynamic select box to ease up the selection of repositories.

To help you get the form right, Backstage (and thus, Roadie) come with a form editor that you can find under /create/edit → Template Editor. It’s quite handy, specially as your form becomes more complex.

Defining steps

Now, let’s review the steps side of the template. You’ll need two steps. First, you’ll fetch an existing YAML file and replace placeholders within it with the users’s values. The resulting file will be available in the Scaffolder workspace’s root with the same file name. From there, it’ll be possible to open a Pull Request against the target repo with the content of the Scaffolder workspace, which will be a catalog-info.yaml file that describes the service.

This code shows how the steps look like:

steps:
      - id: fetch-template
        action: fetch:template
        input:
          url: ./skeleton
          values:
            name: ${{ parameters.name }}
            description: ${{ parameters.description }}
            owner: ${{ parameters.owner }}
            repoOrg: ${{ parameters.repoOwner }}
            repoSlug: ${{ parameters.repoSlug }}
            argoAppName: ${{ parameters.argoAppName }}
            pagerdutyKey: ${{ parameters.pagerdutyKey }}
      - id: create-pull-request
        name: create-pull-request
        action: publish:github:pull-request
        input:
          repoUrl: ${{ parameters.repoHost }}?owner=${{ parameters.repoOwner }}&repo=${{ parameters.repoSlug }}
          branchName: onboard-to-catalog
          title: Onboard service to Catalog
          description: This PR adds a meta data file about this service so that it can be registered in our software catalog.

Let’s unpack what’s going on in each step. In the fetch-template, I’m loading a relative path that contains a file with placeholders formatted with nunjucks templating. The file is a catalog-info.yaml that looks like this:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.name | replace(" ", "-") | lower}}
  title: ${{ values.name }}
  description: ${{ values.description }}
  annotations:
    github.com/project-slug: ${{ values.repoOrg }}/${{ values.repoSlug }}
    {%if values.argoAppName %}argocd/app-name: ${{values.argoAppName}} {% endif %}
    {%if values.pagerdutyKey %}pagerduty.com/integration-key: ${{values.pagerdutyKey}} {% endif %}
spec:
  type: service
  owner: ${{ values.owner }}

You can manipulate strings using filters, use conditional blocks, and pretty much any other templating option available in nunjucks.

Regarding publish:github:pull-request, the only thing worth mentioning is that repoUrl doesn’t look like a familiar URL. That’s because it’s a standard reference used across the Scaffolder actions rather than an actual repository URL.

To help test the steps inputs and outputs, you can dry-run your template with /create/edit → “Load Template Directory.”

Conclusion

Minimizing the friction to using your Catalog will improve the adoption rate. Plus, it can help you provide developers a clear first touch point to start getting familiar with the Developer Portal that you’re building for them. You can find the complete template in our software templates repository.

The people make the Unconference

We had over 95 attendees from around the globe, with people connecting from cities in the US such as San Diego, Chicago, Omaha, and Orlando. We also had the presence of peeps from Buenos Aires, Madrid, Paris, London, and Cambridge, and jumped to India and Singapore after passing by Kazakhstan and Israel.

Most of the people who attended are on a growing internal adoption stage in their Backstage journey (40%), followed by a large group of teams getting started with their PoCs (34%). The Unconference also go input from people who have achieved weekly active usage of their Portal across the board (6%) and folks who have optimized usage for a subset of their users (7%).

Making Documentation easier

Two out of the eight topics chosen by the community revolved around TechDocs. The first discussion was more higher-level and focused on sharing how teams are using Backstage to bring their varied sources of documentation under the same roof. The second topic had a more technical perspective on how to provide tooling to developers writing TechDocs so it’s easier for them to get their docs into Backstage.

Infrastructure and contributions

Using the Scaffolder to spin up infrastructure is a common use case, which got some interesting takes regarding how to manage the templates and the possible race conditions as you manage Terraform files and various PRs. Another topic of discussion was opening up the Portal to receive plugins from other teams and how to define the boundaries for integrations and outline a governance approach.

Recordings

Due to the nature of the unstructured conversations that occur in the Unconference, not all sessions resulted in a recording that could be too interesting to re-watch. We’re working to take pieces from at least two of the sessions to make videos that might be helpful to drive the Backstage discussion further. Stay tuned, we’ll be uploading them to our YouTube channel soon!

Why Attend Unpacked?

1. Cutting-Edge Insights: Unpacked brings together a prestigious lineup of industry leaders who will delve into the latest trends, strategies, and technologies shaping the world of software delivery.
2. Software Distribution at Scale: Scaling software delivery is a complex task, and Unpacked aims to demystify this process. The event will explore strategies and best practices for distributing software.
3. Securing Open Source Dependencies: With open source software playing a crucial role in modern development, ensuring its security is paramount. Unpacked will address this challenge, offering attendees valuable guidance on securing open-source dependencies and mitigating potential vulnerabilities.
4. Reducing Complexity and Cost: Complexity and cost often go hand in hand with software delivery. Unpacked will provide insights into simplifying processes, reducing unnecessary complexity, and optimizing costs associated with software distribution, enabling organizations to achieve greater efficiency and ROI.

"Unpacked" is set to be a groundbreaking virtual conference that will equip attendees with the knowledge and tools they need to excel in software delivery. With an exceptional lineup of industry leaders and a focus on scaling, security, and reducing complexity, Unpacked is a must-attend event for DevOps professionals and engineering leaders. Roadie is proud to be a sponsor of this transformative event, demonstrating our ongoing dedication to advancing software delivery practices. We look forward to virtually connecting with fellow attendees and exploring the future of software distribution together at "Unpacked"!

By far, KubeCon + Cloud Native Conf EU 2023 has been the most impressive since the re-start of the conference series after the p*ndemic. With 10 thousand people in attendance and fascinating talks, being unable to enter the room—even if you arrived 10 mins earlier—became commonplace.

Let’s address the elephant in the room: there was no Backstage booth in the Project Pavilion, as there was in Detroit. A series of miscommunications caused this unfortunate issue, but after talking with other members of the community, including the maintainers, I can assure you: this won’t happen again!

Despite that, I was lucky to have talked with many people in the community, including OSS partners RedHat and VMWare, current Backstage adopters, and people who are just getting started with the framework.

If you’re a Pragmatic Engineer subscriber, make sure not to miss out on this one. Gergely goes through what a Developer Portal is, how to get started, and even compares closed-sourced alternatives. Check it out!

Martina Iglesias Fernández, CTO of Roadie, saw the inception of Backstage from within Spotify and shared her story with The Pragmatic Engineer. At the time, she was a lead backend engineer at Spotify, where she saw the pain points that derived the need for an Internal Developer Portal. In the column, she explains in detail how Backstage came from an internal tool called System-Z.

The Roadie team shared other experiences with Gergely, including information about Backstage’s main features and the typical adoption process from MVP to org-wide release.

Imagine you’re browsing your software catalog looking for a library that you can use for setting up your Go API routes or to find patterns others have used to set up a test harness for their Java apps. In these cases, it would be helpful to be able to filter entities by language.

Since GitHub already compiled this information for you, you only need to bring it into your Developer Portal. Now, Roadie can do this automatically: we’ll label or tag—according to your preference—your entities with their associated language.

If you’re a Roadie customer, you can get your Catalog to start tagging your entities with the corresponding languages by switching the feature on in Administration > Settings > Catalog.

From the service’s entity page in Backstage, you can get a glimpse of the recent workflows runs, the service’s dependencies, and other issues like code quality, vulnerabilities, and documentation without jumping around platforms. Along with the ownership information, you’ll be up to speed to address this incident and contact responsible parties more easily. Additionally, other colleagues who work with this entity will get visibility on the ongoing incident and get ready to collaborate to resolve it.

Thankfully, Backstage has a handful of plugins that integrate incident managers into your Internal Developer Portal. Below is a list, in alphabetical order, of the incident management plugins you can integrate into your Backstage internal development portal:

FireHydrant

With the Firehydrant’s Backstage plugin, you can manage your incidents within Backstage. Teams can stay organized and quickly identify information about services like active incidents and healthiness analytics. The plugin includes an entity widget so you embed PageDuty’s information on the entity’s overview page.

To install it on a self-hosted Backstage instance, check out our FireHydrant Plugin guide. If you’re a Roadie customer, you don’t need to install it; just set it up in your Admin panel.

OpsGenie

The OpsGenie Backstage plugin offers two options: an entity widget and an additional standalone page. The entity widget shows the alerts for its corresponding entity on the overview page:

Additionally, the OpsGenie plugin comes with a standalone page where you’ll get to see who’s on call and an aggregated list of alerts happening in the entities registered in Backstage.

To install this plugin on a self-hosted Backstage instance, follow our OpsGenie Plugin Guide. If you’re a Roadie customer, check out the no-code guide to OpsGenie.

PagerDuty

With PagerDuty’s Backstage plugin, you can view the ongoing incidents related to an entity, as well as who’s on call. Additionally, you get handy buttons to create an incident right from the entity’s overview page.

To install it on a self-hosted Backstage instance, check out our Pager Duty Plugin Guide. If you’re a Roadie customer, set up PagerDuty from your Admin panel.

Rootly

The Rootly Backstage plugin is the most generous: it provides you with three options for viewing incident information in your Internal Developer Portal. First, there’s the standard entity widget for the overview page that every plugin offers:

You can see the ongoing incidents right on the entity overview page, and gives you handy links to create an incident or see a more detailed list. Second is a dedicated entity tab to get more details about the ongoing incidents for the associated entity:

And, Rootly also offers you a dedicated page where the incidents from across all entities are aggregated:

To install this plugin in a self-hosted Backstage instance, follow our Rootly Plugin Guide. If you’re a Roadie customer, set up Rootly in your Admin panel.

Splunk On-call

The Splunk On-call (formerly VictorOps) plugin for Backstage provides you with a widget for your entity overview page. The plugin will show a list of ongoing incidents and provide links to open an incident or acknowledge/resolve one right from the entity page. If you want, you can also set the widget to read-only so that no actions can be triggered from Backstage.

To install this plugin in a self-hosted Backstage instance, check out the plugin’s README. If you’re a Roadie customer, support for this plugin is being implemented at this very moment as requested last week by a customer.

And that’s a wrap! Setting up your incident manager with your Backstage Internal Developer Portal can help you manage incidents and prepare your teammates to collaborate when needed. If you know of an incident management plugin that I missed, please let me know through Twitter or LinkedIn! I’ll be pleased to add it to the list.

The problem

The remote code execution (RCE) vulnerability was possible due to a known issue in the vm2 library used in the Scaffolder, which has been patched since Backstage 1.5.1. By overloading definitions through a software template, the researchers manage to create a function outside the Scaffolder’s sandbox context that allows them the execute arbitrary code in the instance.

Furthermore, the researchers pointed out that Backstage by default doesn’t provide authentication for backend requests. This allowed unauthenticated actors to access the Scaffolder, and therefore, exploit the vulnerability from outside the Developer Portal.

Roadie customers are not affected

Roadie customers were running on Backstage 1.8 at the time of the vulnerability disclosure and were patched for this vulnerability shortly after Backstage 1.5.1 was released because the team keeps a close eye on CVE notifications.

Furthermore, due to Roadie’s architecture, the risk from this vulnerability was greatly mitigated for Roadie customers. Roadie executes templates on a transient ECS task with access to scoped and temporary credentials required for the execution of the template instead of the default execution strategy.

Also, Roadie provides authenticated access to both frontend and backend requests, which means no unauthenticated actor could have accessed the Scaffolder in the first place.

Upgrade your instance ASAP

If you’re running a self-hosted Backstage instance and still use a pre-1.5 version, you’re facing a vulnerability with a 9.8 CVSS score, which is the most severe for exploitability and impact.

If you don’t want to bother to run upgrades again, switch over to Roadie! We’ll keep your instance safe through regular upgrades and extra security layers. Request a demo!

In the most recent Tech Radar, Thoughtworks moved Backstage to Adopt in their Platform quadrant. This means that their advisors “feel strongly that the industry should be adopting” Backstage. And in fact, Thoughtworks offers Backstage as part of their digital transformation offering to their customers.

On the other hand, Red Hat announced that they will start actively participating in the Backstage community. They will be contributing to the framework to support OpenShift and improve the Kubernetes experience in Backstage.

Additionally, VMWare has been a commercial partner of Backstage offering it as a UI for a security product in their Tanzu suite. VMWare has committed an Open Source team dedicated to contribute back to Backstage Open Source.

Previously this year, Gartner reported on Backstage as a solution for Developer Portals. It also mentions Roadie as a solution to tackle the complexity of adopting the framework via de self-hosted route.

Finally, the Linux Foundation is also betting on Backstage after having witnessed the success it has brought to most adopters in the Cloud native space. The Linux Foundation is investing in creating learning resources, starting with an Introduction Course to the framework.

For Roadie customers, this news means you’re about to get all the benefits from the substantial contributions of a robust community without having to do anything. It also shows you’re on the right track by adopting Backstage through Roadie!

Monday: BackstageCon

During the first conversations about setting up a Backstage-exclusive conference, the CNCF event organizers said we could aim for 50-100 attendees because it’d be the first time. Well, BackstageCon ended up being the largest co-located event at KubeCon + Cloud Native NA 2022 with 150+ attendees!

The event was co-hosted by Suzanne from Spotify and Martina from Roadie. They did a fantastic job at curating the line-up, all talks are available on Youtube so check them out!

Roadie had a very orange table at BackstageCon, where we greeted everyone.

Of course, we enjoyed hanging out with each other after the conference. Here’s a picture of Roadie having dinner with Spotify folks, including the Backstage maintainers we all love.

Tuesday: Backstage Project Meeting

At first, a 4-hour meeting to talk about Backstage seemed a bit much. But once we got started, it turned out to be enough to only scratch the surface on topics like the sources of truth for the Catalog, frontend performance, maintainership, and adoption challenges. Having adopters, maintainers, and partners in the same room made ideas flow endlessly and lead to actionable tasks to make it easier to adopt and contribute to Backstage in the near future.

In the evening, Frontside and Roadie had what is, without doubt, the best food anybody attending KubeCon enjoyed, at Yemen Café.

Wednesday-Friday: KubeCon

Roadie had a booth at KubeCon, where we explained Backstage and how we offered a managed version to people attending the event. We also were giving away organic cotton Backstage t-shirts, hand-printed in Barcelona. We heard they were the best t-shirts of the conference in terms of comfort!

We had the wonderful chance to meet and talk with our customers attending the event. Here we are with MyFitnessPal:

See you next time!

We all had a blast last week in Detroit and aligned on how to take Backstage further as it gains popularity. Next KubeCon is in Amsterdam from 17th to the 21st of April, 2023. Hope to see you there!

The Open Source version of Backstage is used by industry leaders such as HP, VMware, and Expedia Group. But, as highlighted by Gartner, it requires significant effort and dedicated staff to stand up and maintain. Instead of self-hosting Backstage, dozens of scale-ups like Netlify, Snyk, and MyFitnessPal have adopted it through Roadie’s managed option. Now, Roadie is expanding its offering by introducing Tech Insights, which doesn’t have an equivalent in the Open Source Backstage.

Tech Insights lets Roadie users create Scorecards to keep track of quality standards within their organization. This is useful for assessing software maturity and security compliance across all services.

*Roadie users can define the criteria to be measured in their services via a Scorecard. * Given that Roadie Backstage users already have all their software assets registered in their Software Catalog, extracting insights is a valuable next step in any Developer Portal.

*Roadie users can overview the health of their ecosystem across services *

Roadie is currently working with a handful of design partners to develop Tech Insights to ensure it brings out the most value for leadership and developers.

Previously, we relied on Backstage’s default behavior for keeping the catalog up to date. This was a pull-based approach where Roadie polls your GitHub and kept the catalog in sync.

By default, the polling interval was set to 2 minutes. This is a long time to wait while you are in the middle of editing your scaffolder templates and still figuring things out.

Polling large catalogs would also result in many requests being sent to the GitHub APIs. This could result in rate limiting and a degraded user experience.

With this release, we are utilizing the GitHub webhooks API to get notifications when you change your Backstage YAML files.

We also added a new feature to the GitHub integrations settings page to be able to manually trigger a sync with your GitHub repos. This is useful if you added a catalog-info.yaml files to a repository where you did not have the Roadie GitHub app installed.

The benefits for Roadie users

We believe this new webhooks based approach brings a number of benefits:

1. We eliminated the usage of Location entities for discovery. We can spare the additional fetches for the whole organization repositories for every configured github-discovery Location entity.
2. It results in an almost immediate reaction from the catalog when you push something to your configured branches.
3. Now you can safely rename your catalog files in your GitHub repo. (This will result in a deleted filename for the old file and an added one for the new file)
4. It can refresh your API entity when the referenced e.g. openapi/grpc file is changed (if it is hosted in GitHub)

Read on for more technical juicy details about the implementation.

Tech Stuff

Let me walk you through this journey to implement and roll out instant updates for Roadie users!

The Past

Before webhooks, we relied on the default implementation of auto-discovery from Backstage. This used the processing loop, and the provided processors to ingest entities from GitHub organizations. We used the GithubDiscoveryProcessor from OSS Backstage.

It works like this:

• This processor is configured and added to your catalog builder.
• This processor is evaluated on every entity when it is processed that should this run or not.
• This processor will execute its logic when an entity is processed that is a Location entity and its type is github-discovery
• It fetches all of your repositories from your organization then creates an optional Location entity for every repo.
• These Location entities then will be processed and they are going to fetch the files and emit the entities that they found in the target paths.

This processor has 2 main drawbacks:

1. It is tied to the processing loop so you cannot set a different interval for it. This is a problem if you’re being rate limited by GitHub. There is no option to lengthen the loop duration.
2. It makes unnecessary requests towards GitHub API by fetching all of the repositories every time it runs.

The present

We built a Roadie-specific entity provider which can act on the incoming GitHub webhooks.

It uses your configured Roadie Backstage GitHub app to forward the GitHub push events from your organization’s repositories to our servers.

The GitHub webhooks API sends Roadie the modified, added, deleted array of files. This indicates what happened in this event. The provider differentiates the modified and added/deleted events.

When a modification event happens:

• we get the event from GitHub
• Get all the modified filenames in this push event
• trigger a refresh on the Backstage database

We will try to refresh with every filename and let the database decide if there was a matching entity to schedule the refresh. This was implemented this way because it enables us to provide an instant refresh on API entities when a referenced $text placeholder’s value is managed in GitHub and you change that open API descriptor we will refresh the API entity that it belongs to.

When the event contains additions/deletions:

• Get the event from GitHub
• Construct a set of filenames for added files
• Construct a set of filenames for deleted files
• Filter these based on the configuration
• Create an optional Location entity for these files with proper location annotations

This path is pretty similar to the previous discovery. We are creating Location entities where the location’s spec.target will point to the file that we got in the GitHub event. For every added/deleted file that matches your configuration and we rely on the processing loop fetch and emit the actual content of the file.

We removed the polling for entities, and we disabled the possibility to add github-discovery Location entities to the catalog.

Some things to iron out

With the current implementation, some edge cases can be confusing or not work as expected.

Multiple entities in one file (catalog-info.yaml)

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: valid-same-file-entity-1
spec:
  type: library
  owner: user:kissmikijr
  lifecycle: production
---
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: valid-same-file-entity-2
spec:
  type: library
  owner: user:kissmikijr
  lifecycle: production

This approach you can cause undesired behaviour if you end up with a validation error in one of your Entities.

If this happens the catalog will create a location entity which will point to this file, however, the processing of the entities won’t finish, this means Backstage will not store the correct information to be able to trigger refreshes and even though you fix your validation errors in the next commit you’ll need to wait for the regular processing loop to handle the refresh.

Registering an Entity via the /register-existing-component page

In this case, because this entity was not added to the catalog via the webhooks, when you delete this file from your GitHub repo the webhook won’t be able to remove it.

Updating this entity will be instant.

Using the Location kind

If you used Location entities before in your repository to register this and let the processing loop find the other targets.

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: roadie-backstage-plugins
spec:
  targets:
    - ./plugins/**/catalog-info.yaml
    - ./utils/**/catalog-info.yaml

The automatic refreshes will not work on the target entities.

This is a shortcoming of the open source implementation of the refresh handling. It is planned to be fixed. Until then, the best advice is to ditch the top-level Locations and configure the targets in the /administration/settings/integrations/github configuration page.

In this case, you’d add two entries to the Targets:

# Entry 1
https://github.com/RoadieHQ/roadie-backstage-plugins/blob/-/plugins/**/catalog-info.yaml 

# Entry 2
https://github.com/RoadieHQ/roadie-backstage-plugins/blob/-/utils/**/catalog-info.yaml 

To configure your targets check out the documentation.

Our customers will be able to use Tech Insights to spot unloved services in production, find teams who need more support in order to produce top quality software, and help engineering organizations understand where the bar is.

Roadie users will be able to create scorecards which define what it means to build quality software within their company. They will be able to apply these scorecards to software in the Backstage catalog. They will see reports to help identify software which needs improvement, and they will be empowered to work with owners to deliver said improvements.

Tech insights will be the first major proprietary Roadie feature. It is in development as I write this post, and we are already working with design partners on rollout.

Throughout our work with dozens of customers, it has become clear that implementing Backstage and cataloging software is only the first step on a longer journey for most organizations.

If you would like to supercharge your Backstage experience with this feature, please request a demo of Roadie.

Measuring the quality of software

The first step towards determining whether or not software is of a high enough quality, is to first define what “quality” means in your organization.

Quality is an aggregate measure which accounts for many dimensions. Quality software is usually easy to operate in production, easy to develop, secure and compliant, amongst many other attributes.

Each of these individual measures is composed of its own factors, each of which is likely tracked in a different tool.

Here are some simple attributes you might check to get a measure of operability:

1. Software which is easy to operate in production usually has adequate uptime. This might be measured against a service level objective recorded in Datadog.
2. It is usually connected to a logging tool. This might be Datadog again, or it might be a self-hosted ELK stack.
3. The software should have an on-call rota associated with it. This would be stored in Pagerduty.
4. It should have an owner who is responsible for keeping the software up-to-date and deploying it frequently. This would be stored in the Backstage catalog.
5. it should have runbooks defined. These might be in a Confluence document.

Once we factor in a few of these attributes, our measurement of good software starts to look more complex.

Run the same process for scalable, secure, easy to develop, compliant and any other top level factors and you quickly start to realise that you have to check 40+ factors and integrate with 20+ tools to determine if a piece of software is high quality or not.

Is there enough quality?

The next level of complexity comes from attempting to determine whether or not the current level of quality is “enough”.

The required level of quality will vary depending on the purpose and exposure of the software in question. Take uptime for example. Your payment processing service might need 5 nines of uptime, while some internal reporting tool might be down for 5 months before anyone notices, and that might be ok.

To account for this in the measurement, you will need to attach different expectations to different software components.

This is frequently done by categorizing and dividing software into tiers. Software in tier zero might be mission critical and require the highest levels of quality. Tier 3 might be far less important from an overall business standpoint, and will be subject to less stringent requirements.

We can contextualize the measurements we defined above by defining quality levels, as shown in the diagram below.

Driving improvements in software quality

Invariably, when you define software excellence and compare the current state of the world against it, you will find gaps. This is a good thing. The next step should be to encourage the improvement of this software.

There can be a multitude of legitimate reasons why a piece of software doesn’t meet a given quality bar. Teams will constantly streamline their priorities to try to match those of the wider business, and quality will sometimes suffer.

The important thing is that this quality gap is visible. We may actively choose not to rectify it, but we should at least be able to spot it and have that conversation.

Teams should be able to independently make realizations like “everyone else in the org seems to have passed the security level 1 bar… perhaps we should think about doing some security work soon”.

We believe Roadie Backstage has a role to play by increasing visibility into organizational software quality. We should nudge people towards increasing software quality, in a thoughtful and conscientious way.

We’re working on Tech Insights at Roadie

1. You will be able to write Checks which continually test the software in your catalog in an automated way. These Checks can be anything from “Does the software have an SLO set in Datadog” to “Is the log4j version semantically greater than 2.16.0”. Checks can leverage custom data or data from the APIs of standard SaaS tools that you already use.
2. You will be able to group Checks into Scorecards, and target those scorecards to subsets of the software in the catalog. Perhaps you want to target Tier 0 and Tier 1 Java services. Perhaps you want to target Python services in the data science org.
3. You will be able to slice and dice reports of software quality so that you can find software which needs more support in order to meet the quality bar.
4. Teams will be carefully and conscientiously nudged towards improving the quality of their software over time.

We’re already rolling out early versions of this software to our design partners. If you would like to learn more, or participate in the betas, please request a demo of Roadie Backstage today.

From the start, we built Roadie with security, availability, and privacy as fundamental values, and we recognize them as essential to our success. Our team is made up of people who have worked in large enterprise companies and scale-ups such as Workday, Spotify, and Intercom, so we are no strangers to enabling and ensuring good security practices. We understand that if you build these processes early, they will grow with your company and help you scale securely and reliably.

We have a set of mature and robust security and availability practices at Roadie and wanted to validate them against industry standards. We see this achievement of SOC2 Type 2 compliance as a milestone in our ever-improving security journey.

What this means

A SOC2 Type 2 report is one of the most well-known IT security and compliance auditing accreditations. It is highly comprehensive: it doesn’t look at any one business area in isolation.

An accredited external audit firm scrutinized Roadie’s engineering practices—such as our database security controls, monitoring and alarming, and testing methods—as well as the ecosystem within which these practices live. Meaning, that we train our staff, we care about who we hire, we restrict access to data, and we review all vendors that we choose to use.

Why is this important?

Simply put, we want this report to give our customers even more peace of mind when choosing to trust Roadie with their data. The SOC2 Type 2 report shows that we have opened our doors to a third-party and allowed them to test and scrutinize our security and availability practices.

As the old but fitting adage goes, “trust but verify.” Our goal is to provide you with confidence that we have robust, mature, and industry-standard practices that are monitored and updated frequently.

A milestone, not the end goal

The comprehensive nature of this audit affirmed our confidence that we are set up with excellent foundational security and availability practices which we can continue to build on as we scale.

We will continue to keep our compliance with SOC2 up to date, and we will undergo an annual audit to test our SOC2 compliance. We also aim to expand our compliance to additional standards as we grow.

If you would like to see a copy of our SOC2 Type 2 report, reach out to legal@roadie.io.

Until recently, the catalog has been the primary way to interact with Backstage. You could pick a software component from the list, and quickly get a sense of how that component is doing, who owns it etc.

But a single software engineer typically has to manage and track multiple components at once and it seems inconvenient to need to visit the Overview page of each individual component in order to get a sense of what’s happening. Why can’t the info be co-located in one place?

Software engineers don’t just interface with software either. They also need up-to-date information on the latest goings on in their organization, and they even have to (begrudgingly 😃) attend meetings sometime. Shouldn’t Backstage also be plugged into this part of an engineers job?

The pulse of the team

Tackling these problems is the remit of the Home plugin. It’s a Backstage page which you can visit first thing in the morning to take the pulse of the team. It’s a page for bringing together information from many different systems, in a way which is most relevant to you!

Here’s how the Home plugin looks for me in Roadie Backstage:

As you can see, important information from a number of sources here.

1. I have quick access to the components that I own, and the entities that I have “starred” in the catalog. I frequently use these components when I’m doing Backstage demos, so it’s useful to have them within easy reach.
2. My calendar is available, thanks to the Google Calendar plugin created by Alex Rybchenko from Box (#9719). I can even click the zoom links to go directly to a meeting.
3. I can see my open review and pull requests on GitHub. If the team need me to review something, I can see that any time I log into Backstage.
4. Finally I have a Roadie News widget. This displays content from a static markdown URL hosted on GitHub. It can be used to share organization news or information about upcoming events. We’ve also seen our customers use it to bookmark links to important pages outside Backstage.

Where this is going

The Home plugin is super new but we’re already seeing amazing demand from our customer base.

At Roadie, we’re hard at work producing more widgets for the Home page. We’d love to use this space to display the Jira tickets you’re working on or keep you updated on the builds running on your PRs for example.

Hopefully, a good portion of the 60+ open-source Backstage plugins which already exist will end up having Home widgets added.

Once this community work happens, the vision of having a single place to get the pulse of your org, your team, your software and your own work will be realised.

Learn more

The Home plugin is available to all Roadie Backstage users. You may need to enable it in you Administration area before it is visible.

Learn more in our Home plugin documentation.

At Roadie, we think deeply about the security of every feature we roll out. This has sometimes slowed us down, or meant that we've had to run without some features or plugins available. For example, we've written at length about steps we take to properly authenticate access for GitHub Apps.

Last year, to ensure our customers security, we made the hard decision to launch Roadie Backstage with the scaffolder disabled. This cost us some customers over the past 6 months, but it was worth it for security.

Today, after months of hard work, we are proud to launch scaffolder support with a completely re-designed and hardened architecture. This new architecture ensures that scaffolder tasks are run in an isolated and ephemeral environment which keeps customer data secure.

The scaffolder is available on all Roadie Backstage environments today.

To use it, start a free trial here, then check out our docs for the scaffolder, and read our walkthrough to learn how to write scaffolder templates.

What is the scaffolder?

Imagine you're an engineer looking to create a new microservice. You want to get started as quickly as possible, with minimal boilerplate and red-tape to jump through. At the same time, engineering organizations benefit from having consistency in production, and often put gates in place to enforce it.

Instead of creating blockers for engineering teams, Spotify use the scaffolder and quickly create new microservices, while helping to ensure that production remains mostly consistent.

Engineers can choose a pre-defined software template, fill out a few form fields to provide values like the name of the GitHub repo that the new service will occupy, and click a button to run the template and create a the new service.

By making it easier to start new projects, your engineers can move more quicly, while preserving standaeds and reducing complexity in your tech ecosystem.

See it in action in this 3 minute video where we demo a scaffolder template which creates a GitHub pages website.

Roadie's scaffolder architecture

Before starting the work to enable the scaffolder, we audited the open source scaffolder actions one by one and found that the Backstage Scaffolder was running them on the API backend process of Backstage. Scaffolder tasks technically had access to the same resources available to the API backend.

Tasks that copied files or accessed network resources might are a little risky even when running Backstage inside your corporate firewall. On a SaaS platform like Roadie, they are unacceptable.

To isolate our scaffolder jobs, we run them in a separate process in a private network on AWS ECS. A single container task is spun up for each execution and destroyed once it completes.

The container can only access to its own Backstage database and the public internet. The container does not have access to the network services available to the API backend process and it cannot do things like copy files from the local API backend services file system.

We already support the most frequently used scaffolder actions on Roadie. You can fetch pre-defined templates, use them to create GitHub repositories and write to the Backstage catalog.

You can even send HTTP requests to the public internet, using an open source library we created, so that newly templated microservices can automatically register with your SaaS tools like Circle CI or PagerDuty.

The full list of supported scaffolder actions is available inside your Roadie Backstage instance at https://[sub-domain].roadie.so/create/actions.

Next steps

We're going to continue working on the scaffolder to make it faster, more featured and more secure. We're already thinking about features such as custom container support and fully custom scaffolder actions.

We've also begun publishing more open-source modules for the scaffolder. We've already created a general utils package and some dedicated AWS actions.

If there's something you'd like to see, please reach out on our public Discord channel.

In this tutorial we are going to learn how to deploy a customized GitHub pages website with the Backstage scaffolder on Roadie.

The website we create will be...

1. Created from a prepared code skeleton.
2. Published from its own GitHub repo.
3. Automatically deployed to the public web via GitHub pages.
4. Customized with information we collect from the user when they run the scaffolder.
5. Available in the Backstage catalog so others can find it.
6. Automatically be hooked up to a monitoring service.

The skills you will learn include:

1. How to write scaffolder code skeletons and templates.
2. How to collect input from the user in the scaffolder UI, and pass it through to the codebase.
3. How to make HTTP requests from the scaffolder, and use the Roadie Backstage proxy to securely add authentication to the requests.
4. How to create proxies in Roadie.

This tutorial involces some steps which are specific to Roadie and won’t work on a vanilla Backstage installation. To try Roadie, apply for a free trial on our website. You will also see the best results if you are an admin of your Roadie Backstage instance.

Just show me it working!

If you'd prefer to see the scaffolder in action before working through this tutorial, you can watch this short demo video. It demonstrates the same GitHub Pages scaffolder action we create in the step that follow.

This video is part of the Backstage Bites series.

Step 1: Create a GitHub repo containing code from a skeleton directory.

A Backstage Scaffolder template will usually consist of at least two things:

1. A skeleton code structure, which is used to stamp out new websites, services, or other types of component.
2. A template.yaml file which describes the steps to run during the scaffolding process.

Create a basic skeleton and template

To get started, make a directory structure to hold our template and the code skeleton we will use to stamp out our GitHub pages website.

.
├── skeleton
│   └── index.html
└── template.yaml

Put the following HTML into the index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>my website</title>
</head>
<body>
  <h1>Welcome to my website</h1>
  <p>This website was created by following the <a href="https://roadie.io/blog/roadie-backstage-scaffolder-website/">Backstage scaffolder tutorial published by Roadie</a></p>
</body>
</html>

And place the following YAML into template.yaml. We’ll explore how this works at a later point in the tutorial. For now, let’s just get it working.

Optionally, you may wish to edit the spec.owner property to refernce the name of a real Group (aka. a team) in your Backstage catalog.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: github-pages-website
  title: GitHub Pages Website
  description: Create a static HTML website and publish it via GitHub pages.

spec:
  owner: my-group-name
  type: website
  parameters:
    - title: Choose a Source Control Management tool to store your new website in.
      required:
        - repoUrl
      properties: 
        repoUrl:
          title: Repository Location
          type: string
          ui:field: RepoUrlPicker
          ui:options:
            allowedHosts:
              - github.com

  steps:
    - id: template
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton

    - id: publishToGitHub
      name: Publish to GitHub
      action: publish:github
      input:
        allowedHosts: ['github.com']
        # This will be used as the repo description on GitHub.
        description: 'A static HTML website. Just like the good old days.'
        repoUrl: ${{ parameters.repoUrl }}
        defaultBranch: main
        repoVisibility: public

  output:
    remoteUrl: '{{ steps.publishToGitHub.output.remoteUrl }}'

Now that we have a basic template.yaml and skeleton, we need to push it to GitHub where the Roadie Backstage scaffolder can access it. You must install the Roadie GitHub App before proceeding. To learn how to do that, read our Getting Started docs. Turn your directory structure into a Git repository and push it to GitHub.

Import your template into Roadie Backstage

Copy the URL of the template.yaml file on GitHub and go to Roadie Backstage to import it into the catalog.

On the main catalog view, click the “CREATE COMPONENT” button, then click the “REGISTER EXISTING COMPONENT” button.

Paste the URL of the template.yaml into the URL text field and click “ANALYZE”.

Assuming there are no errors, click the “IMPORT” button.

Great. Your template is now imported into Backstage and ready to use.

Click the “Create...” link in the Backstage sidebar to go back to the main scaffolder view. You should see your template is now available.

Run the template

Click “CHOOSE” and you should be taken to a form like this:

Fill in the name of your GitHub Org in the “Owner” field. This template will use the Roadie Backstage GitHub app to run the scaffolder, so the Org must be the same org that the Roadie Backstage GitHub app is installed into.

Choose an arbitrary GitHub repository name and fill it into the “Repository” field. Something like “my-cool-website” will work. This repository doesn’t need to exist. The scaffolder will create it as it runs through the steps.

Click “NEXT STEP” and then “CREATE”.

After 15 or 20 seconds, you should see scaffolder logs being to appear. These logs will give you information about what the scaffolder is doing, and will display errors if failures occur.

Assuming everything worked correctly, you will see checkmarks appear in the left column of the scaffolder interface as it does its thing.

Click the link labelled “Repo” in the left sidebar to visit the GitHub repo which we just created. You should see it contains an index.html with the same content we placed in the HTML file earlier.

Step 2: Publish the site to GitHub Pages

Having a repo with some code in it is great, but it’s not a website until you can visit it in the browser.

To turn this HTML into a GitHub pages website, we can manually visit the settings of our newly created GitHub repo and click some buttons to publish the site to the web. We don’t like manual work though. Let’s see if we can do this in an automated fashion instead.

To make this work, we’re going to use the Open Source http:backstage:request scaffolder action from Roadie. You can see this package on GitHub. Don’t worry about following any of the installation steps. This package is already installed in Roadie Backstage.

This scaffolder action allows us to send HTTP requests from our template.yaml. We will use it it hit the GitHub API endpoint which can turn on GitHub pages for a GitHub repository.

Using the HTTP Backstage Request scaffolder action

Add a new step called publishToWeb to your template.yaml, commit it and push it to GitHub.

# ... spec and other properties above this point
  steps:
    # ... existing template and publishToGitHub steps here.

    - id: publishToWeb
      name: Publish to web with GitHub Pages
      action: http:backstage:request
      input:
        method: 'POST'
        path: /api/proxy/github/api/repos/${{ (parameters.repoUrl | parseRepoUrl)["owner"] }}/${{ (parameters.repoUrl | parseRepoUrl)["repo"] }}/pages
        headers:
          content-type: 'application/json'
        body:
          source:
            branch: main
            path: '/'

Let’s go through this YAML to learn what it’s doing.

- id: publishToWeb
  name: Publish to web with GitHub Pages
  action: http:backstage:request

The first three lines are relatively self explanatory. We’re adding a new step with an id and a human readable name. Then we’re declaring that we’re going to use the http:backstage:request action. If you ever want a list of all the available actions, just visit /create/actions inside Roadie Backstage.

input:
  method: 'POST'
  path: /api/proxy/github/api/repos/${{ (parameters.repoUrl | parseRepoUrl)["owner"] }}/${{ (parameters.repoUrl | parseRepoUrl)["repo"] }}/pages
  headers:
    content-type: 'application/json'
  body:
    source:
      branch: main
      path: '/'

The input is passed to the http:backstage:request action when it runs. Our input says we want to make a POST request with a particular body which satisfies the requirements of the GitHub API. We also want to pass some headers. These are the same details you might expect to see if we were calling this API endpoint with curl or Postman.

The path is interesting. Instead of calling the GitHub API directly, we’re proxying through the Roadie Backstage backend. As the request passes through Roadie Backstage, the proxy will transparently add an authentication token to the request. This means we don’t have to hardcode the authentication token into the template.yaml, or ask the user to provide it at runtime.

The first three parts of the path, /api/proxy/github/api , contain the location of the proxy on the Backstage API. Everything after github/api is the path of the API endpoint we want to hit on GitHub.

The GitHub path we want to call is /repos/[owner]/[repository]/pages. Of course, we don’t know the name of the owner and repository until the user runs the template. To work around this, we use a special syntax to parse them out at runtime.

We can get the “Owner” the user types in with ${{ (parameters.repoUrl | parseRepoUrl)["owner"] }} and we can get the “Repository” the user types in with ${{ (parameters.repoUrl | parseRepoUrl)["repository"] }}.

Storing the GITHUB_TOKEN securely

Our proxy knows how to forward requests over to GitHub, but it doesn’t yet have the correct token to be able to successfully authenticate. Lets securely store an authentication token in Roadie so the proxy can use it.

Click “Secrets” in the left sidebar of the SETTINGS tab on Roadie Backstage.

At the top of the list you should see a row for “GITHUB_TOKEN”. Visit your GitHub account settings and create a personal access token which has the pages scope. Be sure to authorize it on your GitHub org if you use SSO.

Click the pencil icon on the secrets page to open a dialog box. Paste in the personal access token you just created and click “SAVE”.

Roadie Backstage has to restart to enable this token. This can take a few seconds. Wait for the GITHUB_TOKEN table row to display a green status indicator and the text “Ready” before proceeding.

Putting it all together

Now we have a template which sends a HTTP request to a proxy and we have a secret which the proxy will use to authenticate with GitHub.

Since we pushed the new version of our template.yaml a while ago, the Backstage catalog should have already looped over it and picked up the new version with the call to the GitHub pages API in it.

Go back to your scaffolder template and fill out the details again. Remember, GitHub repos must have unique names, so if you haven’t deleted the repo you created in step 1, you’ll have to choose a different name this time.

Click “NEXT STEP” and “CREATE”.

This time you should see that we have 3 steps due to run in the left sidebar of the Task Activity page. The scaffolder is going to:

1. Fetch Skeleton + Template
2. Publish to GitHub
3. Publish to web with GitHub Pages

Success! Now visit the repo you just created on GitHub and go to the Pages part of the Settings.

Visit the following URL to see your new website:

https://[owner].github.io/[repository]/

It doesn’t look like much but it’s a start!

Step 3: Customize the website

Creating a website from a template is great, but it would be better if we were able to customize it a little. We don’t want our website looking exactly like hundreds of other scaffolded websites 😃.

For customization, the scaffolder lets you pass values through to the template as it is being processed. To see how this works, let’s collect a website name from the user and use it in the title tag and main heading.

First, we have to update the index.html in our website to render a value called website_name. This will be provided by the user when they run the scaffolder task.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>${{ values.website_name | title }}</title>
</head>
<body>
  <h1>Welcome to ${{ values.website_name }}</h1>
  <p>This website was created by following the <a href="https://roadie.io/blog/roadie-backstage-scaffolder-website/">Backstage scaffolder tutorial published by Roadie</a></p>
</body>
</html>

We also need to update our template to make it ask the user for the website name.

Add the following YAML to the template.yaml in the parameters section, above the item which asks for a repo owner and repository.

- title: Provide some simple information
  required:
    - website_name
  properties:
    website_name:
      title: Website name
      type: string
      description: This will be displayed prominantly on your website and in the title tag.

Finally, we need to name and pass the collected website name value down into the templating step so it is available in the index.html.

steps:
  - id: template
    name: Fetch Skeleton + Template
    action: fetch:template
    input:
      url: ./skeleton
      values:
        website_name: ${{ parameters.website_name }}

Commit and push those changes. Once the catalog refreshes the template from GitHub, you should now see a new text field in the interface. If you don’t want to wait, you can manually reimport the template to refresh it.

If we fill that in with a website name and proceed through the rest of the steps as before, we should eventually end up with a new GitHub pages website which has a unique name. In this example, I typed “My Cool Site 3” into the text field.

Step 4: Add a catalog-info to the new service

This new website looks promising, but it would be even better if it was automatically added to the Roadie Backstage catalog so that other people in your company could discover it.

To make this happen, we can add a catalog-info.yaml to our skeleton codebase and pass some values into it to set sensible defaults. Once this YAML file is created, we can rely on Roadie Backstage’s auto-discovery mechanism to pick it up.

Create a file named catalog-info.yaml inside the skeleton directory we created earlier. Place the following content in it:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Component
metadata:
  name: ${{ values.repo_name }}
  title: ${{ values.website_name | title }}
  description: A static HTML website. Just like the good old days.
  links:
    - url: https://${{ values.repo_owner}}.github.io/${{ values.repo_name }}/
      title: Live website
  annotations:
    github.com/project-slug: ${{ values.repo_owner }}/${{ values.repo_name }}
spec:
  type: website
  owner: engineering
  lifecycle: experimental

You can see that this file relies on a few values which contain information about the website we are templating. In order to have access to these values, we need to pass them in from the template.yaml. Edit the step named “template” to pass in the values.

steps:
  - id: template
    name: Fetch Skeleton + Template
    action: fetch:template
    input:
      url: ./skeleton
      values:
        website_name: ${{ parameters.website_name }}
        repo_name: ${{ (parameters.repoUrl | parseRepoUrl)["repo"] }}
        repo_owner: ${{ (parameters.repoUrl | parseRepoUrl)["owner"] }}

Next time we run this template, we will see that an extra file called catalog-info.yaml is created in the newly scaffolded GitHub repo. Roadie Backstage’s auto discovery will automatically find this file and use it to populate the Backstage catalog.

You may notice that we have specified some links in the catalog-info.yaml. This metadata will automatically populate the Links Backstage plugin with a link to our website on the public internet. Clicking the link will take the user to GitHub pages.

If you don’t see this card, ask a Roadie Backstage admin to add the EntityLinkCard to your website component overview page.

Step 5: Register the service with Better Uptime

Calling the GitHub Pages API was relatively simple, because there was already a proxy in place to use. But what if we want to call an authenticated endpoint which doesn’t have a default proxy?

To see how this works, we’re going to create our own proxy for the website monitoring service, Better Uptime, and then use the HTTP Request scaffolder action to setup a website ping when our scaffolder template is executed.

Roadie has no affiliation with Better Uptime. They have an easy to use UI and a free tier, which is useful for a tutorial like this.

Create a monitor in Better Uptime

The first thing you will need to do is to sign up for Better Uptime. Their free plan allows more than enough features to get through this tutorial.

One you have an account you will need to take note of your API token. Click Integrations in the sidebar, then APIs in the secondary header. Click the Copy to clipboard link.

To see how the Better Uptime API works, let’s create a monitor for https://google.com as a test.

Run the following curl command, making sure to substitute <API_TOKEN> with the API token you copy/pasted from the Better Uptime integrations page.

curl -X POST \
     --header 'Authorization: Bearer <API_TOKEN>' \
     --header 'Content-Type: application/json' \
     --url https://betteruptime.com/api/v2/monitors 
     --data '{"url":"https://google.com"}'

Assuming that runs successfully, you should see a monitor has been created in Better Uptime.

Now that we know how to do that with curl and Google, let's see how to do it with the scaffolder and our GitHub Pages website.

Calling the Better Uptime API from the scaffolder

To set up monitoring for the website we create with the scaffolder, all we need to do is reimplement this curl command in our template.yaml using the HTTP Backstage Request module.

The YAML step to accomplish that looks like this:

steps:
  # all of the previous steps already discussed

  - id: registerInBetterUptime
      name: Register in Better Uptime
      action: http:backstage:request
      input:
        method: 'POST'
        path: /api/proxy/betteruptime/monitors
        headers:
          content-type: 'application/json'
        body:
          url: https://${{ (parameters.repoUrl | parseRepoUrl)["owner"] }}.github.io/${{ (parameters.repoUrl | parseRepoUrl)["repo"] }}/

In this case, are sending a POST request to the path /api/proxy/betteruptime/monitors. This path doesn’t currently exist, so let’s create it using Roadie’s proxy UI.

Create a proxy for Better Uptime

To create a proxy, click “Administration” at the bottom of the main sidebar, then go to the “SETTINGS” tab and click “Proxy” in the minor left sidebar.

Click “ADD PROXY” to add a proxy.

There are a lot of fields and options available here. We don’t need to understand all of them to complete this tutorial. So we will focus on the most important ones.

“Path” is the path on which we will be able to call our proxy. This must match a part of the path we added to our template.yaml earlier. In our case, this is /betteruptime.

“Target” is the URL which we want to forward request on to. The root of the Better Uptime API is at https://betteruptime.com/api/v2, so in this case that is our Target.

Click the Advanced Settings to see some more options.

The “Allowed Methods” field can be used to restrict the HTTP methods that the proxy will accept. For example, if we choose GET and POST, then the proxy will refuse to forward DELETE requests. In this case, we only need to enable POST requests.

The “Headers” section can be used to add headers to the request as it is being forwarded on to the Target. This is how we will add an authentication token to our requests.

To make a Better Uptime proxy for our scaffolder template request, fill out the following:

1. Path = /betteruptime
2. Target = https://betteruptime.com/api/v2
3. Allowed Methods = POST
4. Check the “Secure” and “Change Origin” checkboxes at the bottom of the page.

To add the authentication token to the request, create a header called authorization and set its value to Bearer ${CUSTOMER_TOKEN_1}. When the request is proxied, ${CUSTOMER_TOKEN_1} will be replaced with an actual token, but we don’t want to store that here in plain text. Instead, we will use Roadie’s secure secrets functionality for this.

We also need to add a Content-Type header which specifies application/json.

Our proxy settings now look something like this:

Click “SAVE” and then “APPLY & RESTART” to create the proxy. You will need to wait approximately 3 minutes for the settings to be applied.

Securely store the Better Uptime token

In the previous section, we created an authorization header with the placeholder ${CUSTOMER_TOKEN_1}. In order for this placeholder to be replaced with the actual authentication token, we have to store the token in the Roadie Secrets area, just like we did previously with the GITHUB_TOKEN.

To store the token securely, visit the Secrets page in the Roadie Backstage Administration area and set the CUSTOMER_TOKEN_1 to the value of the API token we got from Better Uptime.

Re-run the scaffolder job

Assuming those steps all worked correctly, we should now be able to re-run our scaffolder template and create a GitHub pages website which is hooked up to Better Uptime.

Go back to the Roadie Backstage scaffolder, choose your GitHub pages template and fill in the website name and GitHub owner and repo name again. Remember to use unique values. Hit CREATE and watch the magic happen.

This time, 4 steps run together, including the new Better Uptime step.

If we check out the Better Uptime monitors page, we should see that our website is registered. Initially it will be reporting as “DOWN”. That happens because it takes GitHub pages a minute or two to publish the website. Once it goes live, Better Uptime will update and go green.

Conclusion

Now that you’ve learned how to use the templates, proxy and secrets together, you should be able to apply this knowledge to other tasks. For example,

1. Try to add some basic TechDocs to the skeleton so that the docs show up in Backstage automatically.
2. Register the website with an error tracking tool like Sentry.
3. Enable branch protection on the GitHub repo created by your scaffolder template.

The scaffolder is capable of automating all of these tasks to help your org be more productive and more standardized.

Future work in Roadie

We love this functionality already, but we also believe we have more work to do to make it the best it can be. Here are some of the areas we will be looking to improve next:

1. Custom secrets so you can add your own secrets and choose their labels.
2. Custom scaffolder actions so you can run completely custom code in the scaffolder.
3. Open source scaffolder packages such as our utils package and an AWS package.

To learn about these new features and releases as they roll out, please join our Backstage Weekly newsletter.

Self-hosted Backstage remains the right choice for many organizations. Large organizations with thousands of developers and abundant resources will be able to staff a team to deploy, customize and manage Backstage.

For companies with a few hundred developers, it’s not so easy. Every engineering hour spent on internal tools is an hour that could be spent delivering customer-facing value. We believe Roadie is the right answer in this situation.

Below are 10 reasons why Roadie might be right for your company.