Achieving immediate catalog updates via the UI in Backstage

Updating an entity in the Backstage catalog generally means manually updating a YAML file in a repository somewhere and getting it merged to the main branch.

Manually updating YAML files can be a jarring experience as it often involves multiple context switches, waits and async dependencies on other people or systems - like getting a pull request merged. When you have to scale this across thousands of repositories (for instance when a group name changes), it can be a months long process of chasing potentially hundreds teams to raise and merge PRs.

This leads to a large amount of friction to keep your catalog up to date, let alone making the entities rich with information and working plugins.

How long does it take to add a PagerDuty plugin?

To illustrate this, lets say I want to get the PagerDuty plugin showing information for an entity representing an backend server. All I need to do is add an annotation to the YAML file with the service id for the correct PagerDuty service.

The quickest route to editing the YAML file in GitHub might be clicking the pencil icon on the About card.

Depending on your SCM provider this opens up GitHub’s web editor interface or a view of the file in your SCM.

In GitHub’s case, we can edit this immediately, commit it to a new branch and open a pull request rather quickly all in the web editor.

In the case of wanting to make the PagerDuty plugin work, we must add an annotation for the correct service like so:

To find the correct Service ID we must then go to PagerDuty. Hopefully we can log in and access the available services, and then search for the correct one using the names in the Component YAML.

However it might not always be a one to one mapping which means we would have to do some more research and asking around the team involved to find the correct service.

Once we find the correct service we’ll need to find the service id, which in this case can be found in the address bar but not the UI 🤦.

If we copy and paste in the ID to our GitHub YAML editor, we can then commit and open a PR for the change.

We’ll then need to get this new Pull Request reviewed, approved and merged before our Backstage entity will have a working PagerDuty annotation to use. This can take some time, especially if you are not able to merge it yourself or the team involved does not have a fast review cadence.

As you can see, this is potentially a slow and painful amount of manual work to make a small change to your Component in the catalog. And changes like this at scale are even more painful when you consider trying to get hundreds of teams to do it.

Making it quick and easy

At Roadie we’ve built UI based tools that allow you to set things like PagerDuty annotations with a few clicks. Instead of seeing a missing annotation card you’ll be able to select from a list of PagerDuty services and update the entity immediately all from the same page in the UI.

Similarly when attempting to add annotations to multiple entities in your catalog we provide a bulk editor which allows setting these with a few clicks using a table format.

How did we build it?

These UI based editors that immediately update the entity in the catalog rely on a backend feature we’ve added called Fragments. Fragments are partial entity data stored in a database table, that is then merged by a processor into existing entities in the catalog.

This partial Fragment data can be provided via API or via our custom UI as we’ve seen above.

Leverage external APIs to make things easier

The custom UI we’ve seen earlier in this post also leverages external API’s to provide a dropdown of available options. In this example we’ve used PagerDuty’s API to fetch a list of services using the API token already added to Backstage to make your PagerDuty plugin work. This call is made via the Backstage proxy which is set up to point to PagerDuty using your API Token on the backend.

Script bulk changes via API

However, sometimes bulk editing of metadata in the catalog is best scripted, and in this scenario, Roadie users make use of our Fragments API to make changes to entities, such as changing the ownership of a group’s entities when the group gets absorbed into another group, or updating any relationship references to a user entity when that individual leaves and someone else takes their role.

We’re not the only ones who’ve done something like this - Twilio and others have built similar fragment functionality so that they can script updates to there catalog via API and free up platform teams to help keep the catalog data rich and accurate.

Reach out to us on Discord or our website’s chat messenger if you want to hear more about this and other improvements Roadie has made to the Backstage experience.