The Backstage Scaffolder, a Powerful New Orchestration Tool

Backstage, the open source internal developer platform (IDP) created by Spotify, has a powerful unsung tool up its sleeve. The Backstage Scaffolder is a cloud orchestration tool that allows a wide variety of meta orchestration workflows. It can leverage sub-workflows like GitHub Actions or Infrastructure as Code cloud orchestrators like Env0 or Terraform Cloud to make a hugely powerful and flexible templating and automation platform for any organization.

Orchestration tools like Jenkins, Ansible or Salt have been around for a while. Yet the ease of use and accessibility for engineers sets the Backstage Scaffolder apart. Unlike traditional tools, it is built directly into an IDP platform, making it readily available to the entire engineering organization and low cost. It also supports role-based access control (RBAC), which allows fine-grained control over who can create, modify or use the templates.

The Backstage Scaffolder was built originally around the idea of templating repositories in Source Control Management systems to create “golden path” standardization in organizations. However, as with many orchestration tools with broad integrations and API scripting opportunities, the Scaffolder can be effectively used for a broad set of orchestration workflows within cloud native and even more traditional companies with HTTP API access to internal systems and orchestration layers.

Let’s look at some examples of what you can do with this tool as well as some of its limitations in comparison to its rivals.

Creating Accessible Golden Paths

Golden paths in software organizations are essentially agreed-on best practice templates for how to develop, build, deploy and monitor software. Templates could simply be a workflow, such as a best practice CI/CD deployment pipeline or a skeleton code repository for certain workloads, like an HTTP server.

The Backstage Scaffolder was initially designed around this use case — templating new code repositories with a golden path. For instance, when a backend developer wants to create a new AWS Lambda function for a ticket, instead of writing it from scratch or making their own decisions on repository layout, tooling, testing and language, they can run a Scaffolder template that gives them a standardized starting point with a common layout, build tooling and CI/CD workflows.

The Scaffolder has integrations to all the major cloud source control management systems like GitHub or Azure DevOps as well as support for Nunjucks and Cookiecutter templating languages. This allows engineers to get new projects off the ground in a matter of minutes using a clear and secure architecture and workflow. It can embed testing standards and approaches in a codebase to save engineering time and reduce bugs.

At the organizational level it can ensure easy transfer of developers between teams as each team has similar codebase structures and workflows allowing them to get off the ground faster.

Golden Paths at Every Level

But it also can be used  for much higher level golden path orchestrations such as AWS account creation, infrastructure bootstrapping on top of IaC automation tools and even orchestrating workflows in other automation tools such as GitHub Actions.

There are a dizzying array of open source Scaffolder actions available to use as well as a relatively straightforward path to writing custom ones or combining existing generic steps like calling an HTTP API and parsing the response to achieve almost any workflow.

For instance, you can write a scaffolder template that sets up a new team with AWS accounts and users using Terraform and GitHub Actions, stub Confluence docs with their own set of templated internal team documentation such as engineer onboarding checklists, create team Slack channels and document some initial team Objectives and Key Results like getting set up and releasing the first bit of production software.

Templates can save days of engineering and management time as well as ensuring standardized approaches that embed governance tooling and security standards in every team by default.

The Discoverability Problem

There are other tools in this area that allow templating repositories, such as Yeoman. Combined with an internally available CI/CD workflow that can be bookmarked, golden path templating can be achieved for code repositories without the Backstage Scaffolder.

However, discoverability is a key factor in how much these kinds of golden paths are actually used. Discoverability in the CI/CD tooling space is generally very poor as they are often associated with individual repositories or lack search and accessible metadata such as a clear title, description and tags.

The Backstage Scaffolder places discoverability at its core, listing templates in a searchable and filterable page with metadata displayed on cards.

If we compare this to GitHub Actions, a popular cloud-based CI/CD workflow tool, we can see the difference clearly.

Without easily accessible and discoverable templates that are front of mind, it can be a challenge to get engineers and managers using templates, even if they are there.

Combining Sub-Orchestration Workflows

The Backstage Scaffolder can call any other specialized orchestration tool that exposes an HTTP API, chaining together orchestration workflows into larger automation processes with GitHub Actions, Terraform Cloud, Jenkins etc.

Other orchestration tooling can of course do similar things with options including Chef Courier, Jenkins, Salt or Ansible Automation Platform. However these tools are often gated and accessible only to DevOps engineers or infrastructure engineers partly due to pricing models, thereby restricting their impact on day-to-day engineering practices. By contrast Backstage is intended  for all members of an engineering organization on a daily basis. Having a powerful automation tool like the Scaffolder front and center in a daily tool makes driving adoption of golden path templates and time-saving workflows much easier.

Limitations

While the Backstage Scaffolder can be immensely powerful as an orchestration tool, it does not compete with dedicated enterprise-grade orchestration tools in certain areas such as the ability to create scheduled workflows, perform complex logic branching in an easy-to-visualize way or run shell commands. Additionally, it must be manually set up using Typescript code in your Backstage instance with integrations added individually with code changes. If you need RBAC you will need to implement and configure Backstage RBAC with code again, which can be time consuming.

Alternatively managed Backstage solutions like Roadie can provide a ready-to-use Scaffolder with RBAC integrated out of the box and secure runtime features. This is similar to tools like Ansible, which has been packaged with additional UI layers in RedHat’s Ansible Automation Platform. Roadie’s Scaffolder comes with additional features such as easy configuration of template groups, self-serve proxy creation for use with the HTTP Request action and certified templates that help users know which templates are stable and ready to use.

Originally published on The New Stack.

Image from Jevanto Productions on Shutterstock.