Improving software standards with Roadie

Uplight is a clean energy technology company that creates, deploys, manages and monetizes energy resources at scale to improve grid reliability, reduce costs, and accelerate decarbonization.

Uplight is dedicated to improving the way they develop software to meet that mission, which in turn means they’re highly engaged with software catalogs and scorecards, and how the combination of the two can be used to drive sought-after improvements.

I met with Doug Ramirez, Principal Architect at Uplight, and Shaw Atkinson, Senior Principal SRE, to discuss how they’ve used Roadie to capture and model a diverse set of software teams and components, then set about the task of improving software standards using scorecards.

Building a software catalog in a complex business

Engineers and architects at Uplight think a lot about how to document, standardise and improve the software they write. Uplight has grown rapidly since its founding in 2019, and any company which embarks on such a journey needs a powerful muscle for folding in new technology and capabilities into an existing stack.

Pulling available levers to gain traction

The initial rationale for working with Roadie was to help the team manage its technology stack through a software catalog: consistently maintaining ownership attribution, standardisation, and transparency of information and folding new units into the wider Uplight technical community.

Driving adoption took time and focus, but the reward was a complete catalog. As Doug puts it, “we went through a big push to make everybody merge in catalog-info.yaml files and to GitHub for detection by Roadie with some helpful context”. That initial surge, as Shaw adds, has now created a virtuous feedback loop: the catalog in Roadie just “solves issues before they arise - it gives the visibility to engineering teams that they could be missing” and that’s a big enough benefit that it’s pushed adoption up towards 100%.

Defining software standards

Once everything is visible and transparently available in a software catalog, Platform and architecture teams have the opportunity to drive up standards and standardisation of software development.

To do that you need to put a stake in the ground and define some standards.

Asking the right questions

Uplight started to ask questions like:

What are our standards of what a service is, what a component is etc?
What does it mean to be ‘production ready’?
How can we verify those standards so we’re not relying on manual reporting?
And ultimately, how can we surface all this data to teams and leadership?

To answer those questions, the team at Uplight looked back on past examples of standardisation, especially their checklists, to start to pull disparate threads together into a single cohesive set of software standards.

“We adapted some of our existing checklists on production readiness, service deployment checklists, service account governance derived through infra etc, but we had to do the groundwork” said Doug.

Forming a software standards document

The end product was a set of 10 categories of software standards that will feel familiar to other businesses:

Security
Logging
Monitoring
etc.

With individual line-items against each one to pinpoint concrete steps by which to measure those areas of a given piece of software.

Uplight focused on the simplicity of their standards rather than dictating how and by what means engineering teams should fix individual tasks. As Doug puts it, “when you’re building these things you think about a check engine light on the dashboard - it means you should go take a look, it suggests some aspect of the checklist not being met,” rather than specifying a solution.

Implementing automated checks in Roadie

To help automate these standards, Uplight uses the Roadie Tech Insights plugin inside Roadie to build checks and scorecards, then run checks against those standards several times a day.

As Shaw highlights, “If a VP asked a very simple question about our infrastructure or code standards a year ago I wouldn’t be able to answer it without some digging. Now, even for a check or data we don’t yet have in Roadie it’s fifteen minutes. I add a Tech Insights Data Source, run it for a bit, then send them the link.”

Starting simple

Uplight started the process of automating these standards as simply as they could.

“There were a lot of checks that we didn’t know how to bring the info up and make it available. So step one for us was looking for annotation - rather than looking at the code or CICD platform to determine deployment strategy. We looked at documentation and annotation first.

This was a “Fantastic first step. It started the conversation. We have a lot of repos, people aren’t always focused on legacy services, stuff gets lost in march of time etc. and now teams have the information they need to be effective at their fingertips,” said Shaw.

These productive discussions allowed Product, Engineering and non-technical teams to share a common language around standards improvement and start to shift behaviour.

“Our goal with scorecards is when we give teams ownership of something in the catalog they can kind of look at each one of these scorecards and decide which one is more most critical to them and then start work to address that category, get that percentage up.”

Embedding standards takes time but Uplight are investing the time and energy in thinking about the social elements of standardisation and standards improvement as much as the technical implementation of creating checks and scorecards within Roadie.

Increasing the complexity of checks

Software standards that check documentation are one thing, but automating the process by which every piece of software an organisation creates is evaluated against that software is a different challenge.

As Shaw puts it, “we threw a lot of checks into Tech Insights, a whole bunch of people got their feet wet and so when we came back to it we could streamline and review. Now we don’t point at documentation, we point to implementation.”

That includes things like checking DataDog directly for SLOs, for example, rather than relying on documentation of service standards.

And what’s next for Uplight?

In a word: campaigns.

“Previously we ran these campaigns very loosely on Slack and with running lists and spreadsheets,” said Doug. Tech Insights tightened up these campaigns and removed the spreadsheets, but it didn’t remove all manual effort.

Currently, Uplight focus the attention on a subset of scorecards each month. This makes it easier to rollout and ensures that teams don’t get overwhelmed if they see a lot of new information in the scorecards.

Of the ten different categories of service delivery maturity that are now embedded into Uplight’s scorecards, only one or two are seen as the priority at any given time. These campaigns focus attention on the top priorities for the wider business and allow a simplicity of communication with teams.

“We started first with a ‘General’ scorecard. It focused on ownership, codeowner files, is there a GitHub slug, etc. It was effectively housekeeping.” That helped engineering teams build a mental model of what a campaign was going to be, what they needed to do, and what good looked like.

Next up came SLO reporting. Then architectural diagramming and documentation. Sometimes a campaign isn’t a full scorecard, it is a subset of checks from a given scorecard, or a few checks across different areas. This flexibility means that teams can understand a campaign is what the rest of Uplight is prioritising at that moment.

“We keep running campaigns at the rate of two or three per quarter to keep pushing up standards” said Doug.

Manual Campaigns

Tech Insights helps here, but it currently doesn’t capture the totality of a campaign. To build an appropriate campaign, the Uplight team identifies what materials (golden paths, runbooks, tooling, supporting teams, etc) are missing and gather and prepare all that information before a campaign is announced. Then they use Tech Insights to measure adoption and offer assistance where needed.

The aspiration is greater than that though. Uplight wants to automate as much as possible and use Tech Insights more to drive the conversation, not just report progress.

As Doug puts it:

“We want to put a scorecard front and center on everybody’s access to the catalog and say ‘This is the campaign. Work on these scorecard checks’ then put the data for that campaign in front of engineering managers and make leaderboards where teams can see how they can leverage knowledge from other teams to accelerate their adoption”.

Tech Insights Campaigns

Taking inspiration and feedback from Uplight, we’ll shortly be introducing functionality so that other customers can use Tech Insight insights in the same way that Uplight are. Doug put it simply a few weeks ago:

“Everything on your roadmap we will use, but if I could vote with Roadie Bucks I’d put my money on full campaigns inside Roadie.”

So that’s what we’re building.

An early version of Roadie Tech Insights Campaigns will be released this year. Roadie customers will be able to select a set of scorecards and checks, allocate a date and time, push those particular standards out to teams via cards on the homepage and entity pages, and monitor the ongoing campaign in a dashboard. External notifications to email and Slack for a Campaign will also be coming soon.