Backstage Splunk On-Call Plugin

Splunk On Call is an incident management service from Splunk. It routes alerts, manages schedules and escalations, and helps teams respond fast. It gives on call visibility and audit trails so you can track what happened and who is working it.

The Backstage plugin brings that context into your service catalog. On a component page you can see recent incidents, who is on call, and related escalation policies. You can trigger a new incident, acknowledge one, or resolve it right from Backstage. Teams can run it in read only mode if they only want visibility. The plugin links a Backstage entity to a Splunk On Call team or to a routing key through annotations, so the data shown is specific to the service you are viewing.

Common use cases are simple. During an outage, open the service page to see current incidents and the people on call. Page the right team without context switching. Afterward, review activity where your engineers already work.

Install the frontend package

Copy

yarn --cwd packages/app add @backstage-community/plugin-splunk-on-call

Add the entity card to your catalog pages

Put the card on the Entity page so users can see it.

Copy

// packages/app/src/components/catalog/EntityPage.tsx
import React from 'react';
import Grid from '@material-ui/core/Grid';
import { EntitySwitch } from '@backstage/plugin-catalog';
import {
  isSplunkOnCallAvailable,
  EntitySplunkOnCallCard,
} from '@backstage-community/plugin-splunk-on-call';

const overviewContent = (
  <Grid container spacing={3} alignItems="stretch">
    <EntitySwitch>
      <EntitySwitch.Case if={isSplunkOnCallAvailable}>
        <Grid item md={6}>
          <EntitySplunkOnCallCard />
        </Grid>
      </EntitySwitch.Case>
    </EntitySwitch>
  </Grid>
);

export default overviewContent;

Read only mode

Use this if you do not want users to trigger or change incidents from Backstage.

Copy

<EntitySplunkOnCallCard readOnly />

Add an optional top level page

This adds a full page at path /splunk-on-call.

Copy

// packages/app/src/App.tsx
import React from 'react';
import { Route } from 'react-router';
import { FlatRoutes } from '@backstage/core-app-api';
import { SplunkOnCallPage } from '@backstage-community/plugin-splunk-on-call';

export const AppRoutes = () => (
  <FlatRoutes>
    {/* other routes */}
    <Route path="/splunk-on-call" element={<SplunkOnCallPage />} />
  </FlatRoutes>
);

Configure the client and proxy

Add the REST endpoint. Then add a proxy that forwards to the Splunk On Call public API with your API id and key.

Copy

# app-config.yaml
splunkOnCall:
  eventsRestEndpoint: ${SPLUNK_ON_CALL_EVENTS_REST_ENDPOINT}

proxy:
  '/splunk-on-call':
    target: https://api.victorops.com/api-public
    headers:
      X-VO-Api-Id: ${SPLUNK_ON_CALL_API_ID}
      X-VO-Api-Key: ${SPLUNK_ON_CALL_API_KEY}

backend:
  cors:
    methods:
      - GET
      - POST
      - PUT
      - DELETE
      - PATCH

Start the backend with the required secrets.

Copy

SPLUNK_ON_CALL_API_ID='...' SPLUNK_ON_CALL_API_KEY='...' SPLUNK_ON_CALL_EVENTS_REST_ENDPOINT='https://events.victorops.com/v2/enqueue' yarn start

Configure the backend new system

Install the proxy backend plugin.

Copy

yarn --cwd packages/backend add @backstage/plugin-proxy-backend

Copy

// packages/backend/src/index.ts
import { createBackend } from '@backstage/backend-defaults';
import { proxyPlugin } from '@backstage/plugin-proxy-backend';

const backend = createBackend();
backend.add(proxyPlugin());
backend.start();

The proxy reads the config under proxy in app config. No extra code is needed for Splunk On Call here.

Configure the backend old system

Install the proxy backend plugin if you do not have it.

Copy

yarn --cwd packages/backend add @backstage/plugin-proxy-backend

Mount the proxy router in the legacy backend.

Copy

// packages/backend/src/index.ts
import { createServiceBuilder } from '@backstage/backend-common';
import { Logger } from 'winston';
import { Config } from '@backstage/config';
import { TokenManager } from '@backstage/backend-common';
import { PluginEnvironment } from './types';
import { createRouter as createProxyRouter } from '@backstage/plugin-proxy-backend';

// create the usual env objects here
// const env = await loadBackendConfig and logger etc

async function main() {
  const logger: Logger = /* your logger */;
  const config: Config = /* your config */;
  const tokenManager: TokenManager = /* your token manager */;

  const proxyRouter = await createProxyRouter({
    logger,
    config,
    tokenManager,
  });

  const service = createServiceBuilder(module)
    .setPort(7007)
    .addRouter('/proxy', proxyRouter); // available at /api/proxy

  await service.start();
}

main().catch(err => {
  process.exit(1);
});

The proxy reads the config under proxy in app config. No extra code is needed for Splunk On Call here.

Add entity annotations

Add one of these to the entity metadata. Pick team or routing key.

Copy

# catalog-info.yaml or in your entity definition
metadata:
  annotations:
    splunk.com/on-call-team: your team name

Copy

metadata:
  annotations:
    splunk.com/on-call-routing-key: your routing key

Create the routing key

Create a routing key in the Splunk portal. Use the same name as the team if you plan to use team based routing. Docs https://help.victorops.com/knowledge-base/routing-keys

Lock down the proxy for read only use

If you only want to read data, restrict methods to GET and use the read only card.

Copy

# app-config.yaml
proxy:
  '/splunk-on-call':
    target: https://api.victorops.com/api-public
    headers:
      X-VO-Api-Id: ${SPLUNK_ON_CALL_API_ID}
      X-VO-Api-Key: ${SPLUNK_ON_CALL_API_KEY}
    allowedMethods: ['GET']

Copy

// packages/app/src/components/catalog/EntityPage.tsx
<EntitySplunkOnCallCard readOnly />

Helm examples for secrets

Copy

# backend-secret.yaml
stringData:
  SPLUNK_ON_CALL_API_ID: {{ .Values.auth.splunkOnCallApiId }}
  SPLUNK_ON_CALL_API_KEY: {{ .Values.auth.splunkOnCallApiKey }}
  SPLUNK_ON_CALL_EVENTS_REST_ENDPOINT: {{ .Values.auth.splunkOnCallEventsRestEndpoint }}

Copy

# values.yaml
auth:
  splunkOnCallApiId: your id
  splunkOnCallApiKey: your key
  splunkOnCallEventsRestEndpoint: https://events.victorops.com/v2/enqueue

Backstage Splunk On-Call Plugin

Installation Instructions

Install the frontend package

Add the entity card to your catalog pages

Read only mode

Add an optional top level page

Configure the client and proxy

Configure the backend new system

Configure the backend old system

Add entity annotations

Create the routing key

Lock down the proxy for read only use

Helm examples for secrets

Changelog

Dependencies

Tooling

Set up Backstage in minutes with Roadie