Backstage Kafka Plugin

Apache Kafka is a distributed event platform. It stores and moves streams of records in topics. Teams use it for data pipelines, messaging, and event driven services.

The Kafka plugin brings Kafka signals into Backstage. It adds a tab on your service pages with topic offsets and consumer group offsets. You can see lag and spot trouble in seconds. It works across multiple clusters and reads simple annotations on your entities. You can add links to external dashboards like AKHQ or Confluent, so people can jump from Backstage to deeper views when they need to. This keeps the day to day checks in one place and helps during incidents or deploys.You can see others using Kafka with Backstage in the wild.

Peaksys Engineering wrote about trying the Kafka plugin during their journey. In their words, they “first tested the Kafka plugin” as part of a broader Kafka integration, before later building extras for multi cluster views.

If you run Kafka already, this plugin gives engineers a shared window into topics and groups. It lowers context switching. It makes offsets and lag visible next to code, docs, and ownership.

Install the packages

Copy

# from your Backstage root directory
yarn --cwd packages/app add @backstage-community/plugin-kafka
yarn --cwd packages/backend add @backstage-community/plugin-kafka-backend

Add the backend plugin legacy backend

Create a router file

Copy

// packages/backend/src/plugins/kafka.ts
import { createRouter } from '@backstage-community/plugin-kafka-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return await createRouter({
    logger: env.logger,
    config: env.config,
  });
}

Mount the router

Copy

// packages/backend/src/index.ts
import kafka from './plugins/kafka';

// inside main
async function main() {
  // other envs
  const kafkaEnv = useHotMemoize(module, () => createEnv('kafka'));

  // other routers
  apiRouter.use('/kafka', await kafka(kafkaEnv));

  // server start stays the same
}

Configure Kafka in app config

Add your clusters to app config

Copy

# app-config.yaml
kafka:
  clientId: backstage
  clusters:
    - name: cluster-name
      brokers:
        - localhost:9092

You can add a dashboard url per cluster if you want

Copy

# app-config.yaml
kafka:
  clientId: backstage
  clusters:
    - name: cluster-name
      brokers:
        - localhost:9092
      dashboardUrl: https://dashboard.com

Annotate your components with consumer groups

Add the consumer groups annotation on the service entity. You can list more than one group with commas.

Copy

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: my-service
  annotations:
    kafka.apache.org/consumer-groups: cluster-name/consumer-group-name
spec:
  type: service

You can also attach dashboard urls by annotation

Copy

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: my-service
  annotations:
    kafka.apache.org/dashboard-urls: cluster-name/consumer-group-name/dashboard-url
spec:
  type: service

The consumer group name in the dashboard urls annotation is optional

Add the Kafka tab legacy frontend

Import the Kafka entity content and add a tab on the service entity page

Copy

// packages/app/src/components/catalog/EntityPage.tsx
import React from 'react';
import { EntityLayout } from '@backstage/plugin-catalog';
import {
  EntityKafkaContent,
  isKafkaAvailable,
} from '@backstage-community/plugin-kafka';

const serviceEntityPage = (
  <EntityLayout>
    {/* other tabs */}
    <EntityLayout.Route path="/kafka" title="Kafka">
      <EntityKafkaContent />
    </EntityLayout.Route>
  </EntityLayout>
);

// Optional show the tab only when Kafka is available on the entity
// wrap your routes in an EntitySwitch if your app uses it
/*
import { EntitySwitch } from '@backstage/plugin-catalog';
const serviceEntityPage = (
  <EntitySwitch>
    <EntitySwitch.Case if={isKafkaAvailable}>
      <EntityLayout>
        <EntityLayout.Route path="/kafka" title="Kafka">
          <EntityKafkaContent />
        </EntityLayout.Route>
      </EntityLayout>
    </EntitySwitch.Case>
  </EntitySwitch>
);
*/

Use the new frontend system

You can let the app auto detect the Kafka plugin or enable it in code

Enable via config

Copy

# app-config.yaml

# enable package discovery for all plugins
app:
  packages: 'all'

---
# or enable discovery only for Kafka
app:
  packages:
    include:
      - '@backstage-community/plugin-kafka'

Enable via code

Copy

// packages/app/src/App.tsx
import { createApp } from '@backstage/frontend-defaults';
import kafkaPlugin from '@backstage-community/plugin-kafka/alpha';

const app = createApp({
  features: [
    kafkaPlugin,
  ],
});

export default app.createRoot();

Customize the Kafka tab with extensions config

Copy

# app-config.yaml
app:
  extensions:
    # disable the Kafka entity content if needed
    - 'entity-content:kafka': false

    # or customize the tab path and title
    - 'entity-content:kafka':
        config:
          path: '/events'
          title: 'Events'

Backend new system

The backend package here exposes an Express router. The steps above show the legacy backend wiring. There is no new backend system module documented for this plugin. Use the router setup shown above in your backend.

Backstage Kafka Plugin

Installation Instructions

Install the packages

Add the backend plugin legacy backend

Configure Kafka in app config

Annotate your components with consumer groups

Add the Kafka tab legacy frontend

Use the new frontend system

Enable via config

Enable via code

Customize the Kafka tab with extensions config

Backend new system

Changelog

Features

Documentation

Maintenance

Breaking changes

Set up Backstage in minutes with Roadie