Creating Backstage EntityProviders at Runtime

Backstage's catalog is the heart of your developer portal. EntityProviders are the mechanism by which data flows into it—they connect to external systems, fetch entity data, and push it to the catalog.

Typically, EntityProviders are registered at application startup via the catalogProcessingExtensionPoint. Once the backend initializes, the set of providers is fixed. But what happens when you need to dynamically create new sources of catalog data without redeploying?

Consider these scenarios:

• A multi-tenant platform where each tenant needs isolated entity management
• User-defined integrations that pull data from custom sources
• Dynamic data pipelines that generate catalog entities on-demand
• Self-service onboarding where teams register their own data sources

Backstage doesn't natively support registering EntityProviders after startup. This post explains how to solve this with a provider pooling pattern.

The Challenge

The EntityProviderConnection that allows emitting entities is established once at startup when providers are registered. After initialization, you cannot add new providers—any attempt to call addEntityProvider after the catalog has started will fail.

The Solution: Provider Pooling

Instead of fighting Backstage's architecture, work with it. The key insight: register a pool of providers at startup, then dynamically assign them to consumers at runtime.

┌─────────────────────────────────────────────────────────────┐
│                    Backend Startup                          │
│  ┌──────────────────────────────────────────────────────┐   │
│  │  1. Create pool of N idle EntityProviders            │   │
│  │  2. Register all with catalogProcessingExtensionPoint│   │
│  │  3. Restore any persisted assignments from database  │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  ProviderRegistryService                    │
│  ┌────────────────────────────────────────────────────┐     │
│  │  getProviderFor(id) → assigns idle provider        │     │
│  │  releaseProvider(id) → clears entities, returns    │     │
│  └────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘
                               │
         ┌─────────────────────┼─────────────────────┐
         ▼                     ▼                     ▼
    Consumer A            Consumer B            Consumer C
    (provider-0)          (provider-1)          (provider-2)

The Pooled EntityProvider

Extend the standard EntityProvider with assignment tracking and the ability to clear entities:

interface PooledEntityProvider extends EntityProvider {
  assignTo(id: string): void;
  clearAssignment(): void;
  clearEntities(): Promise<void>;
  updateEntities(entities: Entity[]): Promise<void>;
}

The key method is clearEntities—emitting an empty full mutation removes all entities that provider previously managed:

async clearEntities(): Promise<void> {
  await this.connection?.applyMutation({
    type: 'full',
    entities: [],
  });
}

The Registry Service

The registry manages the pool, handling assignment and release:

interface ProviderRegistryService {
  getProviderFor(id: string): Promise<PooledEntityProvider>;
  releaseProvider(id: string): Promise<void>;
  getAllProviders(): PooledEntityProvider[];
}

When a consumer requests a provider, the registry either returns an existing assignment or finds an available provider from the pool and persists the new assignment.

When a provider is released, the registry clears its entities, removes the assignment, and returns it to the pool.

Catalog Registration

Register all pooled providers at startup via a backend module:

const providerRegistryCatalogModule = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'provider-registry',
  register(module) {
    module.registerInit({
      deps: {
        catalog: catalogProcessingExtensionPoint,
        registry: providerRegistryServiceRef,
      },
      async init({ catalog, registry }) {
        for (const provider of registry.getAllProviders()) {
          catalog.addEntityProvider(provider);
        }
      },
    });
  },
});

Persistence

Store assignments in a database table so they survive restarts. This ensures the same consumer gets the same provider ID, maintaining entity ownership continuity.

Usage

// Acquire a provider
const provider = await registry.getProviderFor('my-integration-123');

// Emit entities
await provider.updateEntities(entities);

// When done, release (clears entities and returns provider to pool)
await registry.releaseProvider('my-integration-123');

Trade-offs

• Memory overhead — Pre-allocated providers consume memory, though instances are lightweight
• Fixed capacity — Pool exhaustion causes failures; size appropriately for your workload
• Provider ID stability — Assignments must persist to maintain entity ownership across restarts

Conclusion

By pre-registering a pool of EntityProviders and dynamically assigning them at runtime, you can achieve flexible, dynamic entity management without modifying Backstage's core architecture. This pattern works for multi-tenancy, user-defined integrations, or any scenario requiring runtime control over catalog data sources.