Internal Developer Platforms – Part 12: Backstage Entities and OpenAPI Example

In the last blog post, we discussed the various entity types within Backstage’s Software Catalog. As the plain theory is sometimes quite abstract, we will continue with an concrete example and show how to publish the API of a simple Java Spring REST service in Backstage and publish its API in the OpenAPI specification format within Backstage.

The sample app

As we are using Spring Boot, we can simply create a simple application with Spring Initializr. We open the page https://start.spring.io and can provide the input as shown in the screenshot below:

The image shows the Spring Initializr web interface configured to create a new Spring Boot project. The project is set to use Maven for build automation, Java as the programming language, and Spring Boot version 3.2.5. Project metadata includes group ID 'cloud.sclabs.backstage', artifact ID 'sample-app', name 'sample-app', description 'Demo project for Spring Boot', and the package name 'cloud.sclabs.backstage.sample-app'. The project packaging is set to 'Jar' and the Java version is 17. Dependencies include 'Spring Web' for building web applications using Spring MVC.

Once unpacked we can write some simple logic for the RestController.

package cloud.sclabs.backstage.sampleapp;

import java.util.List;

import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
class EmployeeController {


  @GetMapping("/employees")
  List<String> all() {
    …;
  }
  
  @GetMapping("/employees/{id}")
  Employee one(@PathVariable Long id) {
    
    …
  }

  @PutMapping("/employees/{id}")
  Employee replaceEmployee(@RequestBody Employee newEmployee, @PathVariable Long id) {
    
    …
  }

  @DeleteMapping("/employees/{id}")
  void deleteEmployee(@PathVariable Long id) {
    …
  }
}

In order to create the OPEN API documentation, we simply add the following dependency on the classpath:

<dependency>
<groupId>org.springdoc</groupId>
       <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
       <version>2.5.0</version>
</dependency>

Once we have built and started our application (./mvnw clean spring-boot:run) , we can see the documentation when opening the following URL: http://localhost:8080/swagger-ui/index.html

Backstage Entities

Now it is time to write some YAML for the Backstage entities. We create a backstage folder within the repository and add the following directories and files (mkdocs will be explained within the next blog post):

The image shows a directory structure in a terminal for a project related to Spotify Backstage. The structure includes several YAML files and directories. At the root level, there are 'all-apis.yaml', 'all-components.yaml', and 'all.yaml'. There are also 'apis' and 'components' directories, each containing a respective YAML file: 'employees-api.yaml' in 'apis' and 'employees-component.yaml' in 'components'. Additionally, there is a 'docs' directory with Markdown files: 'architecture.md', 'employees.md', 'index.md', 'sample.md', and 'structure.md'. The root directory also contains 'mkdocs.yaml'.

Let’s take a look into the yaml files. Our entry point is the all.yaml:

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: backstage-sample-app
  description: A collection of all Backstage entities
spec:
  targets:
    - ./all-apis.yaml
    - ./all-components.yaml

---
apiVersion: backstage.io/v1alpha1
kind: System
metadata:
  name: backstage-sample-app
  annotations:
    backstage.io/techdocs-ref: dir:.
    github.com/project-slug: gsoeldner/backstage-sample-app
spec:
  owner: user:default/guido.soeldner

Within the YAML, we define a location and load the all-apis.yaml and the all-components.yaml.

The location entity all-apis.yaml covers the different APIs – in our case just simple one:

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: event-sourcing-poc-apis
  description: A collection of all Backstage event-sourcing-poc APIs
spec:
  targets:
    - ./apis/employees-api.yaml

Finally, we have the employees-api.yaml:

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: employees-api
  description: The employees API
  tags:
    - employees
    - query
    - rest
  links:
    - url: https://github.com/gsoeldner-sc/backstage-sample-app
      title: Server Root
      icon: github
spec:
  type: openapi
  lifecycle: experimental
  owner: user:default/guido.soeldner
  definition: |
	SPEC from Spring APP

The last line covers the Spring App OpenAI spec. The most easy way is to retrieve the API as JSON (http://localhost:8080/v3/api-docs) and replace the line in the sample.

Of course we also need to define the components:

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: event-sourcing-poc-components
  description: A collection of all Backstage event-sourcing-poc components
spec:
  targets:
    - ./components/employees-component.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: employees-component
  description: command side of order aggregate
  annotations:
    backstage.io/source-location: url:https://github.com/gsoeldner-sc/backstage-sample-app
  links:
    - url: https://github.com/gsoeldner-sc/backstage-sample-app
      title: Server Root
      icon: github
spec:
  type: service
  lifecycle: experimental
  system: examples
  owner: user:default/guido.soeldner
  providesApis:
    - employees-api

Once, everything is done, we can easily import the definitions into the Software Catalog.

Once done, we can easily browse all the APIs, similar like if you go to some kind of Developer Pages like the Google API docs.

The image shows an interface of the Spotify Backstage plugin for managing an API called 'employees-api' under the Soeldner Consult brand. The interface is in the 'Definition' tab of the 'employees-api'. It displays the OpenAPI definition, version 0, compliant with OAS 3.0. The server URL is set to 'http://localhost:8080'. Below, it lists the endpoints for the 'employee-controller' with three methods: GET, PUT, and DELETE, each for '/employees/{id}', along with parameters required for the requests. The sidebar on the left includes options such as Home, APIs, Docs, Create, and Tech Radar.

Internal Delevoper Platforms – Part 11: Backstage Entities

In the last blog post, we have shown how to register Software Entities into the Software Catalog and have found out that there are different kind of entities.

As a recap, the Backstage Software Catalog is a centralized system designed to manage and track ownership and metadata for all software within an ecosystem, including services, websites, libraries, and data pipelines. This catalog uses metadata YAML files, stored with the code, which are collected and displayed in Backstage, facilitating easy management and visualization.

Backstage and the Backstage Software Catalog make it easy for one team to manage 10 services — and makes it possible for your company to manage thousands of them.

In detail, the Software Catalog supports two primary use-cases:

1. Management and Maintenance: It provides teams with a consistent view of all their software assets, regardless of type—services, libraries, websites, or machine learning models. This enables teams to efficiently manage and maintain their software.

2. Discovery and Ownership: The catalog ensures all software within a company is easily discoverable and clearly associated with its respective owners, eliminating issues related to „orphan“ software that may otherwise be overlooked or lost within the broader ecosystem.

Entitity Overview

Overall, Backstage and its Software Catalog simplify the management of numerous services, making it feasible for a single team to oversee many services and for a company to handle thousands.

Now it’s time to address these entities, which include Components, Templates, APIs, Resources, and Systems among others. Each entity type has its specific descriptors:

  • Component: This type refers to a software component, usually closely tied to its source code, and is meant to be viewed as a deployable unit by developers. It typically comes with its own deployable artifact.
  • Template: Entities registered as Templates in Backstage have descriptor files that contain metadata, parameters, and the procedural steps required when executing a template.
  • API: This type covers interfaces that generally provide external endpoints, facilitating communication with other software systems.
  • Resource: Describes types that act as infrastructure resources, which usually provide the foundational technical elements of a system, such as databases or server clusters.
  • System: Unlike the singular definition of components, entities marked as Systems represent a collection of resources and components, meaning a system may encompass multiple other entities. The key advantage of this model is that it conceals the internal resources and private APIs from consumers, allowing system owners to modify components and resources as needed.
  • Domain: While Systems serve as a fundamental method for encapsulating related entities, but for enhanced organizational clarity and coherence, it is often beneficial to group multiple systems that share common characteristics into a bounded context. These characteristics can include shared terminology, domain models, metrics, key performance indicators (KPIs), business purposes, or documentation.

Quite interesting there are also organizational entities:

  • User: A user describes a person, such as an employee, a contractor, or similar.
  • Group: Describes an organizational entity, such as for example a team, a business unit, or a loose collection of people in an interest group.

Entity Details

Entities in Backstage are written in YAML and have basically a metadata and a spec section:

The metadata section consists of the following fields:

  • metadata.name: A required field that specifies the name of an entity.
  • metadata.namespace: Used for defining the namespace of the entity and for classifying entities.
  • metadata.annotations: Primarily for listing references to external systems, such as links to a GitHub repository.
  • metadata.links: Specifies which links are displayed in the „Overview“ tab of the entity’s page in Backstage.

On the other side, the spec field’s structure and content depend on the entity type selected in the „kind“ key. It determines how an entry is categorized in the software catalog and the possible relationships among entities.

For the entity type „Component,“ the spec fields include spec.type, spec.lifecycle, spec.system, among other relationship types. The fields within the spec section define essential attributes of the entity, such as the lifecycle stage (e.g., active, production, deprecated) and the entity’s owner, which is typically a person, team, or organizational unit responsible for the entity’s maintenance and development.

As it would be too long for the blog post, we will point to the documentation, where all the manifests are described in detail: https://backstage.io/docs/features/software-catalog/descriptor-format

System Model Overview

All together these elements form a complete system model, which is shown in the following architecture diagram:

This diagram provides a comprehensive view of Spotify's Backstage entities and their relationships.

Key Entities:

Template: Defines parameters used in the frontend and steps executed in the scaffolding process.
Location: References other places for catalog data.
Domain (Orange box): Represents domain models, metrics, KPIs, and business purposes.
System (Yellow box): A collection of entities working together to perform a function.
API (Green box): Represents different APIs, including OpenAPI, gRPC, Avro, etc.
Resource (Light Green box): Contains resources such as SQL databases, S3 buckets.
Component (Light Blue box): Backend services, data pipelines, and similar components.
Group (Blue box): Groups related by type (team, business-unit, product-area).
User (Blue box): Represents users belonging to groups.
Relationships:

Domain is part of System.
System:
Depends on Resource.
Depends on Component.
Part of Domain.
Provides API to API.
Resource:
Part of System.
Contains types: database, S3-bucket, cluster.
Component:
Part of System.
Depends on other Component.
Provides API.
Consumes API.
Types: service, website, library.
Group:
Owned by User.
Has members and sub-groups.
User:
Member of Group.
The diagram uses different colored boxes for distinct entity types and directional arrows to represent relationships.

Relationships

Relations between entities always involve two parties, each assuming a specific role within the relationship. These relationships are directional, featuring a source and a target. In a YAML file, the source entity defines the type of relationship as the key name, while the target entity is specified as the value assigned to this key. For example, in the YAML file of a Component type entity, relationships like `dependsOn`, `providesApis`, `consumesApis`, and `subComponentOf` (noted as `partOf` in diagrams) can be defined as keys, followed by an entity reference according to the previously described pattern.

Each entity in the relationship has a corresponding opposite role, which need not be defined in the YAML file but is used in queries or visualizations of relationships. For instance, if Component A has a relationship role `providesApis`, the referenced Component B would assume the opposite role `apiProvidedBy`.

Let’s recap our example from before:

apiVersion: backstage.io/v1alpha1 
kind: Component 
metadata:
 name: exampleappfrontend
 description: Simple Webseite
 links:
 - url: http://simple-website.sclabs.cloud
   title: ExampleApp
   icon: web
 annotations:
   github.com/project-slug: sclabs/exampleApp/frontend/
   backstage.io/techdocs-ref: dir:.
 spec:
   type: Website
   lifecycle: production
   owner: Joseph Cooper
   system: exampleapp
   consumesApis: ['component:exampleappservice']

In this application, the `exampleappfrontend` in its descriptor file might have a key-value pair `consumesApis: [‚component:exampleappservice‘]`, indicating a reference to the `exampleappservice` component, which serves as a backend providing an API.

Entity Lifeycylce

In the Backstage Software Catalog, the process for registering entities follows a standardized technical flow, regardless of how the entities are initially registered. This flow can be visualized in a comprehensive diagram, often referred to as „The Life of an Entity.“

This diagram represents the data flow of Spotify's Backstage catalog ingestion, processing, and stitching pipeline.

Pipeline Stages:

Ingestion:
External Sources (Red box): The origin of entity data.
Entity Providers (Green box): Components that ingest data from external sources.
Unprocessed Entities (Yellow box): Entities directly fetched from providers.
Edges (Yellow box): Relationships extracted between unprocessed entities.
Processing:
Processors (Green box): Modules that transform unprocessed entities.
Processed Entities (Yellow box): Entities after processing.
Relations (Yellow box): Extracted relationships among processed entities.
Errors (Yellow box): Issues detected during processing.
Stitching:
Stitcher (Green box): Combines processed entities and relationships into the final set.
Final Entities (Yellow box): Fully processed entities ready for use.
Search (Yellow box): Indexes entities for quick searching.
Output:
Catalog API (Red box): Serves the final entities via an API.
Directional arrows represent the flow of entities through the different stages and components.
  • Entity Ingestion and Provider: The process begins with the „Entity Provider“ which collects raw entity data from specified sources and translates them into unprocessed entity objects. To avoid duplicates, the database tracks which provider has ingested which entity. This stage also includes an initial validation to ensure critical fields like ‚kind‘ and ‚metadata.name‘ are present.
  • Entity Processing: The next step involves processing these unprocessed entities. This includes validation and further processing through „Policies“ and „Processors.“ Policies are sets of rules for validation, while Processors apply these rules to validate the entities. This step may involve exporting relationships, error messages, or the entity itself from the raw data.
  • Entity Stitching: The final step is „Stitching,“ where processed entities, along with any error messages and relationships, are retrieved from the database and combined into the final entity that will be used in the Software Catalog. This process considers relationships that might be defined in other entities and handles any error messages, displaying them within the catalog as necessary.

Throughout these steps, developers have the flexibility to implement custom Providers and Processors to fetch entities from unique sources or at specific intervals, though some system constraints like processing intervals are predefined. The recommended method for automating entity ingestion in the catalog is through Custom Entity Providers. Once all sub-steps are completed, the final entities are stored in the internal database and presented in the Software Catalog, ready for use.

Internal Developer Platforms – Part 10: Working with Software Templates and Backstage Search

After having described how to install Backstage, we want finally show how to work with Backstage. In this blog post, we will focus on software templates and the Backstage Search.

Software Catalog

The Software Catalog is a fundamental component of a Backstage deployment, serving as an essential management tool. After deployment and login, users can easily access the Software Catalog by clicking the „Home“ button. This interface presents a list of all registered entities—such as software components, resources, or users—and offers filtering options to navigate through them. Entities, defined in YAML files and registered in Backstage, represent either physical or virtual organizational units, each classified by type and kind. Initially, the catalog displays entities of the „Component“ type, but users can adjust the view to include „Location,“ „Resource,“ „System,“ and „Template“ types based on available registrations. The availability of entity types in the dropdown menus depends directly on their presence in the catalog, exemplifying the customizable and interconnected nature of the Backstage ecosystem.

As a user, we can register applications into Backstage in different ways:

  1. Direct Creation from Backstage: Users can create and register new entities directly in Backstage using pre-existing templates. This involves selecting a template via the „Create…“ button in the sidebar, which leads to the „Create a New Component“ page. After completing the creation and registration process, the new component becomes available in the Software Catalog.
Screenshot of the Soeldner Consult Catalog user interface displaying a list of owned components in a management system, similar to the component feature in Spotify Backstage. The interface shows a navigation bar on the left with options like Home, APIs, and Docs. The main part of the interface lists components like fw-alexander and fw-bjorn, all owned by Guido Söldner, marked as 'experimental' under the service type. The top right includes buttons for 'Create' and 'Support'.

2. Configuration File Entry: Entities can be added by specifying a „Location“ in the app-config.yaml configuration file, which links to a YAML descriptor file for each entity. These files are read and their entities are registered in the Software Catalog upon deployment.

catalog:
locations:
- type: url
target: https://github.com/backstage/software-templates/blob/main/scaffolder-templates/react-ssr-template/template.yaml
rules:
- allow: [Template]
- type: file
target: template.yaml # Backstage will expect the file to be in packages/backend/template.yaml


 

3. Custom Entity Providers or Processors: For companies with extensive software portfolios, Backstage’s backend can be extended with custom entity providers or processors. These read from internal or external sources to generate new entities automatically and regularly. For example, you can use a provider for GitHub to automatically load entities like users from an external source into your Backstage instance.

4. Manual Registration via UI: If a component is already stored online in a repository with the required YAML descriptor file, it can be manually registered. Users navigate to the „Create a New Component“ page, click „Register existing component,“ and input the URL of the YAML file. After verification, the „Import“ button adds the component to the Software Catalog.

Screenshot of the 'Register an existing component' page on the Soeldner Consult interface, mirroring functionality found in Spotify Backstage for tracking software components via SC GitOps. The interface features a step-by-step wizard on the right to start tracking by entering a URL to a source code repository and linking to an entity file, with an example URL provided. The left side displays a navigation panel with options like Home, APIs, Docs, and Tech Radar. The top right corner includes a 'Support' button.

How to define entities

When manually registering an entity, a YAML file must be available for the corresponding entry. Each entry in the Software Catalog corresponds to a YAML file, ideally housed alongside the code of the related software in a repository. This YAML file contains metadata and additional information required to display the component in the Software Catalog. While users can choose any name for the YAML file, known as the „descriptor file,“ Backstage recommends the name „catalog-info.yaml.“ Consequently, when creating a repository, especially in „mono-repos“ where multiple components are hosted in a single repository, it’s crucial to structure the repository using folders to avoid name conflicts with existing descriptor files.

In the following we show how you could define your repository structure to register an application consisting of a backend with api and db, and a frontent:

The structure of the YAML files used by Backstage must adhere to specific rules for the entries to be properly recognized during the import process. An example of such a descriptor file, specifically for the exampleApp frontend component, illustrates the essential structure which consists of elements like `apiVersion`, `kind`, `metadata`, and `spec` that are mandatory in the YAML file. The `apiVersion` describes the specification format version for the entity the file is created for, consistently set as `backstage.io/v1alpha1` in the current project. The `kind` field defines the type of entity it should be listed as in the Software Catalog. Recognized types by Backstage and its plugins include Component, API, Group, User, Resource, and Location, among others. Backstage provides detailed definitions for all entity types in its documentation, assisting the platform team in orientation and categorization of the units represented.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: exampleappfrontend
  description: Simple Webseite
  links:
    - url: http://simple-website.sclabs.cloud
      title: ExampleApp
      icon: web
  annotations:
    github.com/project-slug: sclabs/exampleApp/frontend/
    backstage.io/techdocs-ref: dir:.
spec:
  type: Website
  lifecycle: production
  owner: Joseph Cooper
  system: exampleapp
  consumesApis: ['component:exampleappservice']

That’s it for today. In the next blog post, we will describe the different entity types in detail.

Internal Developer Platforms – Part 9: Backstage Kubernetes Deployment

In the last blog post, we talked a lot of the installation and the architecture of Backstage. We also gave some design considerations for the Kubernetes installation. Now it is time to come up with some details for the Kubernetes installation. 

We will use our design from the last blog post as a basis:

A deployment flowchart for a Kubernetes setup used in Spotify Backstage. The process begins with the Internet cloud icon, connected to a yellow rectangle representing an ingress resource labeled 'Ingress: backstage-ingress'. This ingress directs traffic to two services within the cluster. The first service, outlined in a red dashed rectangle, is labeled 'Service: backstage' of 'Type: ClusterIP', indicating it's internally accessible within the cluster. It routes to a blue dotted rectangle symbolizing a pod, with a green rectangle inside denoting the 'container: backstage'. This container is part of the 'Deployment: backstage'. An arrow points from the backstage service to another service for the Postgres database, also a red dashed rectangle labeled 'Service: postgres' of the same type. It connects to another pod, indicated by a blue dotted rectangle containing a 'container: postgres', above which sits a green cylinder representing the 'Postgres database'. This setup within the pod is part of the 'Deployment: postgres', highlighting the backend database connectivity for the application.

In the following, we will use a namespace called “backstage” for the installation. If you don’t have it so far, simply create it with

kubectl create ns backstage

Postgres Database

Let’s begin with the Kubernetes Deployment for the Postgres database. We need the following components:

  • A Persistent Volume Claim for storing the data
  • A Kubernetes Secret to store the credentials
  • The Postgres Deployment itself 
  • A service manifest to expose the deployment

Lets begin with the Persistent Volume Claim:

# kubernetes/postgres-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: backstage
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2G

Now continue with the Secrets manifest. Remember that the data is base64 encoded. So if our user is “backstage”, the password is “test” and the port is 5432, the corresponding YAML looks like the following:

# kubernetes/postgres-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secrets
  namespace: backstage
type: Opaque
data:
  POSTGRES_USER: YmFja3N0YWdl
  POSTGRES_PASSWORD: dGVzdA==
  POSTGRES_SERVICE_PORT: NTQzMgo=

Now let’s continue with the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13.2-alpine
        imagePullPolicy: 'IfNotPresent'
        ports:
        - containerPort: 5432
        envFrom:
        - secretRef:
            name: postgres-secrets
      volumes:
      - name: postgresdb
        persistentVolumeClaim:
          claimName: postgres-pvc

Now, we expose the Postgres deployment as a Service:

# kubernetes/postgres-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: backstage
spec:
  selector:
    app: postgres
  ports:
    - port: 5432

Backstage Components

After having the database running in our Kubernetes cluster, we can continue with Backstage installation itself. We need the following components:

  • A Secret manifest for storing credentials. This includes values for Backstage like GitHub Tokens or docker config json files.
  • The Backstage deployment
  • The Backstage service 
  • An ingress

Let’s begin with the secret:

# kubernetes/backstage-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: backstage-secrets
  namespace: backstage
type: Opaque
data:
  GITHUB_TOKEN: Z2hwX0Q3VGfc bayernblablawdsFsSnEwOW44RW1JMm0fcn2OEZrSDfuerthROMaufstiegDdJQg==
---
apiVersion: v1
kind: Secret
metadata:
  name: pull-secrets
  namespace: backstage
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: esadfjlegojksdfalkideenbuchsfd6000sdfjkls25Fd09XNDRSVzFKTW0wzasfdsfdoiURkSlFnPTfs0ifX19

The following manifest depicts the Backstage deployment (remember you need your backstage container image stored in some registry):

# kubernetes/backstage.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backstage
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backstage
  template:
    metadata:
      labels:
        app: backstage
    spec:
      containers:
      - name: backstage
        image: ghcr.io/my-backstage-project/backstage:0.1
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 7007
        envFrom:
        - secretRef:
            name: postgres-secrets
        - secretRef:
            name: backstage-secrets
      imagePullSecrets:
        - name: pull-secrets
 
 
# Uncomment if health checks are enabled in your app:
# https://backstage.io/docs/plugins/observability#health-checks
#          readinessProbe:
#            httpGet:
#              port: 7007
#              path: /healthcheck
#          livenessProbe:
#            httpGet:
#              port: 7007
#              path: /healthcheck

Now let’s expose backstage as a service:

# kubernetes/backstage-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: backstage
  namespace: backstage
spec:
  selector:
    app: backstage
  ports:
    - name: http
      port: 80
      targetPort: http

Finally, we need an Ingress:

# kubernetes/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: backstage
  name: backstage-ingress
spec:
  defaultBackend:
    service:
      name: backstage
      port:
        number: 80

Setting up an Ingress differs a little bit in every Kubernetes distribution. In the following, we use Google Kubernetes Engine and the manifest would automatically create a Google Load Balancer. In addition, we don’t you HTTPs for our deployment. Setting up encryption would need an certificate and some configuration. However, as we want to concentrate on Backstage, we will skip these configuration here.

That’s it for this blog bost. In the next post, we will focus on some Backstage internals.

Internal Developer Platforms – Part 8: Spotify Backstage Architecture and Installation

After having discussed the main use cases in the last blog post, let’s now continue with some internals. In this blog post, we will discuss the architecture and the installation of Backstage.

Architecture

Backstage is structured around three core components: Core, plugins, and the application instance. The „Core“ provides the basic framework and essential functionalities like the UI interface, including elements such as the navigation bar on the left side. The application itself is a configured instance of a Backstage app, maintained by the platform team, integrating functionalities from Core and various plugins into a cohesive unit for production environments.

Plugins are vital, supplying additional functionalities either developed internally or sourced externally. These are divided into frontend and backend plugins. Frontend plugins are typically client-side React applications linked to the Backstage UI and accessible via specific URLs like „/catalog“ for the Software Catalog. These plugins export „Extension Components“ from the /backstage/plugins directory, which integrate into the frontend with routing information and metadata.

Backend plugins handle operational aspects and may incorporate services like caching systems and databases (primarily PostgreSQL or SQLite for production) managed within a Kubernetes cluster. Each plugin may create its logical database for isolation, enhancing the system’s organization and scalability.

The architecture heavily utilizes npm packages to deliver libraries and structure the code, with both frontend and backend systems employing various packages to define plugin boundaries and facilitate their integration. This structure not only enhances the application’s modularity and scalability but also supports future developments and optimizations in the backend system.

The package infrastructure is depicted in the following picture.

An architecture diagram detailing the structure and dependencies of a Backstage application. The diagram is divided into various sections: 'Frontend App Core', 'Frontend Plugin Core', 'Frontend Libraries', 'Plugin Libraries', 'External Plugin Libraries', 'Common Tooling', and 'Common Libraries'. Each section contains packages with namespaces like '@backstage/core-app-api' and '@backstage/app-defaults' for the Frontend App Core, '@backstage/core-plugin-api' and '@backstage/test-utils' for the Frontend Plugin Core, and others like '@backstage/theme' and '@backstage/integration-react' for Frontend Libraries. Plugin Libraries section includes individual plugin identifiers with React, common, and node modules. External Plugin Libraries follow a similar pattern but for plugins not native to the main application. Common tooling and libraries feature CLI tools and shared resources like '@backstage/cli', '@backstage/types', and '@backstage/config'. Lines connect these components to demonstrate their interoperability and how they contribute to the overall functionality of the Backstage ecosystem. Color coding differentiates between frontend packages, isomorphic packages, backend packages, and CLI packages for clarity.

Installation Considerations

In the following we will provide a detailed summary of deploying and setting up a version 1.15 Backstage instance using a Kubernetes cluster. The installation process begins locally, where Backstage can be installed on either a Linux system or through the Windows Subsystem for Linux (WSL). Essential tools required include curl, wget, yarn, git, docker, and the Node Version Manager (nvm). Nvm is used to install Node.js in its current Long-Term Support (LTS) version. The Backstage application, for example named „my-backstage-app,“ is created via the command line using

`npx @backstage/create-app@latest`.

The directory structure generated during this setup includes a `/node_modules` folder for Node.js project modules, and other critical folders and files like `app-config.yaml` and `app-config.production.yaml`, essential for configuration. These YAML files distinguish between test and production settings, allowing tailored configurations for different environments. Notably, `app-config.production.yaml` has the ability to override settings in `app-config.yaml` if similar settings are present.

Once installed, the application can be started locally using

`yarn dev`,

making the frontend accessible at `localhost:3000` and the backend at `localhost:7007`. At this point, the instance includes fundamental features such as the Software Catalog. Developers are free to further customize or extend functionalities from this base.

Integration with GitHub is essential for managing the Software Catalog. It necessitates setting up an authentication system within Backstage, typically configured for OAuth with GitHub. This setup involves registering the application in GitHub to obtain a client ID and secret, and configuring the `app-config.yaml` to use these credentials.

For deployment in a Kubernetes environment, `app-config.yaml` is further configured to specify PostgreSQL as the database, pulling connection details from the Kubernetes environment variables. It also ensures that backend requests are not upgraded from HTTP to HTTPS in the absence of a valid certificate.

The Kubernetes deployment is based on building a Docker image from the Backstage setup. This image is then pushed to a public Docker Registry. The Kubernetes configuration involves a series of YAML files that define the service architecture, including the setup for the PostgreSQL database and Backstage services, and the establishment of an ingress for public access. Deployment begins with the PostgreSQL database to ensure that the necessary database scripts run before the Backstage services start.

Finally, the entire setup within the Kubernetes namespace „backstage“ involves deploying the database, secrets, services, and ingress to ensure that Backstage is fully operational and accessible via a public IP address defined in the ingress configuration. This setup ensures that Backstage is scalable, secure, and ready for production use.

A simplified Kubernetes deployment diagram illustrating the connection from the Internet to a Postgres database within a containerized environment. The flow starts with the Internet symbol, leading to a Kubernetes cluster represented by a dashed yellow rectangle labeled 'Ingress: backstage/ingress'. Inside the cluster, there are two services: the 'Service: backstage' with a red border, connected to the 'container: backstage', and the 'Service: postgres' also with a red border, which is linked to the 'container: postgres'. Each service is of type 'ClusterIP' suggesting internal cluster communication. The 'container: backstage' is encapsulated within a blue-dotted rectangle, indicating a 'Pod', and labeled 'Deployment: backstage'. Similarly, the 'container: postgres' is part of a 'Pod' labeled 'Deployment: postgres', with a green cylinder representing the 'Postgres database'. This diagram effectively demonstrates the path of a request from the internet through the ingress to the services running in a Kubernetes environment.

That’s it for today. In the next blog post, we will come up with some details for the installation.

Internal Developer Platforms Series


Welcome to our blog series dedicated to unraveling the complexities and potential of Internal Developer Platforms (IDPs). Whether you’re a seasoned tech professional or just embarking on your journey into the world of software development, this series is crafted to guide you through the fundamental aspects of IDPs, their pivotal role within modern development ecosystems, and how they can transform your team’s efficiency and creativity.

Part 1: Introduction

Part 2: What is an Internal Developer Platform?

Part 3: Components of an Internal Developer Platform

Part 4: Deployment Options

Part 5: Spotify Backstage

Part 6: Spotify Backstage Functionality

Part 7: Spotify Backstage Core Functionality

Part 8: Spotify Backstage Architecture and Installation

Internal Developer Platforms – Part 7: Spotify Backstage Core Functionality

As discussed in the last blog post, Backstage already comes with a lot of built-in features. It’s time to discover those functionalities. In detail, we will cover the following:

  • Backstage Search
  • Backstage Catalogue
  • Backstage Tech Docs
  • Backstage Software Templates
  • Backstage Plug-ins

Backstage Search and Backstage Catalogue

For effective management and administration of software applications, the „Software Catalog“ feature of Backstage offers a centralized system for managing internal software within a company. It serves as a comprehensive listing of all developed applications, with each software component represented by its own entry. This allows for a quick overview of the company’s software availability. Each entry can be enriched with metadata to display dependencies between software components or to define the software type. The Software Catalog also facilitates the assignment of software to specific individuals, groups, or organizational units, making it easier to identify ownership and areas of responsibility. Moreover, each software component has various tabs that provide extensive information and management tools. Additionally, Backstage’s integrated „Backstage Search“ function enables searching the Backstage ecosystem for terms, components, people, and more, with the capability to extend search to the StackOverflow platform and integrate other search engines to customize search logic according to needs.

Screenshot of a search interface from an internal developer platform, possibly Spotify Backstage, showing a search bar with the query 'how to'. The results display a variety of programming and software development related questions such as 'How do I redirect to another webpage?', 'How to modify existing, unpushed commit messages?', 'How do I force "git pull" to overwrite local files?', and 'How do I revert a Git repository to a previous commit?'. Each question is tagged with relevant programming languages and tools such as JavaScript, query, git-commit, git-rewrite-history, version-control, git-pull, git-fetch, git-checkout, and git-revert, and includes the number of answers provided and the author's name. This indicates a knowledge sharing platform where developers can find answers to common coding and version control questions within their organization.
A screenshot of the Service Catalog interface from Spotify Backstage, showing an overview of services managed within an organization. The left pane categorizes services under 'Personal' with 32 owned and 2 starred, and under 'Spotify' with all services totaling 824. The main panel lists 'Owned (32)' services that the user currently manages, including services like 'podcast-api', 'artist-lookup', and 'searcher', each associated with a system, such as podcast or artist, and tagged with the tool or language used, like Python or Websites. The lifecycle stage for all listed services is 'Production', and their status varies with indications like 'Up and running', 'Increased failure rates', or 'Deployment failed'. Each service has a description, typically 'Graphql backend for Playlist', and actions for quick access like editing settings or viewing details. The interface also displays the current time in different cities, including NYC, UTC, LON, and STO.

Backstage Tech Docs

The „TechDocs“ feature is designed to aid in the management and administration of technical documentation, offering users a centralized platform to view and provide technical documentation for software components. Documentation can be accessed either through a tab on the respective software component’s page or via a button on the sidebar. Initially, developers write the documentation in the common Markdown format, which is then rendered for the website display using the „MkDocs“ Python module by Backstage. Users have the ability to structure the documentation with a table of contents, split the documentation across multiple Markdown files, and enhance TechDocs with additional functionalities through „TechDocs Addons“.

A screenshot of a documentation webpage titled 'Backend Golden Path' within an internal development platform, likely Spotify Backstage. The page is an introduction to a series on developing and operating server-side infrastructure. The main content explains that this tutorial is the first in the series and emphasizes the importance of setting up a local environment for backend development. It notes 'Operating System Variations', indicating that the tutorials are tailored for Apple machines running macOS, with alternative instructions for Linux or Windows users mentioned in callouts. The sidebar on the right includes tags related to the content such as #backend, #apollo, and #golden-path, and a section titled 'Stack Overflow' indicating no questions found for these tags. Other sections include 'Prerequisites' and 'Support'. The page header shows the component name 'backend-golden-path', the site owner 'pulp-fiction', the page owner 'fabric', and indicates the page was last updated a month ago. The top contributors are also acknowledged, symbolizing a collaborative documentation environment.

Backstage Software Templates

„Software Templates“ provide a smart way for creating new software or infrastructure components. Users are guided through the creation process with a pre-made template, where they can configure the component at each step by entering parameters, culminating in its creation with a simple button click. The output generated by the template is determined by its creator and can vary, including loading skeleton code, creating a new Git repository, or writing input parameters to a file. Additionally, there’s an option to register the newly created component in the Software Catalog. The process can be further customized to meet specific requirements by writing custom template actions, allowing organizations to establish standards and best practices for the creation of certain components.

A screenshot of a user interface from an internal development platform titled 'Create a New Component', showing a selection of available templates for software development. The 'Available Templates' section highlights 'Recommended' templates, including 'Documentation Template', 'React SSR Template', and 'Spring Boot gRPC Service'. Each template has a brief description, the owner (example@example.com), and associated tags like 'recommended', 'techdocs', 'mkdocs', 'website', 'react', 'service', 'java', and 'gRPC'. Below, in the 'Other Templates' section, more options are shown such as 'Create React App Template' and 'Pull Request Action Template'. The interface is designed for easy navigation, with categories and tags on the left for filtering the templates, and prominent 'Choose' buttons for template selection. A sidebar includes options for registering an existing component and seeking support, illustrating a comprehensive tool for managing and scaffolding software projects within the organization.

Integrations and Plugins

Backstage facilitates the integration of external platforms and cloud providers onto a unified platform, leveraging various services that use external providers for storing or publishing Backstage-specific content. For example, the Backstage Software Catalog can access external providers like GitHub to extract relevant information from YAML files, necessary for incorporating software components into the Software Catalog. Fundamentally, Backstage operates as a „single-page“ application, providing a technical framework in which various plugins, implementing desired functionalities, operate. These plugins, including „Core-Features“ provided by default installed plugins, can be extended with additional services via 3rd-party plugins available on the Marketplace. This Marketplace, accessible through the Backstage website, features plugins developed by Spotify, other organizations, or individuals, offering over 134 plugins across categories like „CI/CD,“ „Monitoring,“ „Infrastructure,“ or „Deployment.“ Plugins are typically installed using a package manager like npm, and by manually adding code to Backstage’s source code, integrating them into the existing tooling environment. Examples from the Marketplace include „Scaffolder Git Actions“ Hill or „Argo CD“. While plugins available in the Marketplace are generally free, Spotify also offers „Premium-Plugins“ as part of a paid subscription, developed by Spotify developers and supported officially by Spotify, such as the „Role-Based-Access-Control (RBAC)“ or „Soundcheck“ plugins.

A screenshot of the Backstage platform's plugin marketplace. The marketplace showcases a grid of plugin cards against a dark background, each card representing a different plugin with a distinct icon, title, and brief description. Examples include 'APIs with 3scale' by Red Hat for synchronizing 3scale content, 'Architecture Decision Records' by Phil Kuang for browsing project ADRs, and 'AI Assistant - RAG AI' marked as new, for querying language models with contextual information. Others include 'Allure Reports' by Deepak Bhardwaj for viewing reports, various analytics modules like Google Analytics by Spotify, Matomo Analytics by Red Hat, and Quantum Metric for tracking usage, along with monitoring tools such as 'Airbrake' by Simply Business. Additional plugins featured are 'Announcements' by procore-oss for writing and sharing announcements, 'API Docs' by SDA SE for API discovery, 'API Linter' by Zalando for compliance checking, and 'API SDK Generator' by Konfig for SDK generation. Each card has an 'Explore' button, facilitating easy access to the plugin details and installation options. The overall layout is user-friendly and informative, aiming to enhance the development environment with various integrations and tools.

Internal Developer Platforms – Part 6: Spotify Backstage Functionality

We already discussed in the earlier parts of this blog post series that in the context of software development, developers face a myriad of tasks that can be overwhelming. An Internal Developer Platform (IDP) or an Internal Developer Portal can aid developers in addressing these tasks more efficiently, thereby reducing their cognitive workload. 

Let’s discuss the challenges developers may encounter during software development and show how a platform like Backstage can support developers in overcoming these challenges in the following. We will discuss problems that can be solved with Backstage out-of-the box.

Management of software applications

New developer during the onboarding process in a company are often lacking an adequate system for managing and accessing existing software applications. This inadequacy makes it difficult for the newcomer to grasp the company’s IT infrastructure, find and understand the software available, and know who is responsible for what within the development process. Such challenges include not knowing who oversees certain software, who contributes to its development, or what framework versions are used. Additionally, new developers might be unaware of dependencies that the company’s applications have on other systems like databases or interfaces, leading to potential complexities due to the lack of documentation. As development progresses, further inquiries may arise, such as the existence of software pipelines for specific applications or the details of the software deployment process. While experienced developers could offer assistance, this would result in productivity losses by diverting valuable resources. In summary, the absence of a dedicated tool can significantly delay a developer’s ability to begin contributing productively to software creation.

Working with documentation

Besides, new developers may face challenges, particularly with software documentation during the onboarding process. Without a dedicated software system, accessing information about existing software becomes difficult. Even if information about what specific software does is available within the company, locating this information is not straightforward for new developers. Documentation may be scattered across various platforms such as Google Docs, Readme files, Confluence, or GitHub Pages. Centralized document management and access might either be non-existent or only possible through cumbersome and inefficient methods, like navigating through disorganized collections of links. Moreover, even if the right documentation is found, there’s no guarantee that the information is up-to-date or accurate. Additionally, details about the document’s author or the software version it pertains to might be missing. If a developer discovers an error in the documentation and wishes to report it, identifying and notifying the author can be challenging. Other developers might not be aware of the potential error or could report it again, leading to a non-transparent and inefficient reporting process that negatively impacts productivity. A centralized repository for documentation could potentially solve these issues.

Create software components in a standardized way

Often, the creation of components is not standardized. When building an application with the Angular framework, for example, a developer must first manually generate skeleton code via the command line, requiring knowledge of Angular’s `ng` tool to establish the app’s basic structure. This step is followed by creating a new GitHub repository, pushing the application to it, and setting up software pipelines, all of which demand additional knowledge and time. The developer must also ensure adherence to predefined schemas, standards, and best practices during application development.

Similarly, creating infrastructure components, possibly through GitOps processes, requires understanding GitOps principles and specific details about the component’s definition, including its structure, required parameters, valid values, target Git repository, and operational infrastructure. Manual attention is needed to comply with predefined standards and rules in this scenario too.

For both scenarios, there’s a possibility that the developer may need to modify or delete components post-creation. A tool that consolidates these tasks, automates them, and standardizes the processes would not only relieve mental burden but also save time and reduce the likelihood of errors, allowing developers to quickly return to productive development work.

Integration of external systems

There is a diversity of services available to developers in software development, covering areas like orchestration, databases, Git, deployment, networks, and analytics. These services are typically accessed via the command line or their respective websites, requiring developers to navigate multiple platforms to perform desired actions. The text suggests a significant improvement in efficiency and usability could be achieved by centralizing access to these services on a single platform, allowing developers to perform all necessary actions in one place. For example, obtaining a quick overview of the status of CI pipelines without having to visit GitHub directly. Such a centralized platform should also be flexible enough to incorporate additional services or custom tools to meet the unique needs encountered in software development. Implementing these optimizations could enhance the overall quality of life for developers by simplifying the operation and management of the software landscape, and by increasing productivity in the software development process.

Summary and Outlook

So, it is easy to see, that there are plenty of scenarios where Backstage can really bring benefits out-of-the-box without needing to apply too much customizations.

In the next blog posts, we will have some deep dive into Backstage. In particular, we will introduce the following Backstage core functionalities:

  • Backstage Search
  • Backstage Catalogue
  • Backstage Tech Docs
  • Backstage Software Templates
  • Backstage Plug-ins

Internal Developer Platform – Part 5: Spotify Backstage

Introduction to Backstage

After having written a lot about IDPs in general, it is now time to shift our focus on Spotify Backstage in the next blog posts. Let’s begin with a general introduction.

The Essence and Adoption of Backstage

Backstage is a comprehensive framework designed to go beyond the traditional scope of Internal Developer Portals by offering an open platform capable of constructing tailored developer portals. Distinct from being a standalone Internal Developer Portal written in the blog post series before, Backstage enables development teams to create a portal that align with their specific requirements. Its ongoing development under Spotify underscore its credibility and the innovative approach towards addressing common developmental and operational challenges.

Due to its effectiveness and versatility, Backstage has seen widespread adoption across the tech industry, with over 1000 organizations and more than a million individual developers leveraging the platform. Among its notable users are big IT enterprises such as Netflix, Zalando, and DAZN, showing the platform’s capacity to serve a diverse range of development environments and organizational sizes. This widespread adoption is further validated by Backstage’s inclusion in the „Incubating Projects“ of the Cloud Native Computing Foundation (CNCF), highlighting its potential for future growth and evolution within the cloud-native ecosystem.

The drive behind Backstage’s development was Spotify’s own experience with rapid growth, which brought about significant infrastructural fragmentation and organizational complexities. These challenges, common in fast-scaling tech environments, led to inefficiencies, including reduced productive time for developers and increased cognitive load due to constant context-switching and the need for navigating disparate tools and systems. Backstage was conceived as a solution to these challenges, aiming to streamline and centralize the development process through a unified platform that abstracts away the complexity of underlying infrastructures and toolsets.

Centralization and Customization Through Backstage

Key to Backstage’s functionality is its ability to serve as a central hub for various development-related activities and resources. It offers platform teams the tools to consolidate engineering and organizational tools, resources, technical documentation, and monitoring capabilities for CI/CD pipelines and Kubernetes, among other features. This centralization is facilitated by a user-friendly visualization layer and a variety of self-servicing capabilities, which are foundational to the philosophy of Internal Developer Portals (IDPs). These features are designed to empower developers by providing them with the means to manage software components, rapidly prototype new software projects, and access comprehensive technical documentation—all within a single, integrated platform.

Furthermore, Backstage’s extensible, plugin-based architecture encourages the integration of additional platforms and services, enabling teams to customize and expand their developer portals according to evolving needs. This architecture supports a vibrant ecosystem of plugins, contributed by both external developers and Spotify’s own engineers, available through the Backstage marketplace. This ecosystem not only enhances the platform’s capabilities but also fosters a community of practice around effective development operations (DevOps) and platform engineering principles.

In summary, Backstage represents a strategic tool for addressing the complexities and inefficiencies associated with modern software development and platform engineering. Its development is a response to real-world challenges faced by one of the most innovative companies in the tech sector, and its adoption by leading tech firms underscores its value and effectiveness. Through its comprehensive suite of features, flexible architecture, and supportive community, Backstage offers a promising pathway for organizations looking to enhance their development practices and infrastructure management.

The following table summarizes these keypoints:

AspectDetails
Nature of BackstageAn open framework for creating tailored Internal Developer Portals, not just a portal itself.
Development and AdoptionDeveloped by Spotify, with over 1000 adopters and more than a million developers. Notable users include Netflix, Zalando, and DAZN.
CNCF InclusionIncluded in the „Incubating Projects“ of the Cloud Native Computing Foundation (CNCF), indicating potential for growth.
Purpose and OriginCreated to address Spotify’s challenges with infrastructural fragmentation and organizational complexities during rapid growth.
Core FunctionalityServes as a central hub for development tools, resources, technical documentation, and monitoring of CI/CD pipelines and Kubernetes.
Self-Servicing CapabilitiesEmpowers developers with tools for managing software components, prototyping, and accessing technical documentation.
ArchitecturePlugin-based, allowing for integration of additional services and customization to meet evolving needs.
Community and EcosystemSupported by a vibrant ecosystem of plugins, contributed by both Spotify and external developers, available on the Backstage marketplace.

For integrating Backstage into a productive environment, Spotify recommends delegating the portal’s maintenance and development to a dedicated Platform Team. This team is tasked with ensuring that both internal developers and other infrastructure and platform teams transition to actively using the portal. The goal is to establish Backstage as the central platform for all software development activities within the organization. To facilitate the platform’s adoption, Spotify suggests various tactics and identifies metrics to measure the adoption process. These tactics and metrics, while described in the context of Backstage, could generally apply to the adoption of any Internal Developer Platforms or portals. Additionally, the Platform Team is responsible for implementing best practices in consultation with technical leaders or architects.

Internal Developer Platform – Part 4: Deployment Options

In this blog post, we want to discuss the pros and cons of the different deployment options

Deployment options

When introducing an Internal Developer Platform (IDP), companies have basically two option: building it in-house or acquiring a complete package from an external service provider. For in-house development, the responsibility lies with the operations team or a designated „Platform Team“ within the company. This team’s primary functions include creating, further developing, and maintaining the IDP. They closely interact with other organizational members to identify issues at the team and company levels, set basic configurations for applications, and manage permissions. Additional tasks involve ensuring „Standardization by Design,“ managing infrastructure, Service Level Agreements, optimizing workflows, and configuring the IDP to automate recurring processes.

Contrastingly, the Platform-as-a-Service (PaaS) approach involves an external provider offering the platform, taking on these responsibilities. Companies with specific IDP requirements and an in-house Platform Team generally prefer self-development. Instead of starting from scratch, the team can utilize various tools and services to assemble the platform. These tools and services, categorized into platform orchestrators, portals, service catalogs, Kubernetes Control Planes, and Infrastructure Control Planes like Terraform and GitLab CI, each handle specific functionalities of an IDP. Existing tools and technologies like PostgreSQL, Kubernetes, and GitLab CI can be integrated into the IDP.

Backstage Tech Stack

A potential tech stack for an IDP might include Backstage, serving as an internal developer portal with features like a software catalog and tools for monitoring and technical documentation management. Backstage can be extended through plugins for new functionalities. ArgoCD, a GitOps tool, would handle dynamic application deployment in Kubernetes clusters, while Terraform allows developers to provision the desired infrastructure. In addition, companies like Soeldner Consult offer ready to use GitOps plug-ins for the self-service of infrastructure deployments.

Cost considerations

Tool selection often considers cost structures, with distinctions mainly in usage-based or ongoing expenses. While open-source tools like Backstage, ArgoCD, and PostgreSQL are available for free, commercial tools like the Internal Developer Portals „Roadie“ or „configure8“ have usage fees. Despite being open-source, some tools may incur costs for underlying infrastructure operation or developer effort for setup and maintenance. 

Along with the complex cost structure, differences in integration concepts, scope, and hosting models must be considered when selecting tools. The self-development of IDPs presents challenges like increased effort and a steep learning curve for tool implementation, necessitating a capable platform team with the necessary knowledge and resources. While complex configurations may prolong the IDP’s readiness, self-development offers design freedom, allowing the platform team to tailor the platform to specific requirements.

IdP for Startups

For startups, small companies, or organizations lacking the necessary resources and knowledge to independently develop and operate an Internal Developer Platform (IDP), leveraging pre-built solutions from third-party providers might be a practical choice. This route bypasses the complex tasks of setup, maintenance, and further development, as they are primarily executed by the third-party vendor. Similar to custom development, both open-source and closed-source ready-made solutions are available. Typically, these solutions are offered as „Platform-as-a-Service“ (PaaS) products, encompassing a wide array of toolsets. „Mia-Platform“ and „Coherence“ are examples of such platforms. Many providers also allow for the integration of services from external providers, like GitHub repositories or Kubernetes tools, and often feature their own developed tools designed to be integrated into the IDP.

Most providers offer official support, a benefit not always guaranteed with self-developed IDPs. The focus on the IDP concept varies by provider, with some covering the entire software development process as an „End2End“ development process, while others, particularly open-source solutions, may offer only parts of the IDP’s task spectrum. Currently, a comprehensive IDP that is distributed as open-source but also covers most features of a closed-source software does not seem to exist. Closed-source products are generally offered for a monthly fee, with costs dependent on factors like team size, the number of builds completed per month, or the underlying infrastructure used. Due to the predominantly closed-source approach, there is an expectation of reduced design flexibility in the setup and operation of the platform, as well as an increased dependency on the third-party provider.