Internal Developer Platforms – Part 3

Understanding the Components of Internal Developer Platforms (IDP)

After having motivated the advantages of an IDP, we want to focus on the components of an IDP: Internal Developer Platforms (IDP) are designed to streamline and enhance the efficiency of development tasks through a variety of integrated tools and technologies. These platforms comprise six core components that work together to create a cohesive development environment. 

1. Application Configuration Management (ACM) allows you to centralize the management of application configurations. This makes for more convenient management of configurations and enhances service capabilities for such scenarios as microservices, DevOps, and big data. ACM is essential for managing application configurations in a standardized, scalable, and reliable manner. This includes handling various configurations often stored in YAML files, facilitating easier setup changes, versioning, and maintenance. ACM also relates with a proper GitOps strategy, Infrastructure Orchestration (for example with Terraform or Helm) and Environment Management

2. Infrastructure Orchestration (IO): This component integrates the IDP with existing technology setups, including CI/CD pipelines and hardware infrastructure. It ensures that images created during the CI process are correctly deployed into environments like Kubernetes clusters, potentially managed through service accounts provided by cloud operators. Typically, technologies like Terraform or Ansible are used.

3. Environment Management (EM): EM enables developers to create, update, and delete fully provisioned environments at any time without operations team intervention. This promotes efficiency and cost savings by avoiding unnecessary operational expenses and delays. Once again GitOps and a proper GitOps strategy alongside with Infrastructure Orchestration and Deployment Management is crucial.

4. Deployment Management (DM): Focused on integrating CI/CD pipelines, DM allows developers to concentrate on coding by automating building, testing, and deployment processes. This includes selecting the correct environment for image deployment and keeping a record of all deployments for auditing and debugging purposes. Tools like GitLab CI, Tekton or Argo will be a good choice for the integration into to IDP.

5. Role-Based Access Control (RBAC): RBAC controls who can access what within the IDP. By defining roles like Member, Manager, and Admin, IDPs ensure that access is appropriately managed, minimizing unauthorized access and tailoring permissions to the environment type—production or development.

6. Self-Servicing: This feature enables developers to use IDP services without relying on other teams. Self-servicing can be implemented through no-code solutions like graphical user interfaces or through more open solutions like command-line interfaces (CLIs) and APIs. It encompasses accessing templates, creating microservices, and managing software components, promoting a culture of collaboration and innovation across teams and stakeholders.

By integrating these components, Internal Developer Platforms empower developers to work more autonomously, streamline processes, and enhance collaboration across teams, leading to more efficient and innovative development practices.

Summary of Components

The following table recaps these components:

ComponentDescription
Application Configuration Management (ACM)Manages app configurations in a scalable way, simplifies setup changes, and maintains version control.
Infrastructure Orchestration (IO)Integrates with existing tech tools like Terraform and Ansible and manages the underlying infrastructure
Environment Management (EM)Allows for the creation and management of applicationenvironments.
Deployment Management (DM)Integrates with CI/CD pipelines, enabling developers to focus more on coding than on deployment logistics.
Role-Based Access Control (RBAC)Manages user permissions within the IDP, ensuring secure and appropriate access.
Self-ServicingProvides developers with tools and services for efficient and autonomous project management.

Introducing an IDP

After having discussed the components of an IDP, let’s now discuss how to introduce an IDP in an enterprise environment.

When introducing an Internal Developer Platform (IDP), there are primarily two approaches: self-development or acquisition from an external provider as a complete package. In the case of self-development, the task usually falls to the Operations Team or a designated „Platform Team“ within the company. This team is primarily responsible for creating, developing, maintaining, and optimizing the IDP while ensuring it meets organizational needs.

The Platform Team maintains close communication with the rest of the organization to identify issues at both the team and corporate levels. Their responsibilities include setting basic configurations for applications, managing permissions, ensuring standardization by design, managing infrastructure, service level agreements, optimizing workflows, and configuring the IDP to automate recurring processes.

Alternatively, the Platform as a Service (PaaS) approach involves using a platform provided by an external provider, who takes over the tasks mentioned above.

Companies with specific IDP needs and a capable Platform Team usually prefer the self-development route. Instead of building from scratch, teams can utilize various tools and services to assemble the platform, such as platform orchestrators, portals, service catalogues, Kubernetes control planes, and infrastructure control planes. They can also integrate common tools and technologies, possibly already in use internally, like PostgreSQL databases, Kubernetes for infrastructure, Harbor for image registries, and GitLab CI for continuous integration.

A typical IDP tech stack might include tools like Backstage for internal developer portals, ArgoCD for GitOps, and Terraform for infrastructure provisioning. While some of these tools, like Backstage and ArgoCD, are open-source and free to use, others may have different cost structures, including usage fees. The selection of tools involves considering cost structures, integration concepts, scale, and hosting models. Self-development offers design freedom but comes with challenges such as the increased effort, steep learning curves, and lack of official support for open-source tools, which may necessitate extensive research for technical problems.

Summary of IDP Implementations

ApproachTeam involvedKey responsibilitesBenefitsChallenges
Self-DevelopmentPlatform TeamBuilding and maintaining IDP Managing permissions Standardization and workflow optimizationCustomizability; Integration with existing toolsHigher effort and learning curve Costs for infrastructure and setup
Lack of official support
PaaS (External)External ProviderProviding and maintaining IDP Managing infrastructure and permissionsReduced internal workload Professional supportLess control and customization Potential ongoing costs

Internal Developer Platforms – Part 2

What is an Internal Developer Platform (IDP)?

After having introduced Internal Developer Platform in our first blog post of this series, we want to talk about the features of an IDP in more details and write why IDPs matters for companies within a cloud transformation.

First, the concept of an IDP basically consists of three main components:

Firstly, „Internal“ signifies that the platform is designed for use exclusively within an organization. Unlike public or open-source tools, an IDP is tailored to meet the specific requirements and security standards of the company. This focus ensures that internal workflows, data, and processes remain secure and optimized for the company’s unique ecosystem.

Secondly, the „Developer“ component highlights that the primary users of these platforms are the application developers within the organization. The platform is designed with the needs of developers in mind, aiming to streamline their workflows, improve efficiency, and enhance productivity. By centralizing tools and resources, an IDP reduces the complexity developers face, allowing them to focus more on coding and less on administrative or setup tasks.

Thirdly, „Platform“ denotes that the IDP serves as a foundational framework combining various development, deployment, and operational tools into a cohesive environment. This includes integration with version control systems, continuous integration (CI) tools, GitOps practices, databases, and container technologies. By bringing these elements together, the platform facilitates a more streamlined and coherent development lifecycle.

The main objective of an IDP is to simplify and enhance the developer experience by automating processes, centralizing resources, and eliminating unnecessary manual tasks. This includes enabling developers to request resources, initiate pre-provisioned environments, deploy applications, and manage deployments with greater ease and efficiency. As a result, the deployment process, in addition to development, becomes part of the developer’s realm, increasing control and agility.

An IDP typically requires only a coding environment, the Git tool for version control and merging, and the platform itself. This consolidation reduces the need for external websites or the creation of superfluous scripts to execute certain processes or actions, thereby streamlining the entire application development process.

Internal Developer Platforms are generally developed by a dedicated in-house team, known as the Platform Developer Team, which ensures that the platform is customized to fit the company’s needs and goals. However, for companies lacking the resources or expertise to develop their own IDP, there are Platform-as-a-Service (PaaS) options available, providing a complete, out-of-the-box solution from various vendors.

In contrast to IDPs, the term „Internal Developer Portal“ is occasionally mentioned in literature. While it can be used interchangeably with IDP in some contexts, most sources differentiate between the two. The Internal Developer Portal is typically understood as the graphical user interface (GUI) of the IDP, through which developers and sometimes automated systems interact with the platform’s tools and services. This interface simplifies the user experience, making the platform’s functionality more accessible and intuitive.

The Importance of Self-Service

The concept of self-service is a crucial aspect of Internal Developer Platforms (IDP) and significantly enhances their value and utility for developers. Self-service mechanisms within an IDP empower developers by giving them the autonomy to access resources, tools, and environments directly, without needing to wait for approval or intervention from IT operations or other departments. This approach accelerates workflows, promotes efficiency, and reduces bottlenecks in the development cycle.

In a self-service oriented IDP, developers can perform a variety of actions independently. For example, they can request and allocate computational resources, initiate pre-configured environments, deploy applications, and set up automated deployments. Additionally, they can manage scaling, monitor performance, and if necessary, roll back to previous versions of their applications without external assistance. This autonomy not only speeds up the development process but also encourages experimentation and innovation as developers can try out new ideas and solutions quickly and easily.

The self-service capability is underpinned by a user-friendly interface, typically part of the Internal Developer Portal, which simplifies complex operations and makes them accessible to developers of varying skill levels. By abstracting the underlying complexities and automating repetitive tasks, the IDP allows developers to focus more on their core activities, such as coding and problem-solving, rather than on infrastructure management.

Moreover, self-service in IDPs is often governed by predefined policies and templates to ensure that while developers have the freedom to access and deploy resources, they do so in a manner that is secure, compliant, and aligned with the company’s standards and practices. This balance between autonomy and control helps maintain the organization’s operational integrity while enabling the agility required in modern software development.

In summary, self-service is a key feature of Internal Developer Platforms that transforms the developer experience by providing direct access to tools and resources, thereby streamlining the development process, fostering independence, and enabling a more agile and responsive development cycle.

The following table summarizes these elements

FeatureDescription
Developer AutonomyDevelopers can independently access resources, tools, and environments, eliminating the need for IT operations or other departments‘ intervention.
Speed and EfficiencySelf-service capabilities accelerate workflows and reduce bottlenecks, enabling faster development cycles and promoting efficiency.  
Innovation and ExperimentationEmpowers developers to quickly try new ideas and solutions without prolonged setups or approvals, fostering a culture of innovation.           
User-Friendly InterfaceTypically part of the Internal Developer Portal, it simplifies complex operations, making them accessible and manageable for developers of all skill levels
GovernanceWhile offering freedom, self-service is governed by predefined policies and templates to ensure security, compliance, and adherence to company standards

That’s for this post. In the next post, we will talk about the internal components of an IDP.

Introduction to Internal Developer Platforms with Backstage

In recent years, the software development landscape has undergone significant changes, with terms like „Containerization,“ „Microservices,“ and „Cloud“ becoming increasingly influential. These technologies represent just the tip of the iceberg in a vast array of tools and practices shaping modern software development. The field is continuously evolving, with Artificial Intelligence tools now becoming part of developers‘ everyday toolkit.

Pain Point: Cognitive Load

As the complexity of IT infrastructure grows, so does the cognitive load on developers. They are expected to navigate an ever-expanding universe of tools and technologies, even for basic tasks such as app deployment. This scenario is challenging because developers‘ primary focus should be on coding and creating software efficiently.

Solution: Internal Developer Platform

To address these challenges, leading tech companies like Spotify, Google, Netflix, and Github have developed Internal Developer Platforms (IDPs). These platforms aim to simplify the development process by integrating existing infrastructures and tools, thereby facilitating information exchange throughout the software development lifecycle. IDPs serve to abstract away the underlying complexity, allowing developers to focus more on their primary tasks and less on the intricate details of the technology stack.

The rising interest in such platforms suggests a shift towards more streamlined, efficient development environments. As this trend continues, delving deeper into the technology and understanding its implications for software development could prove beneficial. The ongoing evolution underscores the importance of adapting to new tools and practices in the ever-changing landscape of software development.

Basic Features

Increasingly, cloud providers are offering tools that extend beyond the standard Integrated Development Environment (IDE) to assist developers with ancillary tasks such as management, deployment, and observability. These tools are part of internal platforms, setting them apart from public services like Github, which focus on code management. Recent examples of Internal Developer Platforms (IDPs) include Atlassian and Gitlab. IDPs serve as central hubs, often featuring a „cockpit“ that provides an overview of resources and statuses. Some now incorporate AI to help with routine tasks. The aim of these platforms is to enhance the Developer Experience and reduce cognitive load.

Solutions on the market

More and more commercial vendors provide an IDP based on Spotify Backstage. 

For example, at RedHat,  the core of the Developer Hub announced in May 2023 a self-service portal based on Backstage, where developers can find all the necessary resources for their work, including a software catalog, documentation, and best practices. The portal also offers standardized software templates and pipelines, aiming to simplify the often cumbersome deployment process. For example, Red Hat aims to reduce the cognitive load on teams and make onboarding easier for new members.

Adopting the plugin concept from the underlying Backstage project, vendors have released its own plugins, which are available to customers and the Backstage community. These plugins enhance the portal with features for Continuous Integration/Continuous Delivery (CI/CD), Observability, or Container Management, and the IDP is capable of running non-Red-Hat plugins as well.

Part 10: Supply Chain – Configuring Tekton Chains and Tekton Dashboard

Introduction

This is the last blog for our Tekton tutorial. In the last blog posts, we discussed how to install and configure Tekton. Now we want to briefly discuss how to configure Tekton Chains and the Tekton dashboard.

Tekton Chains

As discussed, an important requirement of the SLSA framework is attestation. Tekton provides Tekton Chains for that, which is part of the Tekton ecosystem.

In addition to generating attestation, Tekton Chains makes it possible to sign task run results with, for example, x.509 certificates or a KMS and store the signature in one of numerous storage backends. Normally the attestation with the artifact is stored within the OCI store. It is also possible to define a dedicated storage location independent of the artifact. Alternative storage options include Sigstor’s record servers, document stores such as Firestore, DynamoDB and Mongo and Grafeas.

An important aspect of Tekton Chains is the ability to integrate the Sigstor project. This also refers to Fulcio and Rekor, which were explained in more detail here. SLSA’s requirements for provenance, (which can be solved by Rekor and Fulcio), were that keys must be stored within a secure location and that there is no possibility for the tenant to subsequently change the attestation in any way . Although key management via a KMS is just as valid as using Fulcio and both solutions would meet the requirements of SLSA, Rekor, in particular, satisfies the requirement of immutability. As already mentioned, Rekor’s core is based on Merkle trees, which make deletion or modification impossible. Both Fulcio and Rekor represent an important trust connection between producer and consumer through the services provided by Sigstore.

Tekton Chains offers the advantage that neither the signature nor the attestation need to be provided through custom steps in the pipeline itself. For example, even if a developer integrates Cosign into a task to sign an image, Chains works regardless. The only requirement in the pipeline are the so-called ‘Results’. These allow Tekton to clearly communicate which artifacts should be signed. Results have two areas of application. A result can be passed through the pipeline into the parameters of another task or into the when functionality, or the data of a result can be output.

The data output by results serves the user as a source of information, for example about which digest a built container image has or which commit SHA a cloned repository has. The Tekton Chains Controller uses Results to determine which artifacts should be attested. The controller searches for results of the individual tasks with the ending “*_IMAGE_URL” and “*IMAGE_DIGEST”, where the IMAGE URL is the URL to the artifact, IMAGE_DIGEST is the digest of the artifact and the asterisk is any name.

Tekton Dashboard

The Tekton dashboard is a powerful tool that makes managing Tekton easier. The dashboard can be used in two modes: read only mode or read/write mode.

The authorizations of the dashboard can be configured by means service accounts, as is typical for Kubernetes. However, this poses a problem because the dashboard itself does not come with authentication or authorization in either mode. There are no options for regulating the dashboard’s permissions through RBAC. In this case, RBAC would only apply to the dashboard’s ServiceAccount, but not to all users. In practice, this means that all authorizations that the Tekton Dashboard service account has also apply to the person accessing the dashboard. This is a big problem, especially if the dashboard is public accessible.

Kubernetes does not have native management of users because – unlike service accounts – they are not manageable objects of the API server. For example, it is not possible to regulate authentication via user name and password. However, there are several authentication methods that use certificates, bearer tokens or authentication proxy.

Two of these methods can be used to secure the Tekton dashboard. On the one hand, OIDC tokens and on the other hand, Kubernetes user impersonation.

OIDC is an extension of the Open Authorization 2.0 (OAuth2) framework. The OAuth2 framework is an authorization framework that allows an application to carry out actions or gain access to data on behalf of a user without having to use credentials. OIDC extends the functionality of OAuth 2.0 by adding standardizations for user authentication and provision of user information.

Kubernetes user impersonation allows a user to impersonate another user. This gives the user all the rights of the user they are posing as. Kubernetes achieves this through impersonation headers. The user information of the actual user is overwritten with the user information of another user when a request is made to the Kubernetes API server before authentication occurs.

There are different tools to achieve this. One of this tools is Open Unison from Tremolo. Open Unison offers some advantages. It is possible to implement single sign-on (SSO) for graphical user interfaces and session-based access to Kubernetes via the command line interface. When using Openunison or similar technologies, communication no longer takes place directly with the Kubernetes API server, but rather runs via Open Unison. Open Unison uses Jetstack’s reverse proxy for OIDC.

When a user wants to access the Tekton dashboard, Open Unison redirects the user to the configured Identity Provider (IDP). After the user has authenticated himself with the IDP, he receives an id_token. The id_token contains information about the authenticated user such as name, email, group membership or the token expiration time. The id_token is a JavaScript Object Notation Web Token (JWT)

The reverse proxy uses the IDP’s public key to get the id_to to validate. After successful validation, the reverse proxy appends the Impersonation header to the request to the Kubernetes API server. The Kubernetes API server checks the Impersonation header to see whether the impersonated user has the appropriate permissions to execute the request. If so, the Kubernetes API server executes the request as an impersonated user. The reverse proxy then forwards the response it received from the Kubernetes API server to the user.

The following steps describe the configuration of the dashboard with OAuth2:

Create a namespace:

kubectl create ns consumerrbac

Installation of Cert Manager:

helm install \                                                                                                                   

  cert-manager jetstack/cert-manager \

  –namespace consumerrbac \

  –version v1.11.0 \

   –set installCRDs=true

In order to create certificates, an issuer is needed:

apiVersion: cert-manager.io/v1

kind: Issuer

metadata:

  name: letsencrypt-prod

  namespace: consumerrbac

spec:

  acme:

    server: https://acme-v02.api.letsencrypt.org/directory

    email:

    privateKeySecretRef:

      name: letsencrypt-prod

    solvers:

    – http01:

        ingress:

          class:  nginx

Now nginx, can be installed:

helm install nginx-ingress ingress-nginx/ingress-nginx –namespace consumerrbac

Now, the ingress can be created.

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: tekton-dashboard

  namespace: tekton-pipelines

  annotations:

    kubernetes.io/ingress.class: “nginx”

    nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.consumerrbac.svc.cluster.local/oauth2/auth

    nginx.ingress.kubernetes.io/auth-signin: https://dashboard.35.198.151.194.nip.io/oauth2/sign_in?rd=httpd://$host$request_uri

    nginx.ingress.kubernetes.io/ssl-redirect: “false”

spec:

  rules:

  – host: dashboard.35.198.151.194.nip.io

    http:

      paths:

      – pathType: Prefix

        path: /

        backend:

          service:

            name: tekton-dashboard

            port:

              number: 9097

OAuth Proxy (we use Google and exptect the application created there):

  • In the Google Cloud dashboard, select APIs & Services
  • On the left, select Credentials
  • Press CREATE CREDENTIALS and select O Auth client ID
  • For Application Type, select Web application
  • Give the app a name and enter Authorized JavaScript origins and Authorized redirect URIs
  • Click Create and remember Client ID and Client Secret
  • A values.yaml must be created for the installation.
config:

  clientID:

  clientSecret:

 

extraArgs:

  provider: google

  whitelist-domain: .35.198.151.194.nip.io

  cookie-domain: .35.198.151.194.nip.io

  redirect-url: https://dashboard.35.198.151.194.nip.io/oauth2/callback

  cookie-secure: ‘false’

  cookie-refresh: 1h

Configuration of Open Unison

Create a namespace:

kubectl create ns openunison

Add Helm repo:

helm repo add tremolo https://nexus.tremolo.io/repository/helm/

helm repo update

Before openunison can be deployed, the oauth must be configured in the Google Cloud.

  • In Credentials, under Apis & Services, click CREATE CREDENTIALS
  • Then on OAuth Client ID
  • Select Web application as the application type and then give it a name
  • Authorized JavaScript origins: https://k8sou.apps.x.x.x.x.nip.io

Now Open Unison can be installed:

helm install openunison tremolo/openunison-operator –namespace openunison

Finally, Open Unison has to be configured with the appropriate settings.

This concludes our series on Tekton. Hope you enjoyed it.

Part 9: Supply Chain – Workspaces and Secrets

Introduction

As mentioned in the last blog post, the next thing we want to talk about and discuss is authentication, workspaces and secrets. Let’s begin with Workspaces.

Workspaces

As already mentioned, in Tekton each task is run in a pod. The concept of workspaces exists in Tekton so that pods can share data between each oterh. Workspaces can also help with other thins: Workspaces can be used to mount secrets, config maps, tools or a build cache in a pod. Tekton Workspaces work similarly to Kubernetes Volumes. This also applies to their configuration.

The configuration of the workspace is done in the pipeline, task run or in a TriggerTemplate.

Configuring a workspace is very similar to configuring a Kubernetes volume. This example creates a workspace that is used to mount the Dockerfile and associated resources from the pod that clones the repository to the pod that builds and uploads the image. In Tekton, VolumeClaimTemplates are used to create a PersistentVolumeClaim and its associated volume when executing a Task or PipelineRun. (Tekton Workspaces, n.d.) The further configuration of the workspaces is similar to that of a PersistentVolumeClaim in Kubernetes. The accessMode specifies how and which pods have access to a volume. ReadWriteOnce means that pods on the same node have read and write access to the volume. The storage space size in this configuration is one Gigabyte.

Of course, the steps to clone the repository and build and upload the container image to a registry require appropriate permissions. This can be done via the two following options:

  • First, the corresponding Kubernetes secrets with the credentials are mounted in the pod.
  • Second, authentication is implemented via a Kubernetes Service Account. The mounted volume is a Kubernetes Secret Volume. The data in this volume is read-only and is managed in the container’s memory via the temporary file system (tmpfs) file system, making the volume volatile. Secrets can be specified under Workspaces in the yaml configuration as follows.

Tekton can also isolate workspaces. This helps to make data accessible only for certain steps in a task or sidecars. However, this option is still an alpha feature and therefore cannot (yet) be used.

Secret Management

Kubernetes secrets are not encrypted by default, only encoded. This means that anyone with appropriate permissions can access the secrets via the cluster or the etcd store. It should also be noted that anyone who has the rights to create a pod has read access to the secrets in the corresponding namespace.

Kubernetes offers two ways to deal with this problem. Option one is the encrypting of secrets in the etcd store. This means that the secrets are still kept within Kubernetes.

Option two involves the utilization of third-party applications and the Container Storage Interface (CSI) driver. In this case, secrets are not managed directly by Kubernetes and are therefore not on the cluster.

One popular tool for the second approach is Hashicorp Vault. Like the other tools, Vault follows the just-in-time access approach. This gives a system access to a secret for a specific time and as needed. This approach reduces the blast radius by compromising the build system.

In addition, this minimizes the configuration effort because extra Role Based Access Control (RBAC) rules for secrets, for example in the namespaces for development, test and production, do not have to be created and the secrets do not have to be stored in this extra.

The Secrets Store CSI Driver makes it possible to mount secrets from Vault, into the CSI. Once the CSI driver knows which secrets should be mounted from the provider, SecretProviderClass objects are configured. These represent custom resources under Kubernetes. When a pod is started, the driver communicates with the provider to obtain the information specified in the SecretProviderClass.

Authentication

In the following two use cases Tekton needs secrets for authentication:

  • Authentication against Git (for example cloning)
  • Authentication against the Container Registry

As described in the last blog post within the PipelineRun example, Secrets can be mounted. The following examples show how to create those secrets:

Both manifest files can be created via the kubectl-command.

If a config.json file does not yet exist, you have to generate it first. To do this, you must log in to the desired registry via docker.

docker login registry.gitlab.com

Within the config.json the credentials from the Docker config.json must be specified in base64 encoded.

cat ~/.docker/config.json | base64

It is important to ensure that this does not happen via Docker Desktop, because then the “credsStore”: “desktop” field is included in the config.json and it must be ensured that the config.json has the following format:

{

        “auths”: {

                “registry.gitlab.com”: {

                        “auth”: “”

                }

        }

}

Furthermore, the secrets can be added to the ServiceAccount, which is specified via the serviceAccountName field.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-bot
  namespace: tekton
secrets:
  - name: git-credentials
  - name: dockerconfig

If the credentials are not provided via the ServiceAccount, they must be defined in the pipeline run under the pod template.

podTemplate:
    securityContext:
      fsGroup: 65532
    imagePullSecrets:
    - name: dockerconfig
    - name: git-credentials

After the pipelinerun.yaml has been configured it can be executed

kubectl create -f pipelinerun.yaml

Pipeline run logs can be viewed using the tkn command line tool:

tkn pr logs clone-read-run- -f -n tekton

After the pipeline has run through, you can check whether it has been signed and attested.

kubectl get tr [TASKRUN_NAME] -o json | jq -r .metadata.annotations

Part 8: Supply Chain – Tasks and Pipelines

Pipelines

Now it is time to gain a better understanding of tasks and pipelines. Before we create a pipeline, lets first create a Tekton namespace:

kubectl create ns tekton

In Tekton, a pipeline can consist of one or more tasks, which can be executed one after the other or in parallel with one another.

The pipeline includes the fetch-source, show-content and build-push tasks. Fetch-source clones the repo in which the Dockerfile is located and build-push builds the image and uploads it to a repo. The show-content task displays the artifacts obtained through Fetch-source.

One of biggest of advantages of Tekton is, that not all tasks and pipelines need to be rewritten because Tekton provides a Tekton Hub where users can share their tasks and pipelines with each other. Two tasks from the Tekton Hub were used in this example.

The first task, called Git-clone, clones a Git repository and saves the data to a workspace. Workspaces will be discussed in more detail later.
The second task, originating from the Tekton hub, builds an image and uploads it to any container registry. The task uses Kaniko to build an image. The task also saves the name and a digest of the image in a result so that Tekton Chains can sign the image and create an attestation. Tekton chains and results will be discussed in a later point.

The example of the “git-clone” task shows the name of the task has in the context of the pipeline. The “taskRef” field is used to reference the individual tasks, in this case git-clone. You can also define here which parameters and workspaces should be passed to the task.
The “url” parameter of the task is assigned the “repo-url” parameter. The names of the parameters can differ from pipeline to task. The notation $(params.repo-url) refers to the parameter that is in the “params” field. Parameters that come from a task or pipeline run are set in this field.

In order to use those tasks, we may not forget to install them. Here is the first task:

It can be applied either via tkn oder kubectl apply:

tkn hub install task git-clone

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.6/git-clone.yaml

The second task can also be found on Tekton Hub and can be installed as follows:

tkn hub install task kaniko

kubectl apply -fhttps://raw.githubusercontent.com/tektoncd/catalog/main/task/kaniko/0.6/kaniko.yaml

If you have some troubles with the builder image, you might change the appropriate section as follows:

The last task is the show-readme.

To apply the pipeline, we just enter the following command:

kubectl apply -f pipeline.yaml

Installation of Tekton Chains

Since Chains is not installed via the operator, Chains must be installed separately.

kubectl apply –filename https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml

If you prefer to install a special version, you can issue the following command:

kubectl apply -f https://storage.googleapis.com/tekton-releases/chains/previous/${VERSION}/release.yaml

After installation, the configmap can be configured.

Configuration in the manifest:

kubectl patch configmap chains-config -n tekton-chains -p=‘{“data”:{“artifacts.taskrun.format”:”in-toto”, “artifacts.pipelinerun.storage”: “tekton, oci”, “artifacts.taskrun.storage”: “tekton, oci”}}’

Furthermore, the keyless signing mode can be activated, which uses Fulcio from the Sigstore project.

kubectl patch configmap chains-config -n tekton-chains -p=‘{“data”:{“signers.x509.fulcio.enabled”: “true”}}’

Chains supports automatic binary uploads to a transparency log and uses Rekor by default. If activated, all signatures and attestations are logged.

kubectl patch configmap chains-config -n tekton-chains -p=‘{“data”:{“transparency.enabled”: “true”}}’

After the ConfigMap has been patched, it is recommended to delete the Chains Pod so that the changes are registered by the Pod

kubectl delete po -n tekton-chains -l app=tekton-chains-controller


Pipeline Runs

A run, whether TaskRun or PipelineRun, is instantiated and executes pipelines and tasks. When a PipelineRun is executed, TaskRuns are automatically created for the individual tasks. Among other things, the pipeline that is to be executed is referenced, the parameters that are to be used in a task are defined, or the pod templates. A blueprint for the executed pods is created using the templates. For example, environment variables can be provided for each pod and scheduling settings can be configured via nodeSelectors, tolerances and affinities.

After everything has been configured, the pipeline run can be executed.

Within the param section, you can specify the git repo for cloning and also choose where the image should be uploaded to.

So that’s it for today. In the next blog post, we will discuss authentication, workspaces, secrets and more. So there is still a lot of interesting stuff left.

Part 7: Supply Chain – How to work with Tekton

How to work with Tekton

In the last blog post, we briefly discussed what is Tekton and how an installation can take place. In this blog post, we go a step further and show how to work with Tekton when we want to build a Supply Chain.

Let’s start with the installation first:

The components required for Tekton are installed via the Tekton Operator. This can be done in 3 different ways: 

  • Via the Operator Lifecycle Manager
  • With a release file
  • By code

We chose the installation via the release file. However, there is one disadvantage – the lifecycle management has to be taken over by yourself:

kubectl apply -f https://storage.googleapis.com/tekton-releases/operator/previous/v0.66.0/release.yaml

Now the Tekton CRDs have been installed Later on, we will show how to configure Tekton via a TektonConfig file – but let’s discuss some theory first.

As already mentioned, the provenance must be immutable in order to reach build level three. This assumes that the user-created build steps have no ability to inject code into the source or modify the content in any way that is not intended. Therefore,  we have to take care of the Tekton pipeline. In detail this means:

  • Artifacts must be managed in a version management system.
  • When working and changing artifacts, the identities of the actors must be clear. The identity of the person who made the changes and uploaded the changes to the system and those of the person who approved the changes must be identifiable.
  • All actors involved must be verified using two-factor verification or similar authentication mechanisms.
  • Every change must be reviewed by another person before, for example, a branch can be merged into git.
  • All changes to an artifact must be traceable through a change history. This includes the identities of the people involved, the time ID of the change, a review, a description and justification for the change, the content of the change and the higher-level revisions.
  • Furthermore, the version and change history must be stored permanently and deletion must be impossible unless there is a clear and transparent policy for deletion, for example based on legal or political requirements. In addition, it should not be possible to change the history.

In order to ensure that the security chain is not interrupted, Tekton provides the option of resolvers.

Basically, a Tekton resolver is a component within the Tekton Pipelines. In the context of Tekton, a resolver is responsible for handling “references” to external sources. These external sources can be anything from Git repositories to OCI images, among others.

Tekton uses resolvers to deploy Tekton resources as tasks and pipelines from remote sources. Hence, Tekton provides resolvers to access resources in Git repositories or OCI registries, for example. Resolvers can be used for both public and private repositories.

The configuration of the provider can be divided into two parts: 

  • The first part of the configuration can be found in a ConfigMap. Tekton uses  the ConfigMap to store, among other things, standard values such as the default URL to a repository or the default Git revision, fetch timeout and API token.
  • The second part is in the PipelineRun and TaskRun definition. Within the PipelineRun and TaskRun definition, the repository URL, the revision and the path to the pipeline or task are defined under the spec field.

The following snippet shows a sample config:

<script src=https://gist.github.com/gsoeldner-sc/762d463a3b10faa752e6520e0213f6bf.js></script>

The TektonConfig can easily be deployed:

kubectl apply -f tekton-config.yaml

One disadvantage of using resolvers is that the git-resolver-configmap configuration applies to the entire cluster. Only one API token can be specified within the configuration. This means that every user of a resolver has access to the same repositories, which would make multi-tenancy impossible.

Another disadvantage is that resolvers cannot exclude the possibility that resources  not coming from a version control system may be used. To ensure that the resolvers are not bypassed, there is an option to sign resources. Policies can then check whether a resource has a valid signature. This ensures that only resources with the correct signature can be executed.

You can use the Tekton CLI tool to sign the resources.

The CLI supports signing key files with the format Elliptic Curve Digital Signature Algorithm (ecdsa), Edwards-curve 25519 (Ed25519) and Rivest-Shamir-Adleman (RSA) or a KMS including Google Cloud Platform (GCP), Amazon Web Services (AWS), Vault and Azure.

The verification of the signatures is done via policies. Filters and keys can be defined in a policy. The filters are used to define the repositories from where the pipelines and tasks can come from. Keys are used to verify the signature.

When evaluating whether one or more policies are applicable, the filters check whether the source URL matches one of the specified filters. If one or more filters apply, the corresponding guidelines are used for further review. If multiple filters apply, the resource must pass validation by all policies. The filters are specified according to the regular-expression (regex) scheme.

After filtering, the signature verification of the resources is carried out using keys. These keys can be specified in three different ways:

  • As a Kubernetes secret
  • As an encoded string,
  • Or via a KMS system 

The policies have three operating modes: ignore, warn, and fail. In “ignore” mode, a mismatch with the policy is ignored and the Tekton resource is still executed. In “warn” mode, if a mismatch occurs, a warning is generated in the logs, but the run continues to execute. In “fail” mode, the run will not start if no suitable policy is found or the resource does not pass a check.

That’s it for today. In the next part, we will talk about Pipelines and Tasks.

Part 6: Supply Chain – Introduction to Tekton

What is Tekton?

Tekton is an open source CI/CD tool and was initially a project of the Continuous Delivery Foundation. Kubernetes is used as the underlying platform for Tekton and can be extended by using Custom Resource Definitions (CRD). They act as an extension of the Kubernetes Application Programming Interface (API). This way, Tekton integrates into Kubernetes and can be used like any other Kubernetes resources.

The smallest unit in the Tekton workflow is a step and each step is executed in its own container. Several steps taken together are called a task. Each task runs in a Kubernetes Pod. This means, that the containers in the pod share the same resources, such as a volume. If there is a need to run steps in multiple pods or due to advanced automation requirements, more tasks can be run in pipelines. Pipelines are formed from several tasks. In addition, the individual components are reusable. Steps can be integrated into different tasks and tasks into different pipelines.

The concept of steps, tasks and pipelines is depicted in the following picture.

The actual execution of tasks and pipelines is managed by two resources: TaskRun and PipelineRun. The TaskRun is responsible for managing the execution of tasks, while the PipelineRun monitors and controls the execution of pipelines. Furthermore, the run resources determine when tasks and pipelines should be executed. These can be triggered at a specific time or by an event. Runs represent the link between resources and tasks and pipelines to ensure the modular structure. The parameters required for the steps are passed to the pipelines and tasks through the runs.

Installation

Like with many other Kubernetes application, Kubernetes operators prove to be indispensable for Tekton and help to ensure that the actual state of the Tekton application matches the desired state. Through their specialized knowledge, operators extend the functions of the Kubernetes API to ensure the automation of applications.

The Tekton Operator is responsible for installing Tekton and managing the individual Tekton resources. Other Tekton resources include the Tekton pipeline and trigger resources, which are managed via the operator. The resources are managed via Custom Resource TektonConfig. The scope of manageable resources depends on the platform on which Tekton is deployed. For example, on Kubernetes it is possible to manage the Tekton dashboard via TektonConfig, but not on OpenShift and vice versa, TektonAddons can be managed on OpenShift but not on Kubernetes. This can be seen in profiles.

There are three profiles to choose from in the Tekton operator version Operator-v0.67.0:

  • The “All” profile installs all resources available to the platform. As already mentioned, the dashboard is deployed for Kubernetes in addition to the resources and the TektonAddon is deployed on OpenShift.
  • The “Basic” profile installs the components TektonPipeline, TektonTrigger and Tekton-Chain.
  • The third profile “lite” only installs the TektonPipelines resource.

Part 5: Supply Chain – SLSA Attestation

Introduction

After having explained the basics of the SLSA framework, we want to give some insights into SLSA attestations now.

What is an attestation? An attestation is a way to generate authenticated metadata about an artifact. This makes it possible for a consumer of software to find out how it was built, who built it and which build system it was built with. Attestations also provide the option of having metadata verified by a policy engine, such as in-toto or binary authorization. For this purpose, the SLSA framework offers the slsa-verifier, which verifies the SLSA provenance format.


General Model

The SLSA framework defines a general model according to which an attestation should be built.

The model defined by the SLSA framework consists of four main components, namely the bundle, the envelope, the statement and the predicate. The outer-right component of the model is the bundle, which describes a bundle of several attestations. Different attestations at different points in the software supply chain can be bundled in one place. These include, for example, vulnerability scans, the build process and the artifact. Combining the attestations in a bundle makes it easier for the software consumer to make well-founded decisions.

The actual attestation of an artifact is located within the envelope, which in turn consists of two components: The signature and the message. The signature contains information about the issuer of the attestation and the message contains further information. The third component is the statement. The statement binds the attestation to a specific artifact and must contain at least one subject and one predicate. The subject indicates which artifacts the predicate applies to. The Predicate is the innermost part of an attestation and contains information about the subject. The SLSA framework does not contain precise definitions regarding the scope of information that can be included in an attestation. However, links are suggested. The links serve to represent a hypergraph. In this hypergraph, the artifacts are represented as nodes and the attestations as hyperedges. The links are intended to enable predicate-agnostic processing of the graph.

Furthermore, the SLSA framework defines that there should be a central storage in which the attestations are stored and retrieved and can be viewed by a consumer.

Provenance

The Provenance Predicate provides verifiable information about a software artifact and precisely explains the place of origin, the date of origin and the creation process of the artifact. This guarantee gives the user of a software the confidence that the artifact meets expectations and can be reproduced independently if necessary.

Verification Summary Attestation (VSA)

The VSA describes at which SLSA level an artifact or group of artifacts was verified and other details about the verification process, including the SLSA level at which the dependencies were verified. This makes it easier to decide whether to use an artifact. This means that a consumer does not have to include all attestations of an artifact and the attestations of their dependencies in the decision-making process. It also allows software producers to keep details of their build process confidential while still being able to communicate that certain verifications have been carried out. The model is based on the existence of a verifier, which has a position of trust with the consumer. The verifier verifies the artifact and the associated attestations against a policy and then generates the VSA.

Part 4: Supply Chain – SLSA Level & Tracks

Introduction

In the last blog post, we talked about the SLSA terminology. Now it is time to focus on SLSA level and tracks.

Within the SLSA framework, there are levels and tracks. Depending on them, it possible to incrementally harden and improve different areas of the supply chain. Tracks focus on individual sub-areas of the supply chain, for example on the build or provenance area. This helps to do the hardening independently of individual areas. The levels within the individual tracks indicate the quality of the hardening within the are. Each level defines requirements in order to reach the corresponding level. In practice, it is possible to be at level 3 in the build track, whereas only level 1 was reached in the source track. The levels are designed in such a way that the implementation costs increase with each level, so level one is basically easy to implement, whereas level 3 is difficult to achieve.

Currently, levels 0 to 3 and the build track are available. The lowest level 0 means that no requirements of the framework have been implemented. As already mentioned, each track defines different requirements for each level in order to be able to fulfill them. In the following we will focus on the build track.

The build track focuses on the provenance and validability of artifacts. It’s about defining how an artifact was built, who built the artifact, what process built the artifact, and what the artifact contains. This gives a consumer of software the certainty of whether the artifact they want to use is what it claims to be.

Levels


  • Level 1 focuses on documentation and error prevention. In this case, documentation means that the build process is documented by provenance data. This makes it easier to trace errors, for example if an incorrect source version was used. The SLSA framework specifies certain requirements for the build process and provenance data. The build process must be consistent. In addition, the provenance data, which contains information how the artifact was created, must be generated by the build platform. This data includes information about who created the artifact, what build process was used, and what dependencies and parameters were used in the build. It is also important that the provenance data is accessible to consumers of the artifact. This ensures transparency and trustworthiness because the origin and construction of the artifact are traceable. Since Level 1 is considered the start of the build track, it is easy for any company or development team to implement. Little or no changes to the workflow are required. The provenance data does not have to be complete or signed. At this level it is still possible to manipulate the provenance data.
  • Build Level 2 focuses on ensuring that an artifact can no longer be manipulated after the build. Unlike Level 1, the software build at Level 2 must be carried out on a hosted build platform. This means that it cannot be carried out on a developer’s laptop, for example. In order to exclude manipulation of the artifact after the build, the provenance data is generated and signed by the build platform itself. The signature prevents manipulation and reduces the attack surface.
  • The third level focuses on the build process itself to prevent manipulation during the build. This is mainly about hardening the build platform. The build platform must be architected in such a way that builds cannot influence each other. This includes, for example, that the key used to sign the provenance data is not accessible to the build scripts. Level 3 is the most complex to implement of the three levels, as entire workflows and system structures sometimes have to be restructured. However, risks are maximally reduced during the build, ensuring confidentiality, integrity and availability.

Tracks

After having described the importance of the individual levels, we will discuss the technical requirements that must be met in order to reach the corresponding level.

The Build Track divides the requirements into two parts. The first part of the requirements must be fulfilled by the producer and the second part contains requirements that are required to be fulfilled by the build system itself.

The producer can be anyone who is responsible for creating it and passing it on to third parties, for example entire organizations or a team of developers. The producer has three tasks. He is responsible for selecting the right build platform, defining the build process and distributing the provenance data.

To choose the right build platform, it is crucial that the build platform is capable of achieving the desired SLSA level.

SLSA version 1 requires the build process to be consistent. This makes it possible to provide the verifier with a clear understanding of how an artifact was created. Maintaining consistency in the build process makes it easier for others to assess the quality, reliability, and security of the artifact. Consistency in the build process creates expectations about how an artifact was built and how to ensure it meets the necessary standards. If a package is distributed through a package ecosystem that requires explicit metadata about the build process in the form of a configuration file, the producer must complete the configuration file and keep it up to date. This metadata can contain information about the artifact’s source code repository and build parameters.

The requirements for the build platform consist of two major requirementsments: the generation of provenance data and isolation.

Regarding provenance generation, SLSA defines 3 points, each of which correlates with the corresponding SLSA levels. To reach level 1, it is sufficient if this data exists, for level 2 the data must be authentic and for level 3 it must be unchangeable.

Level 1 defines that at the end of a build process there is provenance data about the build process and that this must be able to be uniquely identified by a cryptographic digest. The recommended format for provenance data should be SLSA provenance. Other formats can be used, but the other formats must have the same amount of information and should be bidirectionally translatable into SLSA Provenance.

To reach level 2, the provenance data must be generated by the build service’s control plain; unlike Level 1, where it didn’t matter who or what created the provenance data. In this case, however, there are two exceptions. The Subject field and fields marked as not necessary for Level 2 may be generated by a tenant of the build system. Provenance data must be authentic and the authenticity must be validatable. This is achieved by signing the provenance data. The provenance data is signed by a private key. The key may only be accessible to the build platform and not to the tenant.

At level 3 there are three requirements for the build system. The first requirement describes that cryptographic data such as the private key must be stored in a secure environment. For example, a Key Management System (KMS) must be used. The second requirement requires that the cryptographic data be isolated from the custom build steps. The third requirement describes that all fields in the provenance data were generated by the build system itself and that the custom build steps do not change the contents of this data.

SLSA’s requirement for the builds is that they run in isolation from each other so that there is no influence between the builds. Likewise, the build environment must be isolated from the control plane of the build system. Regarding isolation, SLSA has two requirements. For one thing, all steps of a build must take place on a hosted build platform and not, for example, on a developer’s laptop or computer. The implementation of the step is part of SLSA Level 2. On the other hand, all steps of a build must run in isolation from each other, so that there can be no external influences on the steps other than those previously defined as external parameters, and these must be included as such Provenance data must be recognizable. Isolation also includes the fact that one build cannot access cryptographic data of another build, such as signature keys. It must not be possible for two builds running in parallel to change the memory of another process. A short-lived environment must be provided for each build and when using a cache during the build, it must not be possible for a build to change it. Each run of a build must always produce the same result, whether a cache is used or not.