How to create a Tanzu Kubernetes Cluster using vRealize Automation

Purpose of this article

In our environment we are currently using VMware Tanzu in combination with vSphere 7. As we do not use the cloud-based product Tanzu Mission Control to roll out new Tanzu Kubernetes cluster, it should be possible to automatically deploy and, in future configure, Tanzu Kubernetes cluster. Therefore I have created a Cloud Template in vRealize Automation (version 8.8), which allows users to be able to request Tanzu Kubernetes cluster out of the Service Broker catalog.

Disclaimer: In our environment we do have vRealize Automation 8.8, so it is still vRealize Automation (vRA). vRA in general was rebranded to VMware Aria Automation starting with version 8.10.

This article will describe the necessary steps, that need to be taken to integrate Tanzu into vRealize Automation, as well as create the Cloud Template in Cloud Assembly and publish the cloud Template to Service Broker to make it requestable for users.

vRealize Automation 8: Load Balancer – Overview


Introduction

With vRealize Automation, users can use existing load balancers as well as create load balancers within a deployment.

Before we show how to use existing load balancers or create new load balancers, let’s talk about the theory first.

Depending on the deployment type, different kind of load balancers are used. If you provision to a public cloud environment, vRealize Automation will create the appropriate load balancer (e.g. Elastic Load Balancer for Amazon AWS). If the deployment is for an on-premises environment and you have NSX properly set up, vRealize Automation will create NSX edges accordingly.

Features

The configuration is quite vendor-agnostic and provides the following features:

  • HTTP, HTTPS, TCP and UDP ports can be used. When using HTTP or HTTPS, the load balancer operates at the request level (Layer 7), if you go for TCP or UDP, the load balancer operates at the connection level (Layer 4).
  • In order to discover the availability of the instances, the load balancer supports health checks. This means , the load balancer periodically sends pings, attempts connections to test the virtual machines. When using HTTP for a health check, each virtual machine behind that is registered in the load balancer must respond with an HTTP status code 200. Otherwise, it will be consideres unhealthy. Health checks can also use other ports. When setting up TCP or UDP, an additional port has to be configured, to where a connection will be tried to be established. In addition, ICMP can also be used to periodically ping a machine. 
  • Health checks happen in intervals. When setting up a load balancer, you can specify the interval in seconds. In addition, you should set up an Unhealthy threshold, which is the number of continuous failed health checks before a load balancer machine is declared unhealthy. If a machine should be recognized as healthy again, a number of continuous successful health checks must be passed. This is called Healthy threshold.  You can configure the Interval in seconds as well, which is the time between two load balancer health checks. 
  • Load balancer can be internet facing or internal – a property that can be configured on the blueprint.

That’s it. Next we will show how to use load balancers on blueprints.

The image is a screenshot of a configuration page for routing and health check settings within vRealize Automation 8. The "Routes" section includes fields for "port" set to 80, "protocol" set to HTTP, "instance port" also set to 80, and "instance protocol" set to HTTP. Under the "Health Check Configuration," there are settings for "port" at 80, "Url Path" at the root ("/"), "protocol" set to HTTP, "Timeout in Seconds" set to 2, "Interval in Seconds" set to 60, "Healthy Threshold" set to 2, and "Unhealthy Threshold" set to 3. There are buttons for "CANCEL" and "APPLY" at the bottom of the configuration page.

vRealize Automation: Network Profiles – Outbound

Understanding Outbound Network Profiles in Network Configurations

The next network profile, we want to talk about is the outbound network type.

The configuration of the network profile is the same as for routed networks, so will skip the discussion here – the only difference is the configuration on the blueprint.

So what is the difference between routed and outbound?

Outbound networks limit network traffic to occur between the compute resources in the deployment but also allow one-way outbound network traffic.

Routed networks contain a routable IP space divided across available subnets that are linked together. The virtual machines that are provisioned with routed networks, and that have the same routed network profile, can communicate with each other and with an existing network.

Setting Up Outbound Networks: Configuration and Key Components

If you configure an outbound network, the following items will be provisioned:

  • A logical Switch
  • A DHCP Server
  • A T-1 gateway
  • One-to-many SNAT rule

In addition, a NAT route advertisement will be created.

Creating an outbound network from a blueprint is easy. The following snippet shows an appropriate blueprint:

formatVersion: 1
inputs:
  name:
    type: string
    title: VM Name
    description: Name of the virtual machine
resources:
  Cloud_Machine_1:
    type: Cloud.Machine
    properties:
      flavor: small
      image: Ubuntu1804
      cloudConfig: |
        #cloudconfig
        repo_update: true
        repo_upgrade: all
        package_update: true
        package_upgrade: all
        hostname: ${input.name}
        manage_etc_hosts: true
      networks:
        - network: '${resource.Cloud_NSX_Network_1.id}'
      name: Ubuntu
  Cloud_NSX_Network_1:
    type: Cloud.NSX.Network
    properties:
      networkType: outbound

vRealize Automation 8: Network Profiles – Routed

Understanding Routed Network Policies in vRealize Automation

The next type of Network Policy we want to talk about is called Routed Network Policy. If you have worked with vRealize Automation 7 before, you might notice that Routed Network Policies correspond to Routed Networks in vRA 7.

Hence you use a routed network policy when there is a need for end-to-end routable access with unique IP addresses. When deploying a blueprint with a routed network policy referenced, vRealize Automation will instruct NSX to connect the newly created network to the existing DLR and configure the DLR in such a way that the newly created network is routed to the upstream gateway.

Hence, vRealize Automation dynamically creates a network with different subnets and a routing table. Routed networks are a valid choice if there is end-to-end routable access with unique IP addresses needed.

IP Address Assignment in Routed Networks

Routed network polices support static and dynamic IP address assignment. If you prefer static IP address assignment, consider the following items to be configured.

Guest customization specification – IP addresses will be assigned using the guest customization specification. Without configuring this, IP addresses will be reserved from the pool, but they will not be applied to the provisioned machine.

VMware tools – behind the scenes, the configuration of virtual machines with new IP addresses happens with the VMware tools. Dynamic IP address work as well. In this case, there will be a DHCP server for the newly created subnet.

A routed network can be depicted as follows:

The image is a simplified diagram of a network infrastructure, showing a multi-tier architecture. There are three horizontal layers labeled "Web," "Application," and "Database." Each layer consists of two virtual machine (VM) icons, symbolizing the servers or instances in that tier. Lines connect the VMs from each layer to a central point, which then connects to a "Tier 1 LR" icon, symbolizing a Tier 1 Logical Router. Another line connects this Tier 1 LR to a "Tier 0 LR" icon, indicating a Tier 0 Logical Router, which is connected to an outward-pointing arrow, symbolizing the external network or Internet access point. The layout represents the flow of data from the database tier up through the application and web tiers to the internet.

Configuring a Routed Network Policy for Network Connectivity

In order to configure a routed network policy, there are a couple of steps to be done:

First, a network profile must be created. We have shown this in the blog post before, so will skip the instructions here.

Second, make sure, there is a network with a routable network range configured if you want to have external network connectivity (Tier0 LR).

Next, on your network profile change to the Network Polices tab and perform the following configuration:

  • Select „Create an on-demand network“
  • Choose a transport zone (A transport zone controls which hosts an NSX logical switch can reach to. It can span one or more vSphere clusters. Transport zones dictate which clusters and, therefore, which virtual machines can participate in the use of a particular network)
  • Provide a CIDR range: The network address range to be used during provisioning to create isolated subnets. The CIDR address should be large enough, for example 192.168.0.0/16, to create multiple isolated subnets in a deployment.
  • Specify the Subnet size. For example a /24 network provide 256-2=254 IP addresses.
  • Specify the Distributed Logical Router
  • Define the IP range assignment:DHCP (dynamic) – Uses the allocated CIDR to configure an IP pool on a DHCP server. All the IP addresses for this network are dynamically assigned.Static – Uses the allocated CIDR to statically allocate IP addresses. Use this option when no DHCP server is configured for this network.Static and DHCP – Using the CIDR and Subnet range settings, the DHCP server pool is configured to support half of the address space allocation using the DHCP method and half of the IP address space allocation using the Static method. Use this option when some of the machines connected to the on-demand network require assigned, static IPs and some require dynamic IPs.
  • External network
  • Edge cluster: Identifies the edge cluster or resource pool where the edge appliance is to be deployed.
  • Edge datastore: Identifies the edge datastore used to provision the edge appliance.
The image is a screenshot from vRealize Automation 8, showing the "Network Policies" configuration tab. This interface section is for defining network properties in a cloud environment. The main options are to create on-demand networks and specify network settings. The selected option is "Create an on-demand network" with a "Transport zone" field populated with 'GlobalTZ', a "CIDR" field with the value '192.168.0.0/16', a "Subnet size" dropdown menu set to '/24 (<254 IP addresses)', a "Distributed logical router" field with 'DLR1', and "IP range assignment" set to 'Static and DHCP'. Below, there's an unselected option to "Create an on-demand security group". The "Network Resources" section lists an "External network" field with a placeholder 'BundesagenturVLAN-41', and the "Compute Resources" section has an "Edge cluster" field named 'Spielwiese' and an "Edge datastore" field named 'Labs'. At the bottom, "SAVE" and "CANCEL" buttons are displayed.

Once, this has been done, we can continue with configuring a blueprint. The following code snipped shows how to setup a blueprint that uses a NSX network of type „routed“.

formatVersion: 1
inputs:
  name:
    type: string
    title: VM Name
    description: Name of the virtual machine
resources:
  Cloud_Machine_1:
    type: Cloud.Machine
    properties:
      flavor: small
      image: Ubuntu1804
      cloudConfig: |
        #cloudconfig
        repo_update: true
        repo_upgrade: all
        package_update: true
        package_upgrade: all
        hostname: ${input.name}
        manage_etc_hosts: true
      networks:
        - network: '${resource.Cloud_NSX_Network_1.id}'
      name: Ubuntu
  Cloud_NSX_Network_1:
    type: Cloud.NSX.Network
    properties:
      networkType: routed

Once provisioned the deployment can be visualized as follows:

The image is a user interface screenshot from vRealize Automation 8, showing the "Request Details" section with progression tabs for "NETWORK ALLOCATION," "MACHINE ALLOCATION," "NETWORK PROVISIONING," and "MACHINE PROVISIONING." The "NETWORK ALLOCATION" tab is active. The interface is divided into multiple sections. The first section, titled "Request: Cloud_NSX_Network_1," lists the request type as "Allocation," network type as "ROUTED," and shows no constraints. Below, a "Project: Playground" section indicates no network, storage, or extensibility constraints, associated with "Region: SC labs/SCLABS." Further down, there are network profile sections. "Profile: NP Routed" is selected for allocation, associated with the subnet "BundesagenturVLAN-41." Adjacent to it are profiles "Profile: np_vlan41" and "Profile: np41," both highlighted in red, indicating an error or warning, with the message "Network profile isolation type 'NONE' does not satisfy network type 'ROUTED'." The screenshot also displays a toggle for "Dev mode" at the top-right corner, currently turned off.

If you go to the Deployment overview, you can see the created on-demand networks as well.

The image is a snapshot from the vRealize Automation 8 interface, detailing two deployment summaries. The first deployment, named "Routed 2," has no description and is part of the "Playground" project requested by a user named "guido." It consists of "2 Resources," labeled "vra-0057" and "vra-0058," was created "18 minutes ago," has a status of "On," is assigned an IP address "192.168.1.130," and is noted to "Never expire." The second deployment, named "Routed Test," also has no description, belongs to the "Playground" project, was requested by "guido," includes "2 Resources," labeled "vra-0056" and "vra-0055," was created "a day ago," has a status of "On," has an assigned IP of "192.168.0.130," and similarly is set to "Never expire." Each summary has an "ACTIONS" dropdown menu for further options. The overall theme indicates a management console where network-related configurations and resources are monitored and managed.

vRealize Automation 8: Network Profiles – Existing Network

Introduction

The first and most easy way to use a Network Profile, is to deploy in an existing network. In that case there are no network components that needs to be created on the fly, hence the configuration is easy.

The following instructions will show how to create a Network Profile and configure it accordingly.

  1. Go to the Infrastructure > Network Profiles page and click on [+ New Network Profile]
  2. On the Summary page, provide the following information:
    • Account / region
    • Name
    • Optionally a description
    • Capabilities, which have to match constraints on a blueprint
  3. On the Networks page, add the appropriate existing networks. Remember, if you have more than more network, tags can be used as blueprint constraints in order to match to tagged networks to avoid ambiguity.
  4. On the Network Polices page, there is no need to do perform any configuration.
  5. Click on Create
The image shows a "New Network Profile" creation screen from vRealize Automation 8, with tabs for "Summary," "Networks," "Network Policies," "Load Balancers," and "Security Groups" across the top. The "Summary" tab is selected, indicating the section where basic information about the new network profile is provided. The fields include "Account / region" with an icon and text indicating 'SC labs / SCLABS', the "Name" of the profile as 'NP_Default', and a "Description" box containing the text 'This is the default network'. Below, there is a "Capabilities" section where "Capability tags" are displayed with one tag entered as 'network:standard', suggesting this tag is used to match network profile capabilities to constraint tags in service blueprints. The bottom of the screen features "CREATE" and "CANCEL" buttons, allowing the user to finalize the creation of the new network profile or abandon it.
The image displays the "New Network Profile" configuration screen in vRealize Automation 8, with the "Network Policies" tab selected. Users are provided with options for network creation, specifically for outbound and private networks. The first option, "Do not create on-demand network or on-demand security group," is selected, indicating that no automatic network or security group will be created with this profile. The other options to "Create an on-demand network" or "Create an on-demand security group" are not selected. Under the "Network Resources" section, there are search fields for "External network," "Compute Resources," "Edge cluster," and "Edge datastore," all empty and awaiting input. At the bottom of the interface are "CREATE" and "CANCEL" buttons, allowing the user to either save the new network profile or discard the changes.

Once the network profile has been created, you can go to the blueprint and add a Network element. The following yams shows the code, for an ElasticSearch Deployment (we skip the code to install ElasticSeach) that goes to a specific existing network (we use tags to identify the currect network profile):

formatVersion: 1
name: ElasticStack
version: 1
description: Installs Elastic Stack on a single machine
inputs:
  image:
    type: string
    enum:
      - ubuntu1604
      - Ubuntu1804
    description: Image/OS required
    title: Image/OS
    default: Ubuntu1804
  flavor:
    type: string
    enum:
      - small
      - medium
    description: Choose the size of the machine
    title: Flavor
    default: medium
  network:
    type: string
    title: Network
    description: Input the network tag information
    default: 'network:vlan41'
  region:
    type: string
    description: Input the region that the resources belong to
    title: Region
    default: 'region:sc-central-1'
  key:
    type: string
    title: Public Key
    description: Input Public Key outputted from the script
    default: ENTER-PUBLIC-KEY
  name:
    type: string
    title: VM Name
    description: Name of the virtual machine
resources:
  Elastic_Stack_NODE:
    type: Cloud.Machine
    properties:
      name: elastic_stack_node
      flavor: '${input.flavor}'
      image: '${input.image}'
      cloudConfig: |
        #cloudconfig
        repo_update: true
        repo_upgrade: all
        package_update: true
        package_upgrade: all
        hostname: ${input.name}
        manage_etc_hosts: true

        runcmd:
        - echo "${input.key}" >> /root/.ssh/authorized_keys
        - echo "${input.key}" >> /home/ubuntu/.ssh/authorized_keys
        - chmod 400 /root/.ssh/vmware.pub
      networks:
        - name: '${Elastic_Stack_Network.name}'
          tags: []
          network: '${resource.Elastic_Stack_Network.id}'
          securityGroups: []
          assignPublicIpAddress: false
      constraints:
        - tag: '${input.region}'
  Elastic_Stack_Network:
    type: Cloud.Network
    properties:
      name: Elastic_Stack_Network
      networkType: existing
      constraints:
        - tag: '${input.network}'

vRealize Automation 8: Network Profiles

Introduction & Summary

In short, the Network show the following information:

  • The networks and networking objects that can be used for provisioning. Those networks are discovered from the underlying Cloud Accounts and can encompass vSphere network, NSX-T/NSX-V networks or those from Cloud Providers. In order to work with them and allow DHCP from blueprint deployments, we showed how to add additional information like CIDR range, DNS or domain settings.
  • IP ranges could be used to define the set of IP addresses that can be reserved during provisioning.
  • Once those IP ranges have been setup and you configured your blueprint correctly, the IP ranges showed all the IP addresses in use.
  • Load balancers showed all the discovered load balancer
  • Network containers lists containers or groupings of related networks.

Network Profiles

The next thing we want to discuss are Network Profiles.

Basically, Network Profiles are used at runtime to determine which network constructs will be used or created at runtime. Those network constructs encompass existing network constructs (from vSphere, NSX-T, NSX-V, or any VPC, VNET stuff from cloud providers): Those network constructs can be something like networks, load balancers or security groups.

In addition Network Profiles can help to create on-demand network resources. That includes:

  • Public networks
  • Private networks
  • Outbound networks
  • Routed networks

Technically, Network Profiles provide some orchestration stuff, so that we do not need the configure the underlying network management system (for example NSX). For example, when setting up a routed network, vRealize Automation will create a logical switch, a DHCP server and will advertise the NSX routes.

However, configuration is not done with solely in the Network Profile. Some tasks has to be done on the blueprint as well. 

Compared to vRealize Automation 7, the configuration has been simplified, To achieve similar goals, the configuration of network profiles and blueprints was not sufficient in the old version. There was also work to be done on reservations. However, as there are no reservations in vRA 8 anymore, things have become easier.

Besides setting up networks, vRealize Automation can also help you with microsegmentations and firewall rules.  We will discuss everything in detail within the next blog posts.

vRealize Automation 8: Networks – Part 2

Introduction


As discussed in the last blog post, vRealize Automation allows to manage networks and their configuration settings in an intuitive and comfortable manner. Administrators have an overview over all networks imported from the underlying Cloud Accounts and are able to centrally manage stuff like CIDR Ranges, DNS settings, Gateways and so on.

Besides, administrators can use the IPAM built-in functions of vRA if they don’t have a dedicated IPAM solution like Infoblox.

Perform the following steps to manage IP ranges:

  • Within the Networks page, select the network for which you want the IP ranges to be configured and click on „MANAGE IP RANGES“
  • Click on [+ NEW IP RANGE]
  • Provide a Name for the IP Range
  • Optionally, provide a description
  • Enter the Start IP address
  • Provide the End IP address

Click on Create

IPAM Considerations

Once you have saved your new IP range, you can change to the IP Ranges tab and have an overview about all your configured IP Ranges.

However, there are some caveats in order to use the internal IP Ranges and hence you have to make sure that the following requirements are fulfilled:

  • First, you need the VMware Tools installed on the Linux machine
  • You need to setup a Guest specification in vSphere
  • You have to configure the blueprint to use the guest specification
  • You have to change the Network Assignment to Static on the blueprint

The first two items should be easy to be configured or are most likely already set up properly. The following code shows the necessary configuration on the blueprint.

formatVersion: 1
inputs: {}
resources:
  Cloud_vSphere_Machine_1:
    type: Cloud.vSphere.Machine
    properties:
      image: Ubuntu
      cpuCount: 1
      totalMemoryMB: 1024
      networks:
        - tags: []
          network: '${resource.Cloud_vSphere_Network_1.id}'
          assignment: static
          assignIPv6Address: false
          assignPublicIpAddress: false
      customizationSpec: Linux
  Cloud_vSphere_Network_1:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      name: VLAN-41

Once you have accomplished these configuration settings, you can monitor the assigned IP addresses:

Network Load Balancers

When you have NSX integrated, the Network section will also display the load balancers that can be used for provisioned resources.

However, except than setting tags (which is optional), no configuration has to be done here.

Last but not least, the Network Domain section will give you an overview over containers or groupings of related networks. Networks in the network domain are related and non-overlapping. Term equivalents:

  • AWS = VPC (virtual private cloud)
  • Microsoft Azure = Virtual network
  • vSphere = Network

vRealize Automation 8: Networks – Part 1

Network Discovery and Configuration Overview

Like in vRealize Automation 7, setting up network configuration is crucial before being able to create blueprints.

Once a Cloud Account and a Cloud Zone has been configured, you can navigate to the Infrastructure > Networks page and find an overview of networks that have been discovered. 

Streamlining Network Setup with vRealize Automation 8

In addition, vRealize Automation can also keep track about IP ranges. If you configure an IP range and provision a resource from within that range, vRealize Automation is also able to track those IP addresses. If you do not want to rely on the built-in IP adress management (IPAM), you can also integrate a third-party IPAM module. We will show how to integrate Infoblox IPAM in a later post, serving as DNS server as well as an IPAM system.

When provisioning a virtual machine, it is crucial to have network settings injected correctly. This includes the following information:

  • Domain (The domain name is passed to the vSphere machine customization spec),
  • An IPv4 CIDR
  • An IPv6 default gateway
  • DNS Servers
  • DNS Search domains
  • Support Public IPs (this can be used if you want mark the network as a public network, and match it to blueprints having the network type: public property configured).
  • Default for Zone
  • Tags

Setting these information within the network configuration is quite convenient. If your image templates work with DHCP and are using cloud-init for customization, then there is no need for the traditional guest specification anymore, as basic network settings will be injected via DHCP and all the rest via cloud-init (which is by far more powerful than the guest specification and have become a standard).