This section explains how to configure Rancher with vSphere credentials, provision nodes in vSphere, and set up Kubernetes clusters on those nodes.

Prerequisites

This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters.

The node templates are documented and tested with the vSphere Web Services API version 6.5.

Create Credentials in vSphere

Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials.

Refer to this how-to guide for instructions on how to create a user in vSphere with the required permissions. These steps result in a username and password that you will need to provide to Rancher, which allows Rancher to provision resources in vSphere.

Network Permissions

There needs to be two-way communication between Rancher and the vSphere API.

You must ensure that the hosts running Rancher servers are able to establish network connections to the following network endpoints:

  • vCenter server (usually port 443/TCP)
  • Every ESXi host that is part of the datacenter to be used to provision virtual machines for your clusters (port 443/TCP).

By default, Rancher uses port 443 to communicate with vSphere.

The vSphere API websocket port will be 84453 by default.

Valid ESXi License for vSphere API Access

The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license.

Creating Clusters in vSphere with Rancher

This section describes how to set up vSphere credentials, node templates, and vSphere clusters using the Rancher UI.

You will need to do the following:

  1. Create a node template using vSphere credentials
  2. Create a Kubernetes cluster using the node template
  3. Optional: Provision storage

Configuration References

For details on configuring the node template, refer to the node template configuration reference.

Rancher uses the RKE library to provision Kubernetes clusters. For details on configuring clusters in vSphere, refer to the cluster configuration reference in the RKE documentation.

Note that the vSphere cloud provider must be enabled to allow dynamic provisioning of volumes.

1. Create a Node Template Using vSphere Credentials

To create a cluster, you need to create at least one vSphere node template that specifies how VMs are created in vSphere.

After you create a node template, it is saved, and you can re-use it whenever you create additional vSphere clusters.

To create a node template,

  1. Log in with an administrator account to the Rancher UI.

  2. From the user settings menu, select Node Templates.

  3. Click Add Template and then click on the vSphere icon.

Then, configure your template:

A. Configure the vSphere Credential

The steps for configuring your vSphere credentials for the cluster are different depending on your version of Rancher.

    Your account access information is in a cloud credential. Cloud credentials are stored as Kubernetes secrets.

    You can use an existing cloud credential or create a new one. To create a new cloud credential,

    1. Click Add New.
    2. In the Name field, enter a name for your vSphere credentials.
    3. In the vCenter or ESXi Server field, enter the vCenter or ESXi hostname/IP. ESXi is the virtualization platform where you create and run virtual machines and virtual appliances. vCenter Server is the service through which you manage multiple hosts connected in a network and pool host resources.
    4. Optional: In the Port field, configure the port of the vCenter or ESXi server.
    5. In the Username and Password fields, enter your vSphere login username and password.
    6. Click Create.

    Result: The node template has the credentials required to provision nodes in vSphere.

    In the Account Access section, enter the vCenter FQDN or IP address and the credentials for the vSphere user account.

    B. Configure Node Scheduling

    Choose what hypervisor the virtual machine will be scheduled to. The configuration options depend on your version of Rancher.

      The fields in the Scheduling section should auto-populate with the data center and other scheduling options that are available to you in vSphere.

      1. In the Data Center field, choose the data center where the VM will be scheduled.
      2. Optional: Select a Resource Pool. Resource pools can be used to partition available CPU and memory resources of a standalone host or cluster, and they can also be nested.
      3. If you have a data store cluster, you can toggle the Data Store field. This lets you select a data store cluster where your VM will be scheduled to. If the field is not toggled, you can select an individual disk.
      4. Optional: Select a folder where the VM will be placed. The VM folders in this dropdown menu directly correspond to your VM folders in vSphere. Note: The folder name should be prefaced with vm/ in your vSphere config file.
      5. Optional: Choose a specific host to create the VM on. Leave this field blank for a standalone ESXi or for a cluster with DRS (Distributed Resource Scheduler). If specified, the host system’s pool will be used and the Resource Pool parameter will be ignored.

      In the Scheduling section, enter:

      • The name/path of the Data Center to create the VMs in
      • The name of the VM Network to attach to

      • The name/path of the Datastore to store the disks in

        image

      C. Configure Instances and Operating Systems

      The instances are configured differently depending on your Rancher version.

        In this section, configure the number of vCPUs, memory, and disk size for the VMs created by this template.

        In the Creation method field, you will configure the method for setting up an operating system on the node. The operating system can be installed from an ISO or from a VM template.

        VM templates are useful for setting up the operating system and other software, because they allow you to save time. For example, you could use a VM template to automatically install Kubernetes and Docker on each node. You can choose ISOs defined from templates in a vSphere data center or content library.

        The node can be created with any operating system that supports cloud-init.

        Choose the way that the VM will be created:

        • Deploy from template: Data Center: Choose a template that exists in the data center that you selected.
        • Deploy from template: Content Library: In the two fields that appear when you select this option, choose the content library. Then select the VM template from the list of templates within the content library. This template will be used to create the new VM.
        • Clone an existing virtual machine: In the Virtual machine field, choose an existing VM that the new VM will be cloned from.
        • Install from boot2docker ISO: Ensure that the OS ISO URL contains the URL of a VMware ISO release for RancherOS (rancheros-vmware.iso).

        In the Instance Options section, configure the number of vCPUs, memory, and disk size for the VMs created by this template.

        Only RancherOS VMs are supported.

        Ensure that the OS ISO URL contains the URL of the VMware ISO release for RancherOS: rancheros-vmware.iso.

        ![image](https://rancher.com/docs/img/rancher/vsphere-node-template-1.png)

        D. Add Networks

        Available as of v2.3.3

        The node template now allows a VM to be provisioned with multiple networks. In the Networks field, you can now click Add Network to add any networks available to you in vSphere.

        E. If Not Already Enabled, Enable Disk UUIDs

        In order to provision nodes with RKE, all nodes must be configured with disk UUIDs.

        As of Rancher v2.0.4, disk UUIDs are enabled in vSphere node templates by default.

        If you are using Rancher prior to v2.0.4, refer to these instructions for details on how to enable a UUID with a Rancher node template.

        F. Optional: Configure Node Tags and Custom Attributes

        The way to attach metadata to the VM is different depending on your Rancher version.

          Optional: Add vSphere tags and custom attributes. Tags allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects.

          For tags, all your vSphere tags will show up as options to select from in your node template.

          In the custom attributes, Rancher will let you select all the custom attributes you have already set up in vSphere. The custom attributes are keys and you can enter values for each one.

          Note: Custom attributes are a legacy feature that will eventually be removed from vSphere. These attributes allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects.

          Optional:

          • Provide a set of configuration parameters (instance-options) for the VMs.
          • Assign labels to the VMs that can be used as a base for scheduling rules in the cluster.
          • Customize the configuration of the Docker daemon on the VMs that will be created.

          Note: Custom attributes are a legacy feature that will eventually be removed from vSphere. These attributes allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects.

          G. Optional: Configure cloud-init

          Cloud-init is a tool that applies user data to your nodes when they boot for the first time.

          The configuration file for cloud-init is named cloud-config.yml. In the Cloud Init field, it is optional to enter a file name or URL pointing to a cloud-config.yml file. Only YAML format is supported for the cloud config.

          You can use cloud-init to automate tasks that should happen when the instance boots, such as creating users, running shell commands, adding a load balancer, or preinstalling Kubernetes on the VM.

          For examples of how to write a cloud-config file, refer to the cloud-init documentation.

          H. Saving the Node Template

          Assign a descriptive Name for this template and click Create.

          Node Template Configuration Reference

          Refer to this section for a reference on the configuration options available for vSphere node templates.

          2. Create a Kubernetes Cluster Using the Node Template

          After you’ve created a template, you can use it to stand up the vSphere cluster itself.

          To install Kubernetes on vSphere nodes, you will need to enable the vSphere cloud provider by modifying the cluster YAML file. This requirement applies to both pre-created custom nodes and for nodes created in Rancher using the vSphere node driver.

          To create the cluster and enable the vSphere provider for cluster, follow these steps:

          A. Set up the Cluster Name and Member Roles

          1. Log in to the Rancher UI as an administrator.
          2. Navigate to Clusters in the Global view.
          3. Click Add Cluster and select the vSphere infrastructure provider.
          4. Assign a Cluster Name.
          5. Assign Member Roles as required.

            Use Member Roles to configure user authorization for the cluster.

            • Click Add Member to add users that can access the cluster.
            • Use the Role drop-down to set permissions for each user.

          Note:

          If you have a cluster with DRS enabled, setting up VM-VM Affinity Rules is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes.

          B. Configure Kubernetes Options

          Use Cluster Options to choose the version of Kubernetes, what network provider will be used and if you want to enable project network isolation. To see more cluster options, click on Show advanced options.

          C. Add Node Pools to the Cluster

          Add one or more node pools to your cluster.

          A node pool is a collection of nodes based on a node template. A node Template defines the configuration of a node, like what Operating System to use, number of CPUs and amount of memory. Each node pool must have one or more nodes roles assigned.

          Notes:

          • Each node role (i.e. etcd, Control Plane, and Worker) should be assigned to a distinct node pool. Although it is possible to assign multiple node roles to a node pool, this should not be done for production clusters.
          • The recommended setup is to have a node pool with the etcd node role and a count of three, a node pool with the Control Plane node role and a count of at least two, and a node pool with the Worker node role and a count of at least two. Regarding the etcd node role, refer to the etcd Admin Guide.

          D. Optional: Add a Self-Healing Node Pool

          To make a node pool self-healing, enter a number greater than zero in the Auto Replace column. Rancher will use the node template for the given node pool to recreate the node if it becomes inactive for that number of minutes.

          Note: Self-healing node pools are designed to help you replace worker nodes for stateless applications. It is not recommended to enable node auto-replace on a node pool of master nodes or nodes with persistent volumes attached, because VMs are treated ephemerally. When a node in a node pool loses connectivity with the cluster, its persistent volumes are destroyed, resulting in data loss for stateful applications.

          E. Create the Cluster

          Click Create to start provisioning the VMs and Kubernetes services.

          Result:

          • Your cluster is created and assigned a state of Provisioning. Rancher is standing up your cluster.
          • You can access your cluster after its state is updated to Active.
          • Active clusters are assigned two Projects, Default (containing the namespace default) and System (containing the namespaces cattle-system,ingress-nginx,kube-public and kube-system, if present).

          3. Optional: Provision Storage

          For an example of how to provision storage in vSphere using Rancher, refer to the cluster administration section.

          In order to provision storage in vSphere, the vSphere provider must be enabled.

          Enable the vSphere Cloud Provider for the Cluster

          1. Set Cloud Provider option to Custom.

            vsphere-node-driver-cloudprovider

          2. Click on Edit as YAML

          3. Insert the following structure to the pre-populated cluster YAML. As of Rancher v2.3+, this structure must be placed under rancher_kubernetes_engine_config. In versions prior to v2.3, it has to be defined as a top-level field. Note that the name must be set to vsphere.

            rancher_kubernetes_engine_config: # Required as of Rancher v2.3+
  cloud_provider:
      name: vsphere
      vsphereCloudProvider:
          [Insert provider configuration]

            Rancher uses RKE (the Rancher Kubernetes Engine) to provision Kubernetes clusters. Refer to the vSphere configuration reference in the RKE documentation for details about the properties of the vsphereCloudProvider directive.