The vSphere Cloud Provider interacts with VMware infrastructure (vCenter or standalone ESXi server) to provision and manage storage for persistent volumes in a Kubernetes cluster.

When provisioning Kubernetes using RKE CLI or using RKE clusters in Rancher, the vSphere Cloud Provider can be enabled by configuring the cloud_provider directive in the cluster YAML file.

Prerequisites

  1. You’ll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage.¬†Refer to this document to create and assign a role with the required permissions in vCenter.
  2. VMware Tools must be running in the Guest OS for all nodes in the cluster.
  3. All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See Enabling Disk UUIDs.

Clusters provisioned with RKE CLI

To enable the vSphere Cloud Provider in the cluster, you must add the top-level cloud_provider directive to the cluster configuration file, set the name property to vsphere and add the vsphereCloudProvider directive containing the configuration matching your infrastructure. See the configuration reference for the gory details.

Clusters provisioned with Rancher

When provisioning clusters in Rancher using the vSphere node driver or on pre-created custom nodes the cluster YAML file must be modified in order to enable the cloud provider.

  1. Log in to the Rancher UI as admin user.
  2. Navigate to Clusters in the Global view.
  3. Click Add Cluster and select the vSphere infrastructure provider.
  4. Assign a Cluster Name.
  5. Assign Member Roles as required.
  6. Expand Cluster Options and configure as required.
  7. Set Cloud Provider option to Custom.

    vsphere-node-driver-cloudprovider

  8. Click on Edit as YAML

  9. Insert the following top-level structure to the pre-populated cluster YAML. Note that the name must be set to vsphere. Refer to the configuration reference to learn about the properties of the vsphereCloudProvider directive.

    cloud_provider:
        name: vsphere
        vsphereCloudProvider:
            [Insert provider configuration]
  10. Configure the Node Pools per your requirements while ensuring to use a node template that enables disk UUIDs for the VMs (See [Annex - Enable disk UUIDs for vSphere VMs]).

  11. Click on Create to start provisioning the VMs and Kubernetes services.

Configuration Reference

Note: This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions.

The vSphere configuration options are divided into 5 groups:

  • global
  • virtual_center
  • workspace
  • disk
  • network

global

The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the virtual_center directive unless explicitly defined there.

Accordingly, the global directive accepts the same configuration options that are available under the virtual_center directive. Additionally it accepts a single parameter that can only be specified here:

global Options Type Required Description
insecure-flag boolean Set to true if the vCenter/ESXi uses a self-signed certificate.

Example:

(...)
    global:
      insecure-flag: true

virtual_center

This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined.

Each vCenter is defined by adding a new entry under the virtual_center directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the global directive.

virtual_center Options Type Required Description
user string * vCenter/ESXi user used to authenticate with this server.
password string * User’s password.
port string Port to use to connect to this server. Defaults to 443.
datacenters string * Comma-separated list of all datacenters in which cluster nodes are running in.
soap-roundtrip-count uint Round tripper count for API requests to the vCenter (num retries = value - 1).

The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE.

virtual_center Options Type Required Description
secret-name string Name of secret resource containing credential key/value pairs. Can be specified in lieu of user/password parameters.
secret-namespace string Namespace in which the secret resource was created in.
ca-file string Path to CA cert file used to verify the vCenter certificate.

Example:

(...)
    virtual_center:
      172.158.111.1: {}  # This vCenter inherits all it's properties from global options
      172.158.110.2:     # All required options are set explicitly
        user: vc-user
        password: othersecret
        datacenters: eu-west-2

workspace

This configuration group specifies how storage for volumes is created in vSphere. The following configuration options are available:

workspace Options Type Required Description
server string * IP or FQDN of the vCenter/ESXi that should be used for creating the volumes. Must match one of the vCenters defined under the virtual_center directive.
datacenter string * Name of the datacenter that should be used for creating volumes. For ESXi enter ha-datacenter.
folder string * Path of folder in which to create dummy VMs used for volume provisioning (relative from the root of the datastore), e.g. “kubernetes”.
default-datastore string Name of default datastore to place VMDKs if neither datastore or storage policy are specified in the volume options of a PVC. If datastore is located in a storage folder or is a member of a datastore cluster, specify the full path.
resourcepool-path string Absolute or relative path to the resource pool where the dummy VMs for Storage policy based provisioning should be created. If a relative path is specified, it is resolved with respect to the datacenter’s host folder. Examples: /<dataCenter>/host/<hostOrClusterName>/Resources/<poolName>, Resources/<poolName>. For standalone ESXi specify Resources.

Example:

(...)
    workspace:
      server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block
      datacenter: eu-west-1
      folder: k8s-dummy
      default-datastore: ds-1

disk

The following configuration options are available under the disk directive:

disk Options Type Required Description
scsicontrollertype string SCSI controller type to use when attaching block storage to VMs. Must be one of: lsilogic-sas or pvscsi. Default: pvscsi.

network

The following configuration options are available under the network directive:

network Options Type Required Description
public-network string Name of public VM Network to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.

Configuration Example

Given the following:

  • VMs in the cluster are running in the same datacenter eu-west-1 managed by the vCenter vc.example.com.
  • The vCenter has a user provisioner with password secret with the required roles assigned, see Prerequisites.
  • The vCenter has a datastore named ds-1 which should be used to store the VMDKs for volumes.
  • A k8s-dummy folder exists in the root of the datastore.

The corresponding configuration for the provider would then be as follows:

(...)
cloud_provider:
  name: vsphere
  vsphereCloudProvider:
    virtual_center:
      vc.example.com:
        user: provisioner
        password: secret
        datacenters: eu-west-1
    workspace:
      server: vc.example.com
      folder: k8s-dummy
      default-datastore: ds-1
      datacenter: eu-west-1

Annex

Enabling disk UUIDs for vSphere VMs

Depending on whether you are provisioning the VMs using the vSphere node driver in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs.

Using the Vsphere Console

The required property can be set while creating or modifying VMs in the vSphere Console:

  1. For each VM navigate to the tab VM Options and click on Edit Configuration.
  2. Add the parameter disk.EnableUUID with a value of TRUE.

    vsphere-advanced-parameters

Using the GOVC CLI tool

You can also modify properties of VMs with the govc command-line tool to enable disk UUIDs:

$ govc vm.change -vm <vm-path> -e disk.enableUUID=TRUE

Using Rancher node template

When creating new clusters in Rancher using vSphere node templates, you can configure the template to automatically enable disk UUIDs for all VMs created for a cluster:

  1. Navigate to the Node Templates in the Rancher UI while logged in as admin user.

  2. Add or edit an existing vSphere node template.

  3. Under Instance Options click on Add Parameter.

  4. Enter disk.enableUUID as key with a value of TRUE.

    vsphere-nodedriver-enable-uuid

  5. Click Create or Save.

Troubleshooting

If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services:

  • controller-manager (Manages volumes in vCenter)
  • kubelet: (Mounts vSphere volumes to pods)

If your cluster is not configured with external Cluster Logging, you will need to SSH into nodes to get the logs of the kube-controller-manager (running on one of the control plane nodes) and the kubelet (pertaining to the node where the stateful pod has been scheduled).

The easiest way to create a SSH session with a node is the Rancher CLI tool.

  1. Configure the Rancher CLI for your cluster.
  2. Run the following command to get a shell to the corresponding nodes:

    $ rancher ssh <nodeName>
  3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider:

    $ docker logs --since 15m kube-controller-manager
    $ docker logs --since 15m kubelet