vSphere Cloud Provider

The vSphere Cloud Provider interacts with VMware infrastructure (vCenter or standalone ESXi server) to provision and manage storage for persistent volumes in a Kubernetes cluster.

When provisioning Kubernetes using RKE CLI or using RKE clusters in Rancher, the vSphere Cloud Provider can be enabled by configuring the cloud_provider directive in the cluster YAML file.

Prerequisites

1. You’ll need to have credentials of a vCenter/ESXi user account with privileges allowing the cloud provider to interact with the vSphere infrastructure to provision storage. Refer to this document to create and assign a role with the required permissions in vCenter.
2. VMware Tools must be running in the Guest OS for all nodes in the cluster.
3. All nodes must be configured with disk UUIDs. This is required so that attached VMDKs present a consistent UUID to the VM, allowing the disk to be mounted properly. See Enabling Disk UUIDs.

Clusters provisioned with RKE CLI

To enable the vSphere Cloud Provider in the cluster, you must add the top-level cloud_provider directive to the cluster configuration file, set the name property to vsphere and add the vsphereCloudProvider directive containing the configuration matching your infrastructure. See the configuration reference for the gory details.

Clusters provisioned with Rancher

When provisioning clusters in Rancher using the vSphere node driver or on pre-created custom nodes the cluster YAML file must be modified in order to enable the cloud provider.

1. Log in to the Rancher UI as admin user.
2. Navigate to Clusters in the Global view.
3. Click Add Cluster and select the vSphere infrastructure provider.
4. Assign a Cluster Name.
5. Assign Member Roles as required.
6. Expand Cluster Options and configure as required.

7. Set Cloud Provider option to Custom.

   

8. Click on Edit as YAML

9. Insert the following structure to the pre-populated cluster YAML. As of Rancher v2.3+, this structure must be placed under rancher_kubernetes_engine_config. In versions prior to v2.3, it has to be defined as a top level field. Note that the name must be set to vsphere. Refer to the configuration reference to learn about the properties of the vsphereCloudProvider directive.

   rancher_kubernetes_engine_config: # Required as of Rancher v2.3+
     cloud_provider:
         name: vsphere
         vsphereCloudProvider:
             [Insert provider configuration]

10. Configure the Node Pools per your requirements while ensuring to use a node template that enables disk UUIDs for the VMs (See [Annex - Enable disk UUIDs for vSphere VMs]).

11. Click on Create to start provisioning the VMs and Kubernetes services.

Configuration Reference

Note: This documentation reflects the new vSphere Cloud Provider configuration schema introduced in Kubernetes v1.9 which differs from previous versions.

The vSphere configuration options are divided into 5 groups:

• global
• virtual_center
• workspace
• disk
• network

global

The main purpose of global options is to be able to define a common set of configuration parameters that will be inherited by all vCenters defined under the virtual_center directive unless explicitly defined there.

Accordingly, the global directive accepts the same configuration options that are available under the virtual_center directive. Additionally it accepts a single parameter that can only be specified here:

Example:

(...)
    global:
      insecure-flag: true

virtual_center

This configuration directive specifies the vCenters that are managing the nodes in the cluster. You must define at least one vCenter/ESXi server. If the nodes span multiple vCenters then all must be defined.

Each vCenter is defined by adding a new entry under the virtual_center directive with the vCenter IP or FQDN as the name. All required parameters must be provided for each vCenter unless they are already defined under the global directive.

The following additional options (introduced in Kubernetes v1.11) are not yet supported in RKE.

Example:

(...)
    virtual_center:
      172.158.111.1: {}  # This vCenter inherits all it's properties from global options
      172.158.110.2:     # All required options are set explicitly
        user: vc-user
        password: othersecret
        datacenters: eu-west-2

workspace

This configuration group specifies how storage for volumes is created in vSphere. The following configuration options are available:

Example:

(...)
    workspace:
      server: 172.158.111.1 # matches IP of vCenter defined in the virtual_center block
      datacenter: eu-west-1
      folder: kubernetes
      default-datastore: ds-1

disk

The following configuration options are available under the disk directive:

network

The following configuration options are available under the network directive:

Configuration Example

Given the following:

• VMs in the cluster are running in the same datacenter eu-west-1 managed by the vCenter vc.example.com.
• The vCenter has a user provisioner with password secret with the required roles assigned, see Prerequisites.
• The vCenter has a datastore named ds-1 which should be used to store the VMDKs for volumes.
• A kubernetes folder exists in vCenter.

The corresponding configuration for the provider would then be as follows:

(...)
cloud_provider:
  name: vsphere
  vsphereCloudProvider:
    virtual_center:
      vc.example.com:
        user: provisioner
        password: secret
        datacenters: eu-west-1
    workspace:
      server: vc.example.com
      folder: kubernetes
      default-datastore: ds-1
      datacenter: eu-west-1

Annex

Enabling disk UUIDs for vSphere VMs

Depending on whether you are provisioning the VMs using the vSphere node driver in Rancher or using your own scripts or third-party tools, there are different methods available to enable disk UUIDs for VMs.

Using the Vsphere Console

The required property can be set while creating or modifying VMs in the vSphere Console:

1. For each VM navigate to the tab VM Options and click on Edit Configuration.

2. Add the parameter disk.EnableUUID with a value of TRUE.

   

Using the GOVC CLI tool

You can also modify properties of VMs with the govc command-line tool to enable disk UUIDs:

$ govc vm.change -vm <vm-path> -e disk.enableUUID=TRUE

Using Rancher node template

When creating new clusters in Rancher using vSphere node templates, you can configure the template to automatically enable disk UUIDs for all VMs created for a cluster:

1. Navigate to the Node Templates in the Rancher UI while logged in as admin user.

2. Add or edit an existing vSphere node template.

3. Under Instance Options click on Add Parameter.

4. Enter disk.enableUUID as key with a value of TRUE.

   

5. Click Create or Save.

Troubleshooting

If you are experiencing issues while provisioning a cluster with enabled vSphere Cloud Provider or while creating vSphere volumes for your workloads, you should inspect the logs of the following K8s services:

• controller-manager (Manages volumes in vCenter)
• kubelet: (Mounts vSphere volumes to pods)

If your cluster is not configured with external Cluster Logging, you will need to SSH into nodes to get the logs of the kube-controller-manager (running on one of the control plane nodes) and the kubelet (pertaining to the node where the stateful pod has been scheduled).

The easiest way to create a SSH session with a node is the Rancher CLI tool.

1. Configure the Rancher CLI for your cluster.

2. Run the following command to get a shell to the corresponding nodes:

   $ rancher ssh <nodeName>

3. Inspect the logs of the controller-manager and kubelet containers looking for errors related to the vSphere cloud provider:

   $ docker logs --since 15m kube-controller-manager
   $ docker logs --since 15m kubelet

Related Links

• vSphere Storage for Kubernetes
• Kubernetes Persistent Volumes