Charmed Kubernetes on vSphere
Charmed Kubernetes will install and run on vSphere virtual servers.
With the addition of the vsphere-cloud-provider
and the vsphere-integrator
, your cluster will also be able
to directly use native vSphere features such as storage.
Upgrading from 1.25 to 1.26
vSphere has migrated to the out-of-tree provider and the legacy in-tree provider is marked for deprecation. Nevertheless, it is possible to migrate the workload volumes provisioned with the in-tree provider to the new out-of-tree provider. Follow the instructions below to prepare to migrate the volumes:
1. Enable privileged containers support
The new out-of-tree provider requires privileged containers. Please ensure that your Kubernetes cluster supports this. You can enable this feature using:
juju config kubernetes-control-plane allow-privileged=true
2. Install vSphere Cloud Provider
Install the vSphere Cloud Provider charm and relate it to the required components. Follow the instructions in the vsphere-cloud-provider charm documentation.
3. Prepare kube-controller and kubelet
To enable volume migration you must add the CSIMigration and CSIMigrationvSphere flags in kube-controller and kubelet options of the Kubernetes Control Plane. You can do this via Juju using:
juju config kubernetes-control-plane controller-manager-extra-args="feature-gates=CSIMigration=true,CSIMigrationvSphere=true"
juju config kubernetes-control-plane kubelet-extra-config="{featureGates: {CSIMigration: true,CSIMigrationvSphere: true}}"
4. vSphere in-tree volume migrations
Now you can follow the instructions in the vSphere documentation about Migrating In-Tree vSphere volumes.
vSphere Cloud Provider
The vsphere-cloud-provider
charm allows Charmed Kubernetes to connect to the vSphere API
via the out-of-tree cloud provider. This allow your cluster to manage parts of the vSphere infrastructure,
such as virtual disks.
vSphere integrator
The vsphere-integrator
charm simplifies working with Charmed Kubernetes on
vSphere servers. Using the credentials provided to Juju, it acts as a proxy between
Charmed Kubernetes and the underlying cloud. This charm integrates with the vsphere-cloud-provider
charm to share the credentials required for its operation.
Model configuration
If the cluster has multiple datastores or a non-default network name, you'll need to configure the model defaults before deployment. For example:
juju model-config datastore=mydatastore primary-network=mynetwork
Installing
If you install Charmed Kubernetes using the Juju bundle,
you can add both vsphere-cloud-provider
and vsphere-integrator
at the same time by using the following
overlay file (download it here):
description: Charmed Kubernetes overlay to add native vSphere support.
applications:
kubernetes-control-plane:
options:
allow-privileged: "true"
vsphere-integrator:
charm: vsphere-integrator
num_units: 1
trust: true
vsphere-cloud-provider:
charm: vsphere-cloud-provider
relations:
- - vsphere-cloud-provider:certificates
- easyrsa:client
- - vsphere-cloud-provider:kube-control
- kubernetes-control-plane:kube-control
- - vsphere-cloud-provider:external-cloud-provider
- kubernetes-control-plane:external-cloud-provider
- - vsphere-cloud-provider:vsphere-integration
- vsphere-integrator:clients
To use this overlay with the Charmed Kubernetes bundle, it is specified during deploy like this:
juju deploy charmed-kubernetes --overlay vsphere-overlay.yaml --trust
... and remember to fetch the configuration file!
juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config
Configuration
The vSphere integrator supports multiple configuration options which can be used to describe the vSphere environment.
The only required option is datastore
, as it is not included in the Juju
credential that this charm relies on. By default, this is set to datastore1.
This can be changed with:
juju config vsphere-integrator datastore='mydatastore'
You may also configure a folder and resource pool path for this charm. Details about these options can be found in the vmware documentation:
juju config vsphere-integrator folder='juju-kubernetes' respool_path='foo'
The credentials used to interact with vSphere are obtained from Juju (via '--trust' during deployment). These may be overriden by specifying credentials directly in the charm configuration:
juju config vsphere-integrator \
vsphere_ip='a.b.c.d' \
user='joe' \
password='passw0rd' \
datacenter='dc0'
When all of the credential config options are empty, this charm will fall
back to the credential data it received via juju trust
.
Storage
The vSphere charms can make use of vSphere-backed storage for Kubernetes. The steps below create a busybox pod with a persistent volume claim backed by vSphere's PersistentDisk as an example.
1. Create a storage class using the csi.vsphere.vmware.com
provisioner:
If the
vsphere-cloud-provider
charm is installed, skip this step since it creates the StorageClasscsi-vsphere-default
.SC_NAME=csi-vsphere-default kubectl get sc $SC_NAME
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-vsphere-default (default) csi.vsphere.vmware.com Delete Immediate false 0s
Without the
vsphere-cloud-provider
charm, one will need to create a storage class which can be used by Kubernetes against thecsi.vsphere.vmware.com
provisioner.
SC_NAME=mystorage
kubectl create -f - <<EOY
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ${SC_NAME}
provisioner: csi.vsphere.vmware.com
EOY
2. Create a persistent volume claim (PVC) using that storage class:
kubectl create -f - <<EOY
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: testclaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
storageClassName: ${SC_NAME}
EOY
3. Create a busybox pod with a volume using that PVC:
kubectl create -f - <<EOY
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
volumeMounts:
- mountPath: "/pv"
name: testvolume
restartPolicy: Always
volumes:
- name: testvolume
persistentVolumeClaim:
claimName: testclaim
EOY
For more configuration options and details of the permissions which the cloud provider uses, please see the vSphere Cloud Provider charm page.
See the guide to contributing or discuss these docs in our public Mattermost channel.