Skip to content

Create a Kubevirt cluster

Install an OCP cluster running on VMs within a management OCP cluster

Prerequisites

  • Admin access to an OpenShift cluster (version 4.14+) specified by the KUBECONFIG environment variable.
  • The management OCP cluster must have wildcard dns routes enabled. oc patch ingresscontroller -n openshift-ingress-operator default --type=json -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'
  • The management OCP cluster must have Openshift Virtualization (4.14+) installed on it. Instructions for installing Openshift Virtualization
  • The Management OCP cluster must be configured with OVNKubernetes as the default pod network CNI.
  • The Management OCP cluster must have LoadBalancer service support. Instructions for installing MetalLB
  • The management OCP cluster must have default storage class. Storage Configuration Documentation Example of how to set a default storage class: oc patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  • The OpenShift CLI (oc) or Kubernetes CLI (kubectl).
  • A valid pull secret file for the quay.io/openshift-release-dev repository.
  • A network MTU of 9000 or larger on the OCP cluster hosting the KubeVirt VMs is recommended for optimal network performance. Smaller MTU settings will work, but network latency and throughput of the hosted pods will be impacted. Multiqueue should only be enabled on NodePools when the MTU is 9000 or larger.

Installing HyperShift Operator and cli tooling

Before creating a guest cluster, the hcp cli, hypershift cli, and HyperShift Operator must be installed.

The hypershift cli tool is a development tool that is used to install developer builds of the HyperShift Operator.

The hcp cli tool is used to manage the creation and destruction of guest clusters.

Build the HyperShift and HCP CLI

The command below builds latest hypershift and hcp cli tools from source and places the cli tool within the /usr/local/bin directory.

Note

The command below is the same if you use docker.

podman run --rm --privileged -it -v \
$PWD:/output docker.io/library/golang:1.20 /bin/bash -c \
'git clone https://github.com/openshift/hypershift.git && \
cd hypershift/ && \
make hypershift product-cli && \
mv bin/hypershift /output/hypershift && \
mv bin/hcp /output/hcp'

sudo install -m 0755 -o root -g root $PWD/hypershift /usr/local/bin/hypershift
sudo install -m 0755 -o root -g root $PWD/hcp /usr/local/bin/hcp
rm $PWD/hypershift
rm $PWD/hcp

Deploy the HyperShift Operator

Use the hypershift cli tool to install the HyperShift operator into the management cluster.

hypershift install

You will see the operator running in the hypershift namespace:

oc -n hypershift get pods

NAME                        READY   STATUS    RESTARTS   AGE
operator-755d587f44-lrtrq   1/1     Running   0          114s
operator-755d587f44-qj6pz   1/1     Running   0          114s

Create a HostedCluster

Once all the prerequisites are met, and the HyperShift operator is installed, it is now possible to create a guest cluster.

Below is an example of how to create a guest cluster using environment variables and the hcp cli tool.

Note

The --release-image flag could be used to provision the HostedCluster with a specific OpenShift Release (the hypershift operator has a support matrix of releases supported by a given version of the operator)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
export CLUSTER_NAME=example
export PULL_SECRET="$HOME/pull-secret"
export MEM="6Gi"
export CPU="2"
export WORKER_COUNT="2"

hcp create cluster kubevirt \
--name $CLUSTER_NAME \
--node-pool-replicas $WORKER_COUNT \
--pull-secret $PULL_SECRET \
--memory $MEM \
--cores $CPU

Note

A default NodePool will be created for the cluster with 2 vm worker replicas per the --node-pool-replicas flag.

After a few moments we should see our hosted control plane pods up and running:

oc -n clusters-$CLUSTER_NAME get pods

NAME                                                  READY   STATUS    RESTARTS   AGE
capi-provider-5cc7b74f47-n5gkr                        1/1     Running   0          3m
catalog-operator-5f799567b7-fd6jw                     2/2     Running   0          69s
certified-operators-catalog-784b9899f9-mrp6p          1/1     Running   0          66s
cluster-api-6bbc867966-l4dwl                          1/1     Running   0          66s
.
.
.
redhat-operators-catalog-9d5fd4d44-z8qqk              1/1     Running   0          66s

A guest cluster backed by KubeVirt virtual machines typically takes around 10-15 minutes to fully provision. The status of the guest cluster can be seen by viewing the corresponding HostedCluster resource. For example, the output below reflects what a fully provisioned HostedCluster object looks like.

1
2
3
4
oc get --namespace clusters hostedclusters

NAME            VERSION   KUBECONFIG                       PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
example         4.14.0    example-admin-kubeconfig         Completed  True        False         The hosted control plane is available

External infrastructure credentials

When KubeVirt virtual machines run on a separate infrastructure cluster, spec.platform.kubevirt.credentials.infraKubeConfigSecret references a Secret in the HostedCluster namespace. Its name and key must identify a non-empty kubeconfig with a valid current context and referenced user and cluster.

HyperShift validates the credentials before copying them into the control plane namespace or using them for infrastructure discovery. Validation applies to all users and clusters in each consumed kubeconfig, including inactive entries. It rejects the following settings instead of silently removing them:

  • Exec credential plugins (users[].user.exec).
  • Authentication provider plugins (users[].user.auth-provider).
  • Bearer token files (users[].user.tokenFile), even with an inline token.
  • Client certificate/key files (users[].user.client-certificate and users[].user.client-key), even when embedded data is also supplied.
  • CA files (clusters[].cluster.certificate-authority).
  • Disabled TLS verification (clusters[].cluster.insecure-skip-tls-verify: true).

Use static inline bearer tokens or embedded client certificate/key credentials with TLS verification enabled. Use embedded certificate-authority-data when providing a custom CA. Kubeconfigs must not reference files on the machine where they were generated or on a privileged consumer's filesystem.

The control plane Secret is named kubevirt-infra-credentials. CAPK and the KubeVirt CSI driver consume its kubeconfig key, which always carries the validated data of the configured source key, so every consumer targets the selected infrastructure cluster. Optional CAPK namespace metadata is preserved; unrelated source entries, including a separate kubeconfig entry, and stale target keys are not copied. The source Secret is not modified.

Infrastructure discovery uses the configured source key as well, so discovery, image caching, CAPK, and CSI cannot be split across different infrastructure clusters by additional entries in the same Secret. Cached infrastructure clients are replaced when the configured kubeconfig or infrastructure namespace changes.

Upgrades and rejected credentials

This validation intentionally rejects configurations that older operators might have accepted. Check external infrastructure credentials before upgrading, including any inactive users and clusters, and replace unsupported authentication methods with safe static credentials.

When validation fails, the HostedCluster reports PlatformCredentialsFound=False with a message identifying the Secret/key or prohibited field, without including the credential contents. A previously copied unsafe or malformed target Secret is removed. A safe existing target is preserved when an invalid replacement is submitted. Cleanup failures are reported rather than ignored.

To recover, correct the data in the referenced source Secret, keeping its existing name and key. After successful reconciliation, the validated credentials are published and PlatformCredentialsFound becomes True. HyperShift's cached infrastructure clients are revalidated and replaced when credentials change.

Keep the referenced source Secret and valid credentials available until NodePool and HostedCluster teardown finishes. Image-cache cleanup and other infrastructure operations still validate the source before using a cached client. If teardown is blocked by rejected or missing credentials, restore safe credentials and retry; do not rely on an old cached client or remove finalizers to bypass validation.

Previously copied unsafe targets are also inspected during deletion and when the source has already been corrected, before unrelated reconciliation prerequisites. Missing source credentials do not by themselves block non-consuming teardown. Historical target inspection treats every data entry except namespace as kubeconfig credentials, including keys copied by earlier operators. An unsafe or unparsable historical entry causes the operator-managed target to be removed, even if its current credential keys are valid. If safe source credentials are restored while the control plane namespace is still active, HyperShift republishes the validated target for CAPK's remaining Machine/VM deletion operations. It does not recreate deleted or terminating control plane namespaces.

Warning

Removing a Secret does not revoke credentials already loaded by running consumers. When remediating previously accepted unsafe configurations, coordinate credential revocation and any required restarts of CAPK, cloud controller, and CSI workloads with your cluster administrator. Verify recovery after reconciliation; do not assume deleting or correcting a Secret alone removes previously loaded state.

Accessing the HostedCluster

CLI access to the guest cluster is gained by retrieving the guest cluster's kubeconfig. Below is an example of how to retrieve the guest cluster's kubeconfig using the hcp cli.

hcp create kubeconfig --name $CLUSTER_NAME > $CLUSTER_NAME-kubeconfig

If we access the cluster, we will see we have two nodes.

oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

NAME                  STATUS   ROLES    AGE   VERSION
example-n6prw         Ready    worker   32m   v1.27.4+18eadca
example-nc6g4         Ready    worker   32m   v1.27.4+18eadca

We can also check the ClusterVersion:

oc --kubeconfig $CLUSTER_NAME-kubeconfig get clusterversion

NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0        True        False         5m39s   Cluster version is 4.14.0

Influencing VM Scheduling

By default, KubeVirt VMs created by a NodePool are scheduled to any available node with the appropriate capacity to run the VM. A topologySpreadConstraint is used to try to spread VMs for a NodePool out across multiple underlying nodes, but in general it is possible for the VMs to be scheduled to any node that meets the requirements for running KubeVirt VMs. For availability reasons, the topologySpreadConstraint is only a soft constraint (whenUnsatisfiable: ScheduleAnyway policy) so, under certain circumstances, VMs for a single nodepool can “clump” together on a single node due to live migration to available nodes. The De-Scheduler can optionally be used to continuously redistribute VMs so that they again satisfy the topologySpreadConstraint as soon as possible. To achieve that behavior, the cluster-kube-descheduler-operator should be installed and configured with something like: 

apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
spec:
  mode: Automatic
  managementState: Managed
  deschedulingIntervalSeconds: 60
  profiles:
  - SoftTopologyAndDuplicates
  - EvictPodsWithPVC
  - EvictPodsWithLocalStorage
  profileCustomizations:
    devEnableEvictionsInBackground: true

It is possible to influence where the KubeVirt VMs within a NodePool are scheduled through the use of NodeSelectors. Below is an example of using a NodeSelector on the VMs to place the VMs only on nodes with a specific label through usage of the hcp command line and the --vm-node-selector argument.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
export CLUSTER_NAME=example
export PULL_SECRET="$HOME/pull-secret"
export MEM="6Gi"
export CPU="2"
export WORKER_COUNT="2"

hcp create cluster kubevirt \
--name $CLUSTER_NAME \
--node-pool-replicas $WORKER_COUNT \
--pull-secret $PULL_SECRET \
--memory $MEM \
--cores $CPU \
--vm-node-selector labelKey1=labelVal1,labeKey2=labelVal2

In the example above, the KubeVirt VMs will only be scheduled to nodes that contain the labels labelKey1=labelVal1 and labelKey2=labelVal2.

Creating NodePools for a Specific Architecture

On a multi-architecture KubeVirt infra cluster (one that has both amd64 and s390x nodes, for example), you can create additional NodePools that target a specific architecture using the --arch flag. The default NodePool created with the cluster uses the infra cluster's primary architecture; use this command to add a NodePool for a secondary architecture.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
export CLUSTER_NAME=example
export PULL_SECRET="$HOME/pull-secret"
export MEM="6Gi"
export CPU="2"
export WORKER_COUNT="2"

hcp create nodepool kubevirt \
  --cluster-name $CLUSTER_NAME \
  --name $CLUSTER_NAME-s390x \
  --replicas $WORKER_COUNT \
  --pull-secret $PULL_SECRET \
  --memory $MEM \
  --cores $CPU \
  --arch s390x

When --arch is set, the operator:

  • Sets the KubeVirt VM template spec.architecture field so the VM is provisioned with the correct architecture.
  • Injects a kubernetes.io/arch NodeSelector into the VM template so virt-launcher pods are scheduled only on infra nodes of the matching architecture.

Scaling an existing NodePool

Manually scale a NodePool using the oc scale command:

1
2
3
4
NODEPOOL_NAME=$CLUSTER_NAME
NODEPOOL_REPLICAS=5

oc scale nodepool/$NODEPOOL_NAME --namespace clusters --replicas=$NODEPOOL_REPLICAS

After a while, in our hosted cluster this is what we will see:

oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

NAME                  STATUS   ROLES    AGE     VERSION
example-9jvnf         Ready    worker   97s     v1.27.4+18eadca
example-n6prw         Ready    worker   116m    v1.27.4+18eadca
example-nc6g4         Ready    worker   117m    v1.27.4+18eadca
example-thp29         Ready    worker   4m17s   v1.27.4+18eadca
example-twxns         Ready    worker   88s     v1.27.4+18eadca

Adding Additional NodePools

Create additional NodePools for a guest cluster by specifying a name, number of replicas, and any additional information such as memory and cpu requirements.

For example, let's create a NodePool with more CPUs assigned to the VMs (4 vs 2):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
export NODEPOOL_NAME=$CLUSTER_NAME-extra-cpu
export WORKER_COUNT="2"
export MEM="6Gi"
export CPU="4"
export DISK="16"

hcp create nodepool kubevirt \
  --cluster-name $CLUSTER_NAME \
  --name $NODEPOOL_NAME \
  --replicas $WORKER_COUNT \
  --memory $MEM \
  --cores $CPU \
  --root-volume-size $DISK

Check the status of the NodePool by listing nodepool resources in the clusters namespace:

oc get nodepools --namespace clusters

NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
example                   example         5               5               False         False        4.14.0  
example-extra-cpu         example         2                               False         False                  True              True             Minimum availability requires 2 replicas, current 0 available

After a while, in our hosted cluster this is what we will see:

oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

NAME                      STATUS   ROLES    AGE     VERSION
example-9jvnf             Ready    worker   97s     v1.27.4+18eadca
example-n6prw             Ready    worker   116m    v1.27.4+18eadca
example-nc6g4             Ready    worker   117m    v1.27.4+18eadca
example-thp29             Ready    worker   4m17s   v1.27.4+18eadca
example-twxns             Ready    worker   88s     v1.27.4+18eadca
example-extra-cpu-zh9l5   Ready    worker   2m6s    v1.27.4+18eadca
example-extra-cpu-zr8mj   Ready    worker   102s    v1.27.4+18eadca

And the nodepool will be in the desired state:

oc get nodepools --namespace clusters

NAME                      CLUSTER         DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
example                   example         5               5               False         False        4.14.0  
example-extra-cpu         example         2               2               False         False        4.14.0

Delete a HostedCluster

To delete a HostedCluster:

hcp destroy cluster kubevirt --name $CLUSTER_NAME