Skip to main content

Extend your cluster

Extend adds Multicloud capacity to a Kubernetes cluster that you already run. You keep your cluster and its control plane. Multicloud runs a pool of workers for you and shows the pool in your cluster as one virtual node. A pod that opts in runs on the pool. Every other pod stays on your own nodes.

This guide is for the person who sets up and runs Extend.

Contents

  1. What Extend is
  2. Prerequisites
  3. Set up Extend, step by step
  4. Choose which workloads run on the pool
  5. Pool lifecycle and cost
  6. Remove Extend
  7. Troubleshooting
  8. Security
  9. Limits

1. What Extend is

Cloud-managed clusters (GKE, EKS, AKS) do not accept foreign nodes. Extend avoids this limit with a virtual node. A virtual node is a Node object in your cluster. It does not run a VM. When a pod runs "on" the virtual node, a copy of the pod runs on a real worker in your pool. Your tools (kubectl, dashboards, logs) show the pod as a normal pod.

Extend uses Liqo, an open-source project that builds virtual nodes.

What runs where

Part Where it runs Who operates it
Your cluster, your nodes, your own pods Your cloud account You
Liqo control plane, virtual kubelet, and bridge (the liqo-consumer chart and the peering bundle) Your cluster, namespace liqo You install it. Multicloud publishes it.
The pool control plane (a Kubernetes API server for the pool) Multicloud Multicloud
Pool workers (VMs that run your offloaded pods) Multicloud-operated cloud accounts, any provider. Workers are spot VMs by default. Multicloud

The bridge is a cloudflared Deployment in your cluster. It carries calls from your cluster to the pool API server over port 443 through Cloudflare. Your cluster needs outbound access to Cloudflare on port 443. The pool never opens a connection into your cluster.

What you pay for

You pay for the pool workers while they run. A pending pod on the virtual node makes Multicloud buy a worker. An idle worker is removed after a delay (idleReapMinutes, default 10). Each pool has a cost cap (max_cost_per_hour) and a worker cap (max_servers). A hibernated pool runs no worker (see section 5). Your own nodes and your own cloud bill do not change.

UNVERIFIED in the product code: the price per worker hour that appears on your invoice. Check your billing page for the rate.

What Extend does not do

  • Extend does not move your control plane.
  • Extend does not give your own autoscaler more nodes. Your autoscaler keeps its own nodes.
  • A Service that spans your cluster and the pool is not supported. The cross-cluster network path is not built. Standalone pods (burst, batch, and similar jobs) work without it.
  • A pod that requests a GPU or hugepages cannot run on the pool today. The grant lists CPU, memory, pod count, and ephemeral storage only.

Terms

Term Meaning
Pool The set of Multicloud workers and their control plane that backs your virtual node.
Peering The link between your cluster and one pool. You create it with the API.
Bundle One YAML file that the API gives you once per peering. It connects your cluster to the pool.
Pool slug The opaque id of the pool: c plus 12 hex characters. It never changes. A pool that you delete and create again gets a new slug.
Consumer cluster id The id of your cluster in the peering: c- plus 8 hex digits. The API mints it. You install the chart with exactly this id.

2. Prerequisites

You need these items:

  • An organization on Multicloud and an API key with the extend scope. An organization owner creates the key (section 3.1).
  • A consumer cluster on a supported platform (below), with outbound access to Cloudflare on port 443 and to the image registries of the chart (docker.io and ghcr.io).
  • kubectl, helm 3.14 or later, curl, and jq.
  • Permission to install a chart and cluster-scoped objects (CRDs, a RuntimeClass, webhooks) in the cluster.
  • The pod CIDR and the service CIDR of your cluster (below).
  • Spare CPU for the virtual node (below). About 115m of free CPU requests is the minimum.
  • Kubernetes 1.30 or later is recommended. Two chart features need it: the DaemonSet admission policy and the wake-latency helper (section 7.4).

The examples use $API for the base URL of the Multicloud API and $KEY for your API key:

export API=https://api.multicloud.io
export KEY=<your Extend API key>   # store it like a password

Spare CPU for the virtual node

The virtual node counts as one more node in your cluster. Autoscalers that scale with the node count then add replicas. On GKE these are kube-dns-autoscaler (one more kube-dns replica) and konnectivity-agent-autoscaler (one more konnectivity-agent replica). The two replicas request about 115m of CPU together. Your cluster needs this CPU free on a real node, or it needs node autoscaling. With node autoscaling, the extra replicas can add a real node. You pay for that node. Check the free CPU before you install:

kubectl describe nodes | grep -A8 'Allocated resources'

A cluster with 95% or more of the CPU requested is too full.

Supported consumer clusters

Platform Status
GKE, VPC-native Tested end to end. This is the supported platform.
GKE, routes-based Not supported today. The route controller cannot create a route for the virtual node. The node keeps a network-unavailable taint, and offloaded pods stay Pending. A legacy network forces this mode. Use a VPC-native cluster.
GKE Autopilot Not tested. Keep the Liqo pods off serverless pools if you try it (chart values liqo.common.nodeSelector and tolerations).
EKS Not tested. The CIDR lookup commands below, the cloud node controller behavior with a virtual node, DaemonSets that tolerate every taint (aws-node, kube-proxy), and the Helm install are all unproven. Try it on a test cluster first. Do not use Fargate for the Liqo pods.
AKS Not tested.

Find the pod and service CIDRs

Liqo needs both ranges, and the chart refuses to install without them.

GKE (replace the names):

gcloud container clusters describe <cluster> --zone <zone> \
  --format="value(clusterIpv4Cidr,servicesIpv4Cidr)"

The first value is the pod CIDR. The second value is the service CIDR.

To check that the cluster is VPC-native, read ipAllocationPolicy.useIpAliases:

gcloud container clusters describe <cluster> --zone <zone> \
  --format="value(ipAllocationPolicy.useIpAliases)"   # expect True

EKS (not tested; verify the result yourself):

# Pod CIDR: the VPC CIDR, when you use the default aws-vpc-cni
aws ec2 describe-vpcs --vpc-ids <vpc> --query 'Vpcs[0].CidrBlock'
# Service CIDR
aws eks describe-cluster --name <cluster> \
  --query 'cluster.kubernetesNetworkConfig.serviceIpv4Cidr'

3. Set up Extend, step by step

The flow has seven steps:

  1. Create a pool.
  2. Create a peering.
  3. Install the chart.
  4. Fetch the bundle (once) and apply it.
  5. Check the virtual node.
  6. Offload a namespace.
  7. Run a first burst workload.

The MCP server of Multicloud has the same operations as tools (create_extend_pool, create_peering, list_peerings, delete_peering, hibernate_pool, wake_pool, rotate_pool_credentials, get_pool_credential_rotation). This guide shows the REST API.

3.1 Get an API key

An owner of the organization creates the key with the Bearer token of the website session:

curl -fsS -X POST "$API/api/v1/orgs/<org-slug>/keys" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"extend","scopes":["extend"]}'

The response (201) holds the key in the field key. The API shows the key once. An organization can hold 10 active Extend keys. The scope extend opens /api/v1/extend/*. It also adds org:read. A key without the needed scope gets 403, and the message names the scope.

3.2 Create the pool

curl -fsS -X POST "$API/api/v1/extend/pools" -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"burst","max_servers":3,"max_cost_per_hour":0.5}'

The call returns 202. All body fields except name are optional:

Field Limits Default
name 1 to 40 characters: lower-case letters, digits, hyphen. Start and end with a letter or digit. required
max_servers 1 to 50 3
max_cost_per_hour Above 0, at most 50 0.2

The pool keeps at least 1 worker while it is awake. The default worker constraint asks for at least 4000 MB of memory and a low preemption risk. The API has no call to change these values. Set them at creation. An optional Idempotency-Key header makes a retry safe for 24 hours. An organization can hold 3 enabled Extend pools.

Poll until status is READY:

until [ "$(curl -fsS -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/burst" | jq -r .status)" = READY ]; do
  sleep 10
done

While the pool is not ready, message names the gate that blocks it: TCP_NOT_READY, LIQO_NOT_SYNCED, or TENANT_RBAC_NOT_READY. The first start of a pool takes several minutes because Multicloud provisions a worker.

3.3 Create the peering

curl -fsS -X POST "$API/api/v1/extend/pools/burst/peerings" -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" -d '{}'

The body is required. An empty object is valid. Without a body the API returns 422. The call returns 202 with these fields:

Field Meaning
id The peering id. You need it in the next steps.
consumer_cluster_id The id of your cluster (c- plus 8 hex digits).
status PENDING
reason, message Empty at first.
helm_command The exact install command, with two placeholders.

The body may ask for a capacity grant: cpu_millicores (100 to 1,000,000), memory_mb (128 to 4,000,000), pods (1 to 10,000), ephemeral_storage_mb (1024 to 10,000,000). A request is not a guarantee. The pool policy grants the smallest of your request, the per-customer cap, and the pool ceiling (max_servers times the largest worker). An omitted field gets the pool default. To change a grant later, revoke the peering and create a new one. An unknown field gets 422.

A peering that you create before the pool is ready waits in PENDING. It becomes FAILED after 20 minutes without progress. You cannot create a peering on a pool that is HIBERNATING or HIBERNATED (409 POOL_HIBERNATED). Wake the pool first.

3.4 Install the chart

Take helm_command from the response. Replace <POD_CIDR> and <SERVICE_CIDR> with your ranges (section 2). The command looks like this:

helm install liqo-consumer oci://registry-1.docker.io/multicloud/liqo-consumer-chart \
  --version 0.2.2 -n liqo --create-namespace \
  --set liqo.ipam.podCIDR=<POD_CIDR> --set liqo.ipam.serviceCIDR=<SERVICE_CIDR> \
  --set liqo.discovery.config.clusterID=<consumer_cluster_id> \
  --set-string reflectionKick.remoteClusterID=<pool slug>

Use the command that the API returns. It carries the current chart version and your ids. The last option carries your pool slug. This is where you first see the slug. Chart 0.2.2 runs Liqo v1.2.

The chart installs the Liqo control plane in the namespace liqo. Section 8 lists what it installs.

3.5 Fetch and apply the bundle

Poll the peering until status is READY:

curl -fsS -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/burst/peerings/<id>" | jq '{status,reason,message}'

The first read with status READY also contains the field bundle. The API clears the bundle in the same step that returns it. Every later read omits it. Save it immediately and store the file like a secret (section 8):

curl -fsS -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/burst/peerings/<id>" \
  | jq -r '.bundle' > multicloud-peering.yaml
chmod 600 multicloud-peering.yaml

Check that the file is not empty before you continue. If the API answers 503 with "bundle temporarily unavailable", the bundle is still stored. Try again. If you lose the bundle, revoke the peering and create a new one.

Apply the bundle and wait for the pre-flight Job:

kubectl apply -f multicloud-peering.yaml
kubectl -n liqo wait --for=condition=complete job/multicloud-peering-preflight-<id>

The pre-flight Job fails when the cluster id of your install differs from consumer_cluster_id. In that case, reinstall the chart with the correct id.

3.6 Check the virtual node

kubectl get nodes -l liqo.io/type=virtual-node -L multicloud.io/pool

The virtual node must be Ready. It carries the taint multicloud.io/pool=<pool slug>:NoSchedule and the label multicloud.io/pool=<pool slug>. The label column shows your pool slug.

Measured on a Multicloud test environment with a GKE consumer cluster (one pool, 2026-10-13):

Step Time
Apply the bundle to virtual node Ready about 2 minutes
Start of a pod when a worker is already running about 12 seconds
Start of a pod when the pool must add a worker (cold scale-out) about 5.5 minutes

A wake of a hibernated pool to a Ready virtual node took about 5 minutes (2026-10-29, one run). These numbers come from a few runs on one pool. They are not a service level.

3.7 Offload a namespace

A namespace must be offloaded before a pod in it can use the pool. Create a NamespaceOffloading object in the namespace:

apiVersion: offloading.liqo.io/v1beta1
kind: NamespaceOffloading
metadata: {name: offloading, namespace: my-team}
spec:
  namespaceMappingStrategy: DefaultName
  podOffloadingStrategy: LocalAndRemote
  clusterSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - {key: liqo.io/type, operator: In, values: [virtual-node]}

Keep the clusterSelector. With Liqo v1.1.2 and v1.2.0 an object without it stays in NoClusterSelected, and no pod is offloaded. liqoctl offload namespace my-team writes the selector for you. Keep LocalAndRemote. The strategy Remote forces every pod of the namespace onto the pool, and a pod without the toleration then never schedules.

3.8 Run a first burst workload

Replace <slug> with your pool slug (section 3.6):

apiVersion: v1
kind: Pod
metadata: {name: burst-hello, namespace: my-team}
spec:
  tolerations:
  - {key: multicloud.io/pool, operator: Equal, value: <slug>, effect: NoSchedule}
  nodeSelector: {multicloud.io/pool: <slug>}
  restartPolicy: Never
  containers:
  - name: hello
    image: busybox:1.36
    command: ["sh", "-c", "echo hello from the pool; uname -n; sleep 30"]
    resources:
      requests: {cpu: 100m, memory: 64Mi}

kubectl apply -f burst-hello.yaml
kubectl -n my-team get pod burst-hello -o wide -w   # NODE shows the virtual node
kubectl -n my-team logs burst-hello

Declare requests on every offloaded container. The grant bounds declared requests. If the pool has no free worker, the pod stays Pending until Multicloud adds one (about 5.5 minutes in the measured case). Delete the test pod when you are done.

4. Choose which workloads run on the pool

Nothing lands on the pool unless a pod opts in. The virtual node carries a taint and a label that contain the pool slug:

  • Taint: multicloud.io/pool=<slug>:NoSchedule
  • Label: multicloud.io/pool=<slug>

You do not need to reference any other label or taint. Do not use liqo.io/type or virtual-node.liqo.io/* in your manifests. They belong to the bridge implementation and may change. Do not edit or remove the taint or label of the virtual node.

Goal What to add to the pod template
Run on one specific pool Toleration for multicloud.io/pool=<slug> and nodeSelector: multicloud.io/pool: <slug>. You need both.
Run on any Multicloud pool Toleration with operator: Exists for multicloud.io/pool and a required nodeAffinity with multicloud.io/pool Exists.
Run on your own nodes only (default) Nothing. A pod without the toleration can never schedule on the virtual node.
Allow either (use the pool when local nodes are full) The toleration, no nodeSelector, and a topologySpreadConstraint so replicas spread.

Run on one specific pool

spec:
  tolerations:
    - key: multicloud.io/pool
      operator: Equal
      value: <slug>
      effect: NoSchedule
  nodeSelector:
    multicloud.io/pool: <slug>
  containers:
    - name: worker
      image: my-registry/batch-worker:1.0
      resources:
        requests: { cpu: 500m, memory: 512Mi }
        limits: { cpu: "1", memory: 1Gi }

Run on any Multicloud pool

spec:
  tolerations:
    - key: multicloud.io/pool
      operator: Exists
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: multicloud.io/pool
                operator: Exists

Keep a pod local

Add no toleration. To make the intent visible in the manifest, add a required nodeAffinity with multicloud.io/pool DoesNotExist.

Allow either

spec:
  tolerations:
    - key: multicloud.io/pool
      operator: Equal
      value: <slug>
      effect: NoSchedule
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels: { app: report-renderer }

The toleration only permits the virtual node. The Kubernetes scheduler spreads replicas across your nodes and the pool, based on fit.

Rules

  • With operator: Equal, the toleration value must equal the slug exactly. Use Exists to tolerate every pool.
  • A recreated pool has a new slug. Update your manifests.
  • A nodeSelector without a toleration does nothing. The taint blocks the pod, and it stays Pending.
  • A toleration without a nodeSelector means "either". It does not mean "prefer the pool".
  • Ready-made manifests are in the repository folder docs/examples/pool-targeting/.

5. Pool lifecycle and cost

Statuses

GET /api/v1/extend/pools/<name> returns status and message:

status Meaning
PROVISIONING A readiness gate is not met. message names it. STATE_STALE means the pool state was not refreshed for 5 minutes.
READY The pool serves pods.
HIBERNATING A hibernate was accepted. message: DRAINING_WORKERS or CONTROL_PLANE_STOPPING.
HIBERNATED No worker runs and the control plane is stopped.
WAKING A wake was accepted. message names the first gate that blocks.
DISABLED The pool was deleted.

The response also has max_servers, max_cost_per_hour, and hibernated_reason. hibernated_reason is PAYMENT_SUSPENDED when Multicloud hibernated the pool because the payment of the organization is suspended. It is null otherwise.

Scaling and cost

  • The pool starts with one worker and adds workers when pods wait on the virtual node.
  • The pool never exceeds max_servers workers.
  • When the cost of active workers reaches max_cost_per_hour, the pool adds no worker until the cost falls. Pending pods wait.
  • An idle worker is removed after 10 minutes by default.
  • Workers are spot VMs by default. A cloud provider can reclaim a spot VM at any time. Multicloud then launches a replacement. Write offloaded workloads so that they tolerate a restart.
  • The grant limits the capacity of the virtual node. A pod whose declared requests exceed the grant is refused.

Hibernate and wake

Hibernate is the off switch that you can reverse. It removes every worker and stops the pool control plane. The pool keeps its name, its identity, and its peerings. It still counts toward the limit of 3 pools.

curl -fsS -X POST -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/burst/hibernate"   # 202
curl -fsS -X POST -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/burst/wake"        # 202

Both calls are idempotent and need no body. Wake does not need any action on your cluster. Poll the pool until status is READY, then check the virtual node:

kubectl wait node -l liqo.io/type=virtual-node --for=condition=Ready --timeout=10m

A wake took about 5 minutes to a Ready virtual node in the measured case (2026-10-29). The pool is READY later, because it waits for a worker. Earlier runs on the test environment took 8 to 18 minutes from the wake request to READY. Spot reclaim of the first worker caused the longest run. Plan for up to 20 minutes. A wake of an organization with a suspended payment is refused with 402. Nothing hibernates or wakes a pool by itself. You call the API.

What you see while the pool is off

Event Pool status Peering status Your cluster
Normal READY READY Virtual node Ready
Hibernate accepted HIBERNATING, then HIBERNATED SUSPENDED, reason POOL_HIBERNATED Virtual node NotReady. Offloaded pods stop.
Wake accepted WAKING SUSPENDED, reason POOL_WAKING Virtual node NotReady
Wake complete READY READY Liqo re-creates the virtual node if needed. Offloaded pods run again.

SUSPENDED is a view. The peering works again when the pool is READY. A READY peering does not prove that the virtual node is Ready. Check the node.

What happens to burst pods while the pool is off:

  • Offloaded pods stop, because no worker runs.
  • New pods that need the pool stay Pending until the pool wakes. Liqo retries them after the pool returns. A helper CronJob in the chart speeds this up (section 7.4).
  • Pods that hold only a toleration (the "either" mode) may run on your own nodes (inferred from the Kubernetes scheduler; not tested during a pool sleep).
  • The restart behavior of your workload controllers (Deployment, Job) for pods that were on the pool at hibernate time is not documented in detail. Test your workload before you rely on it.

Long pool outage: being tested

A long outage (hours to days) of the pool control plane is being tested now. This applies to a long hibernate and to an unplanned outage. Documented behavior today:

  • Your Liqo objects stay in your cluster. You do not need to create a new peering after a hibernate.
  • The virtual node returns by itself when the pool returns (tested for short sleeps).
  • While the pool is away, your cluster keeps retrying the bridge. This produces error log lines in the liqo namespace.
  • The credentials of the bridge are renewed by Multicloud before they expire (see section 8).

Behavior for outages longer than that is being tested. This guide will describe the result when the test ends. Do not plan a production workload that depends on a recovery after a multi-day outage until then.

Rotate the credentials

Rotate the credentials when you think a bundle, a key or a kubeconfig leaked. The pool keeps its name, its slug and its peerings. The call needs an API key with the org:admin scope (the same scope as delete); an extend-only key gets 403.

curl -fsS -X POST -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/$POOL/rotate-credentials"
curl -fsS -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/$POOL/rotate-credentials"   # poll until COMPLETED

The platform replaces the control-plane admin certificate, the worker-join secret, and the bundle of every READY peering. Each peering returns to PENDING, then READY with a new bundle. The old bundle is revoked only after the pool is ready and its Liqo part is healthy, so it can keep working for a short time. When it is revoked, the virtual node is lost and offloaded pods stop until you apply the new bundle. You must re-fetch and re-apply the bundle:

  1. Wait until the peering is READY again (section 7.2).
  2. Fetch the new bundle once (GET .../peerings/<id>) and apply it with kubectl apply -f. It is the same peering, so you do not install the chart again.
  3. Check the virtual node (section 3.6).

While a rotation runs, you cannot hibernate the pool (409 ROTATION_IN_PROGRESS). Workers that joined before the rotation keep working for 24 hours. Replace them in that time (hibernate and wake the pool). The certificate authority is not rotated. If the authority leaked, delete the pool and create a new one.

Delete the pool

DELETE /api/v1/extend/pools/<name> ends the pool for good. A deleted pool never returns. A new pool with the same name gets a new slug and needs a new peering. The call returns 409 POOL_HAS_PEERINGS while a peering is PENDING, READY, or FAILED. Revoke the peerings first (section 6), or add ?cascade=true to revoke them with the delete.

6. Remove Extend

Follow this order:

  1. Remove burst workloads from the pool. Delete them, or remove the toleration and nodeSelector.
  2. Delete the NamespaceOffloading objects: 
    kubectl delete namespaceoffloadings.offloading.liqo.io --all -A
  3. Revoke the peering and remove the bundle objects: 
    curl -fsS -X DELETE -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/burst/peerings/<id>"   # 202
kubectl delete -f multicloud-peering.yaml
    The peering moves to REVOKING, then REVOKED. Multicloud removes the access token and the pool identity. If you lost the bundle file, skip kubectl delete -f. Step 4 removes the bundle objects.
  4. Uninstall the chart: 
    helm uninstall liqo-consumer -n liqo --wait --timeout 10m
  5. Check what is left (optional but recommended): 
    kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations | grep liqo
kubectl get clusterroles,clusterrolebindings | grep liqo
  6. Delete the namespace (optional). Helm leaves it behind: 
    kubectl delete namespace liqo
  7. Delete the pool if you do not need it (section 5).

Notes:

  • helm uninstall alone also removes the Liqo objects, the bundle objects, and the liqo-tenant-* namespaces (chart 0.1.1 and later), even when the pool is unreachable. It cannot revoke the peering on the Multicloud side, so always run step 3.
  • Uninstall before unpeering can leave stuck objects on the pool side.
  • Do not delete the liqo namespace before helm uninstall finishes. This strands cluster-scoped objects.

What may be left behind:

Item Why What to do
The liqo namespace Helm does not delete a namespace it created with --create-namespace. Delete it (step 6).
Liqo CRDs Helm never deletes CRDs. Keep them, or delete them if no other Liqo installation runs in the cluster: kubectl get crd -o name | grep liqo.io | xargs -r kubectl delete.
Webhook configurations named liqo-webhook after a timeout The uninstall timed out. A leftover webhook whose Service is gone can reject API requests. kubectl delete mutatingwebhookconfiguration,validatingwebhookconfiguration liqo-webhook
Liqo ClusterRole and ClusterRoleBinding objects The uninstall timed out. List them with -l app.kubernetes.io/part-of=liqo, check the list, then delete.
Uninstall helper objects (ServiceAccounts, ClusterRoles, Jobs) They stay when a hook fails or when you use --no-hooks. They keep broad permissions. kubectl delete clusterrole,clusterrolebinding -l app.kubernetes.io/name=multicloud-uninstall and kubectl -n liqo delete serviceaccount,role,rolebinding,job -l app.kubernetes.io/name=multicloud-uninstall
The virtual node, if it is stuck Liqo removed a finalizer by force. kubectl delete node -l liqo.io/virtual-node
A failed uninstall job log Failed hook Jobs are kept. Read the log: kubectl -n liqo logs job/mc-uninstall-pre. Look for a line PREFLIGHT FAILED.

If the uninstall hook image cannot be pulled (restricted egress), helm uninstall stops at the Job deadline. Mirror the image, or run helm upgrade --reset-then-reuse-values --set uninstall.enabled=false first and clean up by hand with the commands above.

7. Troubleshooting

7.1 Check order

When something fails, check in this order:

kubectl -n liqo get pods                                  # all Liqo pods and multicloud-bridge-<slug> must be Running
kubectl get nodes -l liqo.io/type=virtual-node            # virtual node must be Ready
curl -fsS -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/<name>"                   # pool READY, not HIBERNATED
curl -fsS -H "X-API-Key: $KEY" "$API/api/v1/extend/pools/<name>/peerings/<id>"     # peering status

7.2 Peering statuses and messages

status Meaning What to do
PENDING The pool is not ready, or Multicloud is creating the peering. message starts with the blocking gate: TCP_NOT_READY, LIQO_NOT_SYNCED, TENANT_RBAC_NOT_READY, or LIQO_UNAVAILABLE. Wait. It continues by itself. It becomes FAILED after 20 minutes.
READY The bundle is stored (until the first fetch). Fetch and apply the bundle.
SUSPENDED The pool is hibernating, hibernated, or waking. Wake the pool, or wait.
FAILED Final. reason and message say why. Create a new peering.
REVOKING Cleanup is running. Wait.
REVOKED Cleanup finished. None.

Reasons:

reason Meaning What to do
POOL_NOT_READY The pool did not pass all gates in 20 minutes. Check the pool status. Create a new peering when the pool is READY.
LIQO_UNAVAILABLE The Liqo part of the pool was not usable. With one pool worker, a spot reclaim can cause this. Wait. It ends FAILED after 20 minutes. Then create a new peering.
POOL_HIBERNATED The pool slept while the peering was PENDING. No bundle was issued. Wake the pool and create a new peering.
POOL_DISABLED The pool was deleted. The bundle can never work again. Delete the old bundle objects. Create a new pool and peering.
POOL_APISERVER_UNREACHABLE, IDENTITY_MINT_FAILED Temporary errors. Multicloud retries until the timeout. Wait.
CREDENTIALS_ROTATION You rotated the credentials of the pool. The peering is PENDING while Multicloud issues a new bundle. Wait for READY, fetch the new bundle once, and apply it (section 5, "Rotate the credentials").
FEATURE_DISABLED, NOT_A_POOL, GRANT_DISABLED, APISERVER_ENDPOINT_UNCONFIGURED, INTERNAL A platform-side problem. Contact Multicloud support with the peering id.

API errors:

HTTP Code Cause Action
401 The X-API-Key header is missing, malformed, or unknown. Send a valid key.
402 The organization has not passed the payment gate, or is suspended. Resolve billing.
403 The key lacks the extend scope. Use a key with the scope.
404 Unknown pool or peering, or one that belongs to another organization. Check the name.
409 POOL_HIBERNATED A peering create on a sleeping pool. Wake the pool.
409 POOL_DISABLED A peering create on a deleted pool. Create a new pool.
409 PEERING_LIMIT The pool has 10 peerings. Revoke one.
409 POOL_HAS_PEERINGS A pool delete with open peerings. Revoke them, or use ?cascade=true.
409 POOL_DELETED Hibernate or wake of a deleted pool. None.
422 Invalid body or unknown field. Fix the request.
429 A rate limit, or the limit of 3 pools. Wait, or delete a pool.
503 A concurrent update, or the bundle is temporarily unavailable. Retry.

Pool and peering creation allow 5 calls per minute. Other calls allow 60 per minute.

7.3 The virtual node is NotReady or missing

Check Cause and action
Pool status is HIBERNATED or WAKING Expected. Wake the pool and wait.
Pool is READY, the node is NotReady kubectl -n liqo get pods. If the bridge pod multicloud-bridge-<slug> or a Liqo pod is not Running, check why (image pull, node capacity).
The node disappeared GKE and other clouds can delete a NotReady node that is not a VM. Liqo creates it again when the connection returns. Chart 0.1.8 and later does this automatically. A peering made with chart 0.1.7 or older needs a restart of the virtual kubelet pod (vk-<slug> in the namespace liqo-tenant-<slug>).
Cluster has routes-based networking The node has the taint node.kubernetes.io/network-unavailable. Use a VPC-native cluster.
The Cloudflare path is blocked The cluster needs outbound access on port 443. Check egress rules and proxies.

Do not create a new peering for a lost virtual node. Create a new peering only if the bundle objects are lost and you have no copy of the bundle. Revoke the old peering first.

Chart 0.2.2 and later gives the virtual kubelet pod the priority of the system pods (mc-virtual-kubelet-priority, Kubernetes 1.36 or later). On an older cluster, or with a chart before 0.2.2, a full cluster can make the virtual node disappear and return every 1 to 2 minutes. You see this:

  • The pod vk-<slug> shows the event Preempted.
  • kube-dns and konnectivity-agent pods show FailedScheduling with Insufficient cpu and 1 node(s) had untolerated taint(s).

Free about 115m of CPU requests on a real node, or turn on node autoscaling. Upgrade the chart to 0.2.2 or later on Kubernetes 1.36 or later.

7.4 A pod is Pending

Event or symptom Cause Action
Event: "untolerated taint" for the virtual node The toleration is missing, misspelled, or has the wrong slug. Copy the slug from kubectl get nodes -L multicloud.io/pool.
The pod never lands on the pool, although it has the toleration No nodeSelector or affinity. Local nodes have room. This is "either" mode. Add the nodeSelector to force the pool.
The pod is Pending on the virtual node and the pool has no free worker The pool adds a worker. This takes about 5.5 minutes. Wait. Check max_servers and max_cost_per_hour in the pool status.
Pending for longer, cost cap reached The pool shows READY, but the cost of active workers has reached max_cost_per_hour. Wait for workers to become idle, or create a pool with a higher cap.
Pending with an event about quota limit The declared requests exceed the grant, or the pod requests a resource the grant does not list (GPU, hugepages). Lower the requests. A GPU pod cannot run on the pool today.
Pending after a wake Liqo retries with a growing delay. The CronJob mc-reflection-kick in the liqo namespace speeds this up on Kubernetes 1.30 or later. Wait about 30 seconds after the pool is ready. On older clusters the delay can be minutes.
All pods of a namespace are Pending after a change Someone set podOffloadingStrategy: Remote and not every pod has the toleration. Set LocalAndRemote, or add the toleration to every pod.
NamespaceOffloading stays NoClusterSelected The object has no clusterSelector. Use the manifest in section 3.7.
The pod is Pending after the pool was recreated The new pool has a new slug. Update the toleration and nodeSelector.

To follow a wake or an offload in detail, read the virtual kubelet log, not Events:

kubectl -n liqo-tenant-<slug> logs deploy/vk-<slug> --tail=100

The event recorder drops events after 25 per pod.

7.5 DaemonSets show desired = current + 1

Node-level DaemonSets do not run on the virtual node. Examples are GKE system agents such as gmp-system/collector, fluentbit-gke, gke-metrics-agent, and pdcsi-node, and on EKS aws-node and kube-proxy. These DaemonSets tolerate every taint, so the DaemonSet controller tries to create a pod for the virtual node. A pod cannot run there, because the Liqo virtual kubelet refuses DaemonSet pods.

Consumer chart 0.2.1 and later blocks these pods with a ValidatingAdmissionPolicy named mc-no-daemonset-on-virtual-node. You see this:

  • No Pending or failing pod exists for the virtual node.
  • kubectl get ds shows DESIRED one higher than CURRENT and READY, for example 2 / 1 / 1.
  • The DaemonSet has FailedCreate warning events with the text "DaemonSet pods are not placed on a Liqo virtual node".

All three are expected. No action is needed. Details:

  • The policy needs Kubernetes 1.30 or later. On older clusters the chart does not install it, and the DaemonSet pod for the virtual node stays Pending. Delete it by hand or upgrade the cluster.
  • The policy stops new pods only. If a DaemonSet pod already sits on the virtual node after an upgrade, delete it once: kubectl delete pod <name>.
  • To turn the policy off: --set daemonSetAdmission.enabled=false. Then the Pending pods return.
  • Check the policy: kubectl get validatingadmissionpolicy mc-no-daemonset-on-virtual-node.

7.6 The uninstall fails

See section 6, "What may be left behind".

8. Security

What the bundle contains

The bundle is one multi-document YAML file. It holds credentials.

Object Content
Pre-flight Job Checks that the cluster id of your install equals consumer_cluster_id.
Bridge (named multicloud-bridge-<slug>) A Secret with one Cloudflare Access service token (client id and client secret) for the host of your pool only. A cloudflared Deployment. A Service. Images are pinned by digest.
Liqo tenant Namespace The namespace for this peering.
ResourceQuota mc-critical-pods In that namespace. Allows up to 10 pods with a system critical priority class. GKE needs it for the virtual kubelet pod (chart 0.2.2 and later). It grants no extra access.
ControlPlane kubeconfig Secret A ServiceAccount token for your tenant on the pool, and the pool API server address. The kubeconfig verifies the server certificate. It never sets insecure-skip-tls-verify.
ResourceSlice The capacity that you request.
VirtualNode The virtual node, with the taint and label multicloud.io/pool=<slug> and disableNetworkCheck: true.

Handle the file like a secret:

  • Save it with mode 600. Do not commit it to a repository. Do not paste it in a ticket or chat.
  • Anyone with the file can connect to your pool API server as your tenant until you revoke the peering.
  • The API shows the bundle once and then deletes it from the Multicloud database.
  • If the file leaks, revoke the peering at once (section 6, step 3) and create a new one.
  • Multicloud never logs the bundle, the token, or the client secret.
  • The credentials last one year. Multicloud refreshes the Access token and the pool renews the virtual kubelet certificate before they expire. You do not need to act.

What the pool can see

Liqo copies the objects that an offloaded pod needs to the pool: the pod, and the Secrets, ConfigMaps, and image pull secrets that the pod references. Only pods that opt in are copied. Your other workloads, Secrets, and namespaces stay in your cluster. Treat the pool as a place where you run workloads, and do not offload a pod that uses a Secret you must keep out of a third-party cluster. Each customer has its own pool, its own control plane, and its own datastore. The pool is not shared with other customers. The database that holds pool state uses storage encryption. The pool does not yet add its own envelope encryption for Secrets inside the datastore.

How tenant access on the pool is limited:

  • Your tenant ServiceAccount on the pool can write only your ResourceSlice, Renew, and NamespaceMap objects. It cannot write its own granted capacity.
  • The pool signs your virtual kubelet certificate only for the exact subject it expects, with no extra groups.
  • Each peering has its own Cloudflare Access policy and service token. The token works for your pool host only. Revoking the peering removes it.
  • The pool does not connect into your cluster. Your cluster connects out to the pool.

What the chart installs

The liqo-consumer chart installs, with networking turned off:

  • Four Deployments in the namespace liqo: Liqo controller-manager, webhook, crd-replicator, and proxy. Liqo's own ClusterRoles and ClusterRoleBindings give them access. Review them with helm template.
  • The Liqo CRDs and a liqo RuntimeClass.
  • Webhook configurations with failurePolicy: Ignore, so a missing webhook never blocks your API. While the webhook is down, pods in offloaded namespaces are admitted without Liqo changes.
  • The virtual kubelet image docker.io/multicloud/liqo-virtual-kubelet (a patched Liqo build).
  • No PVCs and no LoadBalancer Services. No network gateway.

Extra objects from Multicloud, and their RBAC:

Object What it needs Limit
mc-no-daemonset-on-virtual-node policy and binding (Kubernetes 1.30+) Admission only. No RBAC. Denies only DaemonSet pods that the controller pins to a virtual node. failurePolicy: Ignore.
CronJob mc-reflection-kick (Kubernetes 1.30+) Read nodes. Read pods in all namespaces (the full pod objects, not Secrets). Patch pods. An admission policy lets this ServiceAccount change one annotation, multicloud.io/reflection-touch, on pods of the virtual node, and nothing else. It makes no network call except to the cluster API. Turn it off with --set reflectionKick.enabled=false.
Uninstall hooks (pre-delete and post-delete Jobs) Pre-delete: get, list, patch, and delete Secrets in all namespaces, and scale Liqo Deployments. Post-delete: delete namespaces, and delete Deployments, Services, Secrets, and Jobs in the liqo namespace. They run only during helm uninstall. The scripts select by label and delete only Liqo and peering objects. The objects are removed when the hooks succeed. The Jobs run as non-root with a read-only root file system and pass the Pod Security restricted level.

The chart pulls images from docker.io and ghcr.io/liqotech. For a restricted network, mirror the images and set liqo.imagePullSecrets and uninstall.image.

9. Limits

Limit Value
Enabled Extend pools per organization 3. A hibernated pool counts. A delete frees a slot.
Active Extend API keys per organization 10
Peerings per pool that hold a slot 10
Peerings per consumer cluster and pool 1. A second bundle for the same pool replaces the first bridge.
max_servers 1 to 50
max_cost_per_hour Above 0, at most 50. Default 0.2.
Bundle One fetch.
Pending timeout of a peering 20 minutes
Unsupported Cross-cluster Services. GPU and hugepage requests on the pool. Routes-based GKE.
Untested EKS, AKS, GKE Autopilot.