Self-Hosted Kubernetes (K3s) on Bare Metal: A Lightweight Alternative to Proxmox/VMware

Proxmox and VMware solve a different problem than Kubernetes does. They virtualize hardware into separate machines. Kubernetes orchestrates containers within machines. If what you actually need is to run and scale containerized workloads — not spin up isolated VMs — installing a hypervisor first and Kubernetes on top of it is often unnecessary overhead.

K3s, a lightweight, certified Kubernetes distribution, lets you skip the virtualization layer entirely and run Kubernetes directly on bare metal.

K3s vs Proxmox vs VMware: What Problem Are You Actually Solving?

These tools get compared constantly, but they're not really substitutes for each other:

• Proxmox / VMware (Hypervisors): They carve one physical server into multiple isolated virtual machines, each with its own OS. Best when you need strict isolation between workloads, need to run different operating systems side by side, or want traditional VM-based infrastructure.
• Kubernetes / K3s (Container Orchestrator): It schedules, scales, and manages containerized applications across one or more machines, with all containers sharing the host OS kernel. Best when your workloads are already containerized and you want automated scaling, self-healing, and rolling updates.

The common mistake is running K3s inside a Proxmox VM or LXC container by default, adding a virtualization layer between Kubernetes and the hardware for no real benefit — and LXC in particular causes real friction here, since features like persistent storage drivers often need kernel-level access that containers don't provide.

If your goal is to run Kubernetes and nothing else needs to be a separate VM, installing K3s straight onto bare metal removes that layer entirely: no hypervisor overhead, no nested virtualization tax, and one fewer thing that can misconfigure.

Why K3s Specifically (Not Full Kubernetes)

Standard Kubernetes (installed via kubeadm) is built for large-scale, multi-node production clusters and comes with a correspondingly heavy set of components. K3s, originally built by Rancher, strips this down into a single binary:

• Single binary install — the control plane, kubelet, and container runtime ship as one process.
• SQLite by default — no separate etcd cluster required for single-node or small setups (etcd is still available for multi-server HA).
• containerd bundled — no separate container runtime to install and manage.
• Batteries included — Traefik ingress controller, a lightweight load balancer (Klipper), and Flannel CNI come pre-configured.
• Small footprint — runs comfortably on hardware far smaller than a standard kubeadm cluster needs, including Raspberry Pi-class devices.

This makes it a practical fit for a single bare-metal server, a small homelab cluster, or an edge deployment — not just resource-constrained IoT devices, which is where it's often (incorrectly) assumed to stop being useful.

Prerequisites

• One or more bare-metal servers running Ubuntu 22.04/24.04 or Debian 11/12
• Root or sudo access
• At least 2 CPU cores and 2GB RAM per node (more if running GPU-based or memory-heavy workloads)
• Nodes able to reach each other over the network if building a multi-node cluster
• Swap disabled (sudo swapoff -a and remove the swap entry from /etc/fstab) — Kubernetes expects swap off by default

Step 1: Install K3s on the First Node (Control Plane)

K3s ships a single install script that handles the whole setup:

curl -sfL https://get.k3s.io | sh -
                            

This installs K3s as a systemd service and starts it immediately. Confirm it's running:

sudo systemctl status k3s
sudo k3s kubectl get nodes
                            

You should see your node listed with status Ready.

Step 2: Retrieve the Node Token for Joining Workers

If you're building a multi-node cluster, other machines need a token to join:

sudo cat /var/lib/rancher/k3s/server/node-token
                            

Copy this — you'll need it for each additional node.

Step 3: Join Additional Worker Nodes

On each additional bare-metal server that should join the cluster as a worker:

curl -sfL https://get.k3s.io | K3S_URL=https://<control-plane-ip>:6443 K3S_TOKEN=<token-from-step-2> sh -
                            

Back on the control plane node, confirm the new worker appears:

sudo k3s kubectl get nodes
                            

Step 4: Set Up kubectl Access Without sudo

By default, K3s's kubeconfig is only readable by root. To use kubectl as a regular user (and from your own workstation, if managing remotely):

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config
                            

Add the export line to your shell profile (e.g., ~/.bashrc) so it persists across sessions.

Step 5: Deploy a Test Workload

Confirm the cluster actually schedules and runs containers correctly:

kubectl create deployment nginx-test --image=nginx
kubectl expose deployment nginx-test --port=80 --type=NodePort
kubectl get svc nginx-test
                            

The NodePort output shows which port on any cluster node now serves that container. Hit it with curl from another machine on the network to confirm end-to-end connectivity.

Step 6: GPU Workloads and Containerized Inference

For AI/ML workloads — running an LLM inference server across multiple GPUs, for example — K3s needs the NVIDIA device plugin to expose GPUs to pods, since Kubernetes doesn't schedule GPU resources natively out of the box:

kubectl create namespace nvidia-device-plugin
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yml
                            

This lets you request GPUs in a pod spec the same way you'd request CPU or memory:

resources:
  limits:
    nvidia.com/gpu: 2
                            

This is the same underlying mechanism used when containerizing large model inference workloads (such as a multi-GPU DeepSeek-V3 deployment) — the model-serving container doesn't need to know it's running under K3s specifically, only that GPUs are exposed as a schedulable resource. Running that kind of inference workload under K3s instead of a bare Docker Compose setup gets you rolling updates, automatic restarts on failure, and the ability to scale replicas across multiple physical GPU servers without re-architecting anything.

Step 7: Persistent Storage

Containers are ephemeral by design — anything written inside one disappears when it restarts, unless backed by a persistent volume. On bare metal (without a Proxmox/Ceph backend to lean on), the common lightweight option is Longhorn, a distributed block storage system built for Kubernetes:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
                            

For a single-node setup, local-path-provisioner (bundled with K3s by default) is often sufficient and requires no extra install — it's the right choice until you actually need data replicated across multiple nodes.

High Availability: Beyond a Single Control-Plane Node

A single control-plane node is a single point of failure — fine for testing, not for anything you depend on. For HA, K3s supports an embedded etcd cluster across an odd number of server nodes (3 or 5):

curl -sfL https://get.k3s.io | sh -s - server --cluster-init
                            

Additional server nodes then join with --server https://<first-node-ip>:6443 instead of running as plain workers.

Common Issues

When Proxmox/VMware Still Makes More Sense

K3s on bare metal isn't a universal replacement. If you genuinely need to run multiple different operating systems on one physical box, need strict VM-level isolation for compliance reasons, or your workloads simply aren't containerized and rewriting them isn't practical, a hypervisor is still the right tool.

K3s earns its place when the workloads are already (or can reasonably be) containers, and the extra virtualization layer would only add overhead without adding isolation you actually need.