How to Setup Kubernetes Cluster with Kubeadm on Debian 11
Kubernetes or k8s is an open-source platform for container orchestration that automates deployments, management, and scaling of containerized applications. Kubernetes is a container orchestration created by Google, and now become an open-source project and become standard for modern application deployment and computing platforms.
Kubernetes is the solution for the modern container deployment era. It provides service discovery and load-balancing, storage orchestration, automated rollout and rollback, self-healing service, secret and configuration management. Kubernetes enables cost-effective cloud-native development.
In this tutorial, you will set up the Kubernetes Cluster by:
- Setting up systems, which includes - setup /etc/hosts file, enabling kernel modules, and disabling SWAP.
- Setting up UFW firewall by adding some ports that are required for Kubernetes and CNI Plugin (Calico).
- Installing CRI-O as the container runtime for Kubernetes.
- Installing Kubernetes packages such as kubelet, kubeadm, and kubectl.
- Initializing one control-plane node and adding two worker nodes.
Prerequisites
To complete this tutorial, you will need the following requirements:
- Three or more Debian 11 servers.
- A non-root user with root/administrator privileges.
Setting Up Systems
Before you start installing any packages for Kubernetes deployment, you will need to set up all of your systems as required for Kubernetes Deployment. This includes the following configurations:
- Setup correct /etc/hosts file: Each server hostname must be resolved to the correct IP address. This can be done in multiple ways, but the easiest and simple one is by using the /etc/hosts file on all servers.
- Setup UFW Firewall: For the production environment, it's always recommended to enable the firewall on both control-plane and worker nodes. You will set up a UFW firewall for the Kubernetes control plane, worker node, and the CNI plugin Calico.
- Enable kernel Modules: The Kubernetes required some kernel modules on the Linux system to be enabled. The kernel module "overlay" and "br_netfilter" is required to let iptables see bridged traffics.
- Disable SWAP: This is mandatory, you must disable SWAP on all Kubernetes nodes, both control-plane and worker nodes. Otherwise, the kubelet service will be running with issues.
Setup /etc/hosts file
In this first step, you will set up the system hostname and the /etc/hosts file on all of your servers. For this demonstration, we will use the following servers.
Hostname IP Address Used as
--------------------------------------------
k8s-master 192.168.5.10 control-plane
k8s-worker1 192.168.5.115 worker node
k8s-worker2 192.168.5.116 worker node
Run the following hostnamectl command below to set up the system hostname on each server.
For the control-plane node, run the following command to set up the system hostname to "k8s-master".
sudo hostnamectl set-hostname k8s-master
For Kubernetes worker nodes, run the following hostnamectl command.
# setup hostname k8s-worker1
sudo hostnamectl set-hostname k8s-worker1
# setup hostname k8s-worker2
sudo hostnamectl set-hostname k8s-worker2
Next, edit the /etc/hosts file on all servers using the following command.
sudo nano /etc/hosts
Add the following configuration to the file. Be sure each hostname is pointed to the correct IP address.
192.168.5.10 k8s-master
192.168.5.115 k8s-worker1
192.168.5.116 k8s-worker2
Save and close the file when you are finished.
Lastly, if you run the ping command against each hostname, you will be pointed to the correct IP address as defined on the /etc/hosts file.
ping k8s-master -c3
ping k8s-worker1
ping k8s-worker2 -c3
Setup UFW Firewall
Kubernetes required some ports to be open on all of your systems. On the default Ubuntu system, the UFW firewall is used as the default firewall. You will install the UFW firewall on all your Debian systems and add some UFW rules for the Kubernetes deployment.
For the Kubernetes control-plane, you need to open the following ports:
Protocol Direction Port Range Purpose Used By
-----------------------------------------------
TCP Inbound 6443 Kubernetes API server All
TCP Inbound 2379-2380 etcd server client API kube-apiserver, etcd
TCP Inbound 10250 Kubelet API Self, Control plane
TCP Inbound 10259 kube-scheduler Self
TCP Inbound 10257 kube-controller-manager Self
For the Kubernetes worker nodes, you need to open the following ports:
Protocol Direction Port Range Purpose Used By
--------------------------------------------------
TCP Inbound 10250 Kubelet API Self, Control plane
TCP Inbound 30000-32767 NodePort Services† All
In this example, we will use Calico as the CNI (Container Network Interface) plugin. So, you will open some additional ports below:
Protocol Direction Port Range Purpose Used By
-------------------------------------------------------
TCP Bidirectional 179 Calico networking (BGP)
UDP Bidirectional 4789 Calico networking with VXLAN enabled
TCP Incoming 2379 etcd datastore
UDP Bidirectional 4789 flannel networking (VXLAN)
Install the UFW package to your Debian servers using the following apt command. Input Y to confirm the installation and press ENTER, and the installation will begin.
sudo apt install ufw
Next, add the OpenSSH application to your firewall using the below command. Then, enable the UFW firewall. When prompted for the confirmation, input "y" to enable and run the UFW firewall.
sudo ufw allow "OpenSSH"
sudo ufw enable
On the control-plane node "k8s-master", run the following ufw command to open ports. Then, check and verify UFW rules.
Firewall rules for Kubernetes control plane.
sudo ufw allow 6443/tcp
sudo ufw allow 2379:2380/tcp
sudo ufw allow 10250/tcp
sudo ufw allow 10259/tcp
sudo ufw allow 10257/tcp
Firewall rules for Calico CNI plugin.
sudo ufw allow 179/tcp
sudo ufw allow 4789/udp
sudo ufw allow 4789/tcp
sudo ufw allow 2379/tcp
sudo ufw status
On worker nodes "k8s-worker1" and "k8s-worker2", run the following ufw command to open some ports. Then, check the UFW firewall rules.
Firewall rules for Kubernetes worker nodes.
sudo ufw allow 10250/tcp
sudo ufw allow 30000:32767/tcp
Firewall rules for Calico on Kubernetes worker nodes.
sudo ufw allow 179/tcp
sudo ufw allow 4789/udp
sudo ufw allow 4789/tcp
sudo ufw allow 2379/tcp
sudo ufw status
Enable Kernel Modules and Disable SWAP
The Kubernetes required the kernel modules "overlay" and "br_netfilter" to be enabled on all servers. This will let the iptbales see bridged traffics. Also, you will need to enable the port forwarding and disable SWAP.
Run the following command to enable the kernel modules "overlay" and "br_netfilter".
sudo modprobe overlay
sudo modprobe br_netfilter
To make it permanent, create the configuration file to "/etc/modules-load.d/k8s.conf" using the below command. This will allow Linux systems to enable kernel modules during the system boot.
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
Next, create the systemctl params required using the following command.
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
To apply the new sysctl configuration without reboot, use the sysctl command below. You should get the list of default sysctl params on your system and be sure you get sysctl params that you just added in the file "k8s.conf".
sudo sysctl --system
To disable SWAP, you will need to comment on the SWAP configuration on the "/etc/fstab" file. This can be done by using the single command via sed (stream editor) or manually editing the /etc/fstab file.
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
or
sudo nano /etc/fstab
Next, turn off the SWAP on the current session using the below command. Then, verify the SWAP is off using the "free -m" command. You should see the SWAP has "0" values, which means it's now disabled.
sudo swapoff -a
free -m
Installing Container Runtime: CRI-O
To set up Kubernetes Cluster, you must install the container runtime on all servers so that Pods can run. Multiple container runtimes can be used for Kubernetes deployments such as containerd, CRI-O, Mirantis Container Runtime, and Docker Engine (via cri-dockerd).
In this demonstration, we will use the "CRI-O" as the container for our Kubernetes deployment. So, you will install CRI-O on all servers, control-plane, and worker nodes.
Before installing CRI-O, run the apt command below to install the basic package "gnupg2" and "apt-transport-https". Input Y to confirm the installation and press ENTER.
sudo apt install gnupg2 apt-transport-https
Now create a new environment variable for the CRI-O installation. The variable "$OS" with the value "Debian_11" and the variable "$VERSION" with the value "1.24". In this example, we will install CRI-O container v1.24 (current version) for "Debian_11" systems.
export OS=Debian_11
export VERSION=1.24
Run the following command to add the CRI-O repository for Debian 11 system.
echo "deb [signed-by=/usr/share/keyrings/libcontainers-archive-keyring.gpg] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb [signed-by=/usr/share/keyrings/libcontainers-crio-archive-keyring.gpg] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
Run the following command to add the GPG key for the CRI-O repository.
mkdir -p /usr/share/keyrings
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | gpg --dearmor -o /usr/share/keyrings/libcontainers-archive-keyring.gpg
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | gpg --dearmor -o /usr/share/keyrings/libcontainers-crio-archive-keyring.gpg
Now update system repositories and refresh the package index using the below command. You should see the CRI-O repository is added to Debian 11 servers.
sudo apt update
To install the CRI-O container runtime, run the following apt command. Input Y to confirm the installation and press ENTER, and the CRI-O installation will begin.
sudo apt install cri-o cri-o-runc cri-tools
After installation is finished, edit the CRI-O configuration "/etc/crio/crio.conf" using the below command.
sudo nano /etc/crio/crio.conf
On the "[crio.network]" section, uncomment the option "network_dir" and the "plugin_dir".
# The crio.network table containers settings pertaining to the management of
# CNI plugins.
[crio.network]
# The default CNI network name to be selected. If not set or "", then
# CRI-O will pick-up the first one found in network_dir.
# cni_default_network = ""
# Path to the directory where CNI configuration files are located.
network_dir = "/etc/cni/net.d/"
# Paths to directories where CNI plugin binaries are located.
plugin_dirs = [
"/opt/cni/bin/",
]
When you are finished, save and close the file.
Next, edit the CRI-O bridge configuration "/etc/cni/net.d/100-crio-bridge.conf" using the below command.
sudo nano /etc/cni/net.d/100-crio-bridge.conf
Change the default subnet of IP address using your custom subnet. This subnet IP address will be used for your Pods on the Kubernetes cluster. Also, you will need to ensure the subnet IP address is matched with the IP address configuration on the CNI plugin.
In this example, we will use the subnet IP address "10.42.0.0/24" for Pods on the Kubernetes cluster.
...
"ranges": [
[{ "subnet": "10.42.0.0/24" }],
[{ "subnet": "1100:200::/24" }]
]
...
Save and close the file when you are done.
Next, run the following systemctl command to restart the CRI-O service and apply new changes.
sudo systemctl restart crio
Lastly, enable the CRI-O service to run at system boot. Then, check and verify the CRI-O service status. And you should see the CRI-O service is enabled, and the current status is running.
sudo systemctl enable crio
sudo systemctl status crio
Installing Kubernetes Packages
You have installed the CRI-O container runtime. Now you will install Kubernetes packages on all of your Debian systems. This includes the kubeadm for bootstrapping the Kubernetes cluster, kubelet the main component of the Kubernetes Cluster, and the kubectl the command-line utility for managing the Kubernetes cluster.
In this example, we will install Kubernetes packages using the repository provided by Kubernetes. So, you will add the Kubernetes repository to all of your Debian systems.
Run the following command to add the Kubernetes repository and GPG key.
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
Update and refresh your Ubuntu repository and package index.
sudo apt update
When the update is finished, install Kubernetes packages using the following apt command. Input Y to confirm the installation and press ENTER to continue, and the installation will begin.
sudo apt install kubelet kubeadm kubectl
After installation is finished, run the following command to pin the current version of Kubernetes packages. This will prevents Kubernetes packages to be updated automatically and prevent the version skew between Kubernetes packages.
sudo apt-mark hold kubelet kubeadm kubectl
