How to install docker and Kubernetes cluster on RedHat 8 (CentOS 8)

(2020/02/11 Update: After installation, I keep facing pod network issue which is like deployed pod is unable to reach external network or pods deployed in different workers are unable to ping each other even I can see all nodes (master, worker1 and worker2) are ready via kubectl get nodes. After checking through the Kubernetes.io official website, I observed the nfstables backend is not compatible with the current kubeadm packages. Please refer the following link in “Ensure iptables tooling does not use the nfstables backend”. Please see the we link below:

Installing kubeadm

The workaround will be to switch iptables to ‘legacy’ mode, but unfortunately, RHEL8(CentOS8) doesn’t support switching to legacy mode. The suggested OS on Kubernetes website is Ubuntu 16.04+, Debian 9+, CentOS 7, RHEL 7 and Fedora 25+. If you still want to install Kubernetes cluster on your bare metal machines, you can still refer this article but please change your OS to supported ones. )

Recently due to project requirement, I have this opportunity to install Docker CE and Kubernetes cluster on RedHat 8 (CentOS 8). RedHat 8 (CentOS 8) comes with its own tools, buildah, podman and openshift. In this article, I will go through how to add the external repository for Docker CE and Kubernetes. And then I will instruct how to add pod hosts and install overlay network to construct our Kubernetes cluster on bare metal machines.

\x01 Install Docker CE on RedHat 8 (CentOs 8)

First of all, we have to add the external repository to obtain Docker CE.

# dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo

And we can check if the external Docker CE repository has been added correctly.

# dnf repolist -v
Repo-id : docker-ce-stable
Repo-name : Docker CE Stable — x86_64
Repo-revision: 1573753657
Repo-updated : Thu 14 Nov 2019 12:47:37 PM EST
Repo-pkgs : 57
Repo-size : 1.2 G
Repo-baseurl : https://download.docker.com/linux/centos/7/x86_64/stable
Repo-expire : 172,800 second(s) (last: Sun 08 Dec 2019 11:56:28 PM EST)
Repo-filename: /etc/yum.repos.d/docker-ce.repo

When everything is ready, we can install Docker CE now.

# dnf -y install docker-ce --nobest
# yum install containerd.io

Once installation completed, we can enable Docker daemon and let it start up every time when the machine reboots. And use systemctl command to check if Docker daemon is running well.

# systemctl enable --now docker
# systemctl status docker

Now we have to add one normal user who has the privilege to run docker command and this user will be responsible for the coming Kubernetes cluster setup stuff.

$ adduser kube --shell /bin/bash
$ passwd kube

The last step for Docker CE installation is to disable firewalld on RedHat 8 (CentOS 8)

# systemctl disable firewalld

\x02 Install and Initialize Kubernetes on RedHat 8 (CentOS 8)

Just like Docker CE installation, we have to add external repository first for Kubernetes installation. Please edit the file /etc/yum.repos.d/kubernetes.repo and add the following content.

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

Once done, we can install the Kubernetes for now. After installation, we have to enable kubelet.

# dnf install -y kubelet kubeadm kubectl kubernetes-cni --disableexcludes=kubernetes
# systemctl enable --now kubelet

And add the following two lines into /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

Use the following command to reload the configuration

# sysctl --system

The last step of Kubernetes installation is disable swap. We can use the command to disable it one time or edit /etc/fstab to make this change permanent.

$ swapoff -a

Or comment out the line starts with /dev/mapper/cl-swap swap.

If you encounter no error on above steps, you can initialize your Kubernetes master node for now. Before you initialize our Kubernetes master node, we have to make sure our ip forward is enabled. Adding the line net.ipv4.ip_forward = 1 into /etc/sysctl.conf and type the sysctl -p to enable it. Once everything is ready, use the following command to initialize your Kubernetes master node.

# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=[ip address] --kubernetes-version stable-1.16

And sudo to the user: kube who is created for running docker and kubectl command and run the following command.

$ cd ~
$ mkdir .kube
$ sudo cp -i /etc/kubernetes/admin.conf .kube/config
$ sudo chown $(id -u):$(id -g) .kube/config

When you use kubeadm to init your Kubernetes master node, lots of output come up and once you see the following output, that means your initialization is completed successfully.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join [ip address]:6443 — token [token]\
— discovery-token-ca-cert-hash [sha256:hash]

Please note the line of kubeadm join, we will use this to let our worker nodes to join the cluster.

\x03 Install Cluster Networking — Overlay Network

There are lots of CNI network add-on we could chose for Kubernetes cluster networking, like Flannel, Calico and Weave Net. We can refer the Kubernetes official web site for the installation. Here comes the example for Flannel. Please use the following command to install Flannel CNI network add-on.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

\x04 Join Pod Hosts — Add Worker Nodes

Now, we can jump to our worker hosts and run the following command which is copied from the output of kubeadm init on master node to let our worker nodes to join our Kubernetes cluser.

# kubeadm join [ip address]:6443 --token [token]\ --discovery-token-ca-cert-hash [sha256:hash]

Once you see the following output, it means your worker nodes have been added into cluster successfully.

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.

You can jump back to the master node and run kubectl get nodes to make sure all your workers are under running status.

If you follow all the steps and comes with me here, your Kubernetes cluster is successfully up and running. Congratulations !!!

\x05 Drain Pod Hosts — Delete Nodes

You might encounter some situation for some reason, you want to tear down your worker nodes. Please use the following command on control-plane node (master node) to drain your worker nodes first.

# kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
# kubectl delete node <node name>

Then, on the worker node being removed, reset all kubeadm installed state and please don’t forget reset your iptables as well.

# kubeadm reset
# iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

Now, your worker node has been removed successfully.

Hope this article helps in any way. Happy Learning !!!