05.Cluster Maintenance - Cluster Upgrade Process

 

Cluster Maintenance

1) OS Upgrades
2) Cluster Upgrade Process
3) Backup and Restore Methods

 

01. This lab tests your skills on upgrading a kubernetes cluster.

We have a production cluster with applications running on it. Let us explore the setup first.

What is the current version of the cluster?

 

ask : 

v1.19.0

root@controlplane:~# kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
controlplane   Ready    master   51m   v1.19.0
node01         Ready    <none>   50m   v1.19.0

02. How many nodes are part of this cluster?

Including master and worker nodes

ask : 2

root@controlplane:~# kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
controlplane   Ready    master   51m   v1.19.0
node01         Ready    <none>   50m   v1.19.0

 

03. How many nodes can host workloads in this cluster?

Inspect the applications and taints set on the nodes.

 

ask : 2

# hint
Check the taints on both controlplane and node01.
If none exists, then both nodes can host workloads.

# solution
By running the kubectl describe node command, 
we can see that neither nodes have taints.

root@controlplane:~# kubectl describe node controlplane | grep -i taint
Taints:             <none>
root@controlplane:~# kubectl describe node node01 | grep -i taint
Taints:             <none>

This means that both nodes have the ability to schedule workloads on them.

 

04. How many applications are hosted on the cluster?

Count the number of deployments.

ask : 1

 

root@controlplane:~# kubectl get deployments
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
blue   5/5     5            5           9m42s

 

05. What nodes are the pods hosted on?

 

ask : controlplane,node01

root@controlplane:~# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
blue-746c87566d-2z56n   1/1     Running   0          10m   10.244.0.4   controlplane   <none>           <none>
blue-746c87566d-9v57m   1/1     Running   0          10m   10.244.1.3   node01         <none>           <none>
blue-746c87566d-j99dm   1/1     Running   0          10m   10.244.0.5   controlplane   <none>           <none>
blue-746c87566d-w9sv7   1/1     Running   0          10m   10.244.1.2   node01         <none>           <none>
blue-746c87566d-xmn2q   1/1     Running   0          10m   10.244.1.5   node01         <none>           <none>
simple-webapp-1         1/1     Running   0          10m   10.244.1.4   node01         <none>           <none>

 

06. You are tasked to upgrade the cluster. User's accessing the applications must not be impacted.

And you cannot provision new VMs. What strategy would you use to upgrade the cluster?

 

클러스터를 업그레이드해야 합니다. 사용자의 애플리케이션 액세스는 영향을 받지 않아야 합니다. 

그리고 새 VM을 프로비저닝할 수 없습니다. 클러스터를 업그레이드하기 위해 어떤 전략을 사용하시겠습니까?

 

1) Add new nodes with newer versions while taking down existing nodes

    기존 노드를 제거하면서 최신 버전으로 새 노드 추가

2) Upgrade one node at a time while moving the workloads to the other (정답)

    워크로드를 다른 노드로 이동하면서 한 번에 하나의 노드 업그레이드

3) Users will be impacted since there is only one worker node

    작업자 노드가 하나뿐이므로 사용자가 영향을 받습니다. 

4) Upgrade all nodes at once

    모든 노드를 한 번에 업그레이드

 

# hint
In order to ensure minimum downtime, upgrade the cluster one node at a time,
while moving the workloads to another node. 
In the upcoming tasks you will get to practice how to do that.

가동 중지 시간을 최소화하려면 클러스터를 한 번에 한 노드씩 업그레이드하고,
워크로드를 다른 노드로 이동하는 동안
다음 작업에서 이를 수행하는 방법을 연습하게 됩니다.

 

07. What is the latest stable version available for upgrade?

Use the kubeadm tool

 

ask :) v1.19.16

# hint
Run the  kubeadm upgrade plan command


root@controlplane:~# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.19.0
[upgrade/versions] kubeadm version: v1.19.0
I0121 08:06:45.166819   25704 version.go:252] remote version is much newer: v1.23.2; falling back to: stable-1.19
[upgrade/versions] Latest stable version: v1.19.16
[upgrade/versions] Latest stable version: v1.19.16
[upgrade/versions] Latest version in the v1.19 series: v1.19.16
[upgrade/versions] Latest version in the v1.19 series: v1.19.16

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
kubelet     2 x v1.19.0   v1.19.16

Upgrade to the latest version in the v1.19 series:

COMPONENT                 CURRENT   AVAILABLE
kube-apiserver            v1.19.0   v1.19.16
kube-controller-manager   v1.19.0   v1.19.16
kube-scheduler            v1.19.0   v1.19.16
kube-proxy                v1.19.0   v1.19.16
CoreDNS                   1.7.0     1.7.0
etcd                      3.4.9-1   3.4.9-1

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.19.16

Note: Before you can perform this upgrade, you have to update kubeadm to v1.19.16.

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

 

***** 08. We will be upgrading the master node first.

Drain the master node of workloads and mark it UnSchedulable

• Master Node: SchedulingDisabled

# hint
Run the kubectl drain controlplane --ignore-daemonsets

# solution
There are daemonsets created in this cluster, especially in the kube-system namespace.
To ignore these objects and drain the node, 
we can make use of the --ignore-daemonsets flag.

root@controlplane:~# kubectl drain controlplane --ignore-daemonsets
node/controlplane cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-dddrc, kube-system/kube-proxy-p2c2l
evicting pod default/blue-746c87566d-lxnwv
evicting pod kube-system/coredns-f9fd979d6-dpsbl
evicting pod default/blue-746c87566d-js9fr
evicting pod kube-system/coredns-f9fd979d6-x8kzr
pod/coredns-f9fd979d6-dpsbl evicted
pod/blue-746c87566d-lxnwv evicted
pod/coredns-f9fd979d6-x8kzr evicted
pod/blue-746c87566d-js9fr evicted
node/controlplane evicted

 

 

09. Upgrade the controlplane components to exact version v1.20.0

Upgrade kubeadm tool (if not already), then the master components, and finally the kubelet.

Practice referring to the kubernetes documentation page.

 

Note: While upgrading kubelet,

if you hit dependency issue while running the apt-get upgrade kubelet command,

use the apt install kubelet=1.20.0-00 command instead

 

• controlplane Upgraded to v1.20.0
• controlplane Kubelet Upgraded to v1.20.0

hint Make sure that the correct version of kubeadm is installed and then proceed to upgrade the controlplane node. Once this is done, upgrade the kubelet on the node.

# On the controlplane node, run the command run the following commands:

apt update

# This will update the package lists from the software repository.

apt install kubeadm=1.20.0-00

# This will install the kubeadm version 1.20

kubeadm upgrade apply v1.20.0

#This will upgrade kubernetes controlplane. Note that this can take a few minutes.

apt install kubelet=1.20.0-00

# This will update the kubelet with the version 1.20.

# You may need to restart kubelet after it has been upgraded.

Run: systemctl restart kubelet

apt update && install -y kubeadm=1.20.0-00

 

# kubeadm

# replace x in 1.20.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.20.0-00 && \
apt-mark hold kubeadm


# since apt-get version 1.1 you can also use the following method
apt-get update && \
apt-get install -y --allow-change-held-packages kubeadm=1.20.0-00

sudo kubeadm upgrade apply v1.20.0

 

10. Mark the controlplane node as "Schedulable" again

• Master Node: Ready & Schedulable

root@controlplane:~# kubectl uncordon controlplane
node/controlplane uncordoned

 

11. Next is the worker node. Drain the worker node of the workloads and mark it UnSchedulable

• Worker node: Unschedulable

# worker node 는 마스터 node 가 아닐테니,, 
# kubectl get nodes 결과를 참고하여 node select.

root@controlplane:~# kubectl drain node01 --ignore-daemonsets
node/node01 cordoned
error: unable to drain node "node01", aborting command...

There are pending nodes to be drained:
 node01
error: cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): default/simple-webapp-1

 

12. Upgrade the worker node to the exact version v1.20.0

• Worker Node Upgraded to v1.20.0
• Worker Node Ready

# hint
Make sure that the correct version of kubeadm is installed 

and then proceed to upgrade the node01 node. 

Once this is done, upgrade the kubelet on the node.

 

# solution


# step 01
# On the node01 node, run the command run the following commands:
# If you are on the master node, run "ssh node01" to go to node01

ssh node01

# step 02
# This will update the package lists from the software repository.
apt update


# step 03
# This will install the kubeadm version 1.20
apt install kubeadm=1.20.0-00

# step 04
# This will upgrade the node01 configuration.
kubeadm upgrade node

# step 05
# This will update the kubelet with the version 1.20.
apt install kubelet=1.20.0-00

# step 06
# You may need to restart kubelet after it has been upgraded.

Run: systemctl restart kubelet


Type exit or enter CTL + d to go back to the controlplane node.

# step 01. node01 에 접속
ssh node01


# step 02. kubeadm 
# replace x in 1.20.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.20.0-00 && \
apt-mark hold kubeadm

# step 03.
# since apt-get version 1.1 you can also use the following method

apt-get update && \
apt-get install -y --allow-change-held-packages kubeadm=1.20.0-00

# step 04.
# Upgrade kubelet and kubectl 
# replace x in 1.20.x-00 with the latest patch version

sudo kubeadm upgrade apply v1.20.0


apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.20.0-00 kubectl=1.20.0-00 && \
apt-mark hold kubelet kubectl
 
# step 05. 
# since apt-get version 1.1 you can also use the following method

apt-get update && \
apt-get install -y --allow-change-held-packages kubelet=1.20.0-00 kubectl=1.20.0-00

sudo systemctl daemon-reload
sudo systemctl restart kubelet

 

13. Remove the restriction and mark the worker node as schedulable again.

• Worker Node: Schedulable

kubectl uncordon node01

 

---

kubeadm: 클러스터를 부트스트랩하는 명령이다. kubelet: 클러스터의 모든 머신에서 실행되는 파드와 컨테이너 시작과 같은 작업을 수행하는 컴포넌트이다. kubectl: 클러스터와 통신하기 위한 커맨드 라인 유틸리티이다.

---

노드 쿠버네티스는 컨테이너를 파드내에 배치하고 노드 에서 실행함으로 워크로드를 구동한다. 노드는 클러스터에 따라 가상 또는 물리적 머신일 수 있다. 각 노드는 컨트롤 플레인에 의해 관리되며 파드를 실행하는 데 필요한 서비스를 포함한다. 일반적으로 클러스터에는 여러 개의 노드가 있으며, 학습 또는 리소스가 제한되는 환경에서는 하나만 있을 수도 있다. 노드의 컴포넌트에는 kubelet, 컨테이너 런타임 그리고 kube-proxy가 포함된다. - 컨트롤 플레인   컨테이너의 라이프사이클을 정의, 배포, 관리하기 위한 API와 인터페이스들을 노출하는 컨테이너 오케스트레이션 레이어. - 파드   파드는 컨테이너에서 실행중인 컨테이너의 집합 - kubelet 클러스터의 각 노드에서 실행되는 에이전트 , kubelet 은 파드에서 컨테이너가 확실하게 동작하도록 관리한다. - 컨테이너 런타임  컨테이너 런타임은 컨테이너 실행을 담당하는 소프트웨어이다. - kube-proxy kube-proxy 는 클러스터의 각 노드에서 실행되는 네트워크 프록시로, 쿠버네티스의 서비스 개념의 구현부이다.

---

- Bookmark

 

https://kubernetes.io/ko/docs/concepts/architecture/nodes/

노드

https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

kubeadm 설치하기

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#upgrading-control-plane-nodes

Upgrading kubeadm clusters