CKA (Certified Kubernetes Administrator)/Kode Kloud

10.Troubleshooting - Application Failure

seulseul 2022. 1. 26. 17:44
LABS – CERTIFIED KUBERNETES ADMINISTRATOR WITH PRACTICE TESTS > TROUBLESHOOTING
Troubleshooting

01. Application Failure
02. Control Plane Failure
03. Worker Node Failure
04. Troubleshoot Network

 

Troubleshooting Test 1: 

 

A simple 2 tier application is deployed in the alpha namespace.
간단한 2계층 애플리케이션이 알파 네임스페이스에 배포됩니다.

It must display a green web page on success.
성공 시 녹색 웹 페이지를 표시해야 합니다.

Click on the app tab at the top of your terminal to view your application. It is currently failed.
터미널 상단의 앱 탭을 클릭하여 애플리케이션을 봅니다. 현재는 실패했습니다.

Troubleshoot and fix the issue.
문제를 해결하고 수정합니다.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram.
주어진 아키텍처를 고수하십시오. 아래 아키텍처 다이어그램에 제공된 것과 동일한 이름과 포트 번호를 사용하십시오.

Feel free to edit, delete or recreate objects as necessary.
필요에 따라 개체를 자유롭게 편집, 삭제 또는 다시 생성할 수 있습니다.

  • Fix Issue
 

01

 

The service name used for the MySQL Pod is incorrect. 

MySQL Pod에 사용된 서비스 이름이 올바르지 않습니다. 

According to the Architecture diagram, it should be mysql-service.
아키텍처 다이어그램에 따르면 mysql-service여야 합니다.

To fix this, first delete the current service: 

kubectl -n alpha delete svc mysql

Then create a new service with the following YAML file (or use imperative command):


apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  namespace: alpha
spec:
    ports:
    - port: 3306
      targetPort: 3306
    selector:
      name: mysql
      
Create the new service: kubectl create -f <service.yaml>
controlplane ~/01 ➜  curl http://web-service-ip:node-port
curl: (3) URL using bad/illegal format or missing URL
controlplane ~/01 ✖ kubectl describe svc web-service -n alpha

Name:                     web-service
Namespace:                alpha
Labels:                   <none>
Annotations:              <none>
Selector:                 name=webapp-mysql
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.46.7
IPs:                      10.43.46.7
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30081/TCP
Endpoints:                10.42.0.9:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

 

Troubleshooting Test 2: 

The same 2 tier application is deployed in the beta namespace.
동일한 2계층 애플리케이션이 베타 네임스페이스에 배포됩니다.


It must display a green web page on success.
성공 시 녹색 웹 페이지를 표시해야 합니다.


Click on the app tab at the top of your terminal to view your application.
터미널 상단의 앱 탭을 클릭하여 애플리케이션을 봅니다.


It is currently failed. Troubleshoot and fix the issue.
현재는 실패했습니다. 문제를 해결하고 수정합니다.


Stick to the given architecture.
주어진 아키텍처를 고수하십시오.
 
Use the same names and port numbers as given in the below architecture diagram.
아래 아키텍처 다이어그램에 제공된 것과 동일한 이름과 포트 번호를 사용하십시오.
 
Feel free to edit, delete or recreate objects as necessary.
필요에 따라 개체를 자유롭게 편집, 삭제 또는 다시 생성할 수 있습니다.

 

Troubleshooting Test 2
controlplane ~ ✖ kubectl get all -n beta
NAME                                READY   STATUS    RESTARTS   AGE
pod/mysql                           1/1     Running   0          24m
pod/webapp-mysql-75dfdf859f-mf4kv   1/1     Running   0          24m

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/mysql-service   ClusterIP   10.43.248.216   <none>        3306/TCP         24m
service/web-service     NodePort    10.43.148.157   <none>        8080:30081/TCP   24m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/webapp-mysql   1/1     1            1           24m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/webapp-mysql-75dfdf859f   1         1         1       24m
 
apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  namespace: beta
spec:
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  # targetPort 가 8080 으로 돼있어서 문제였음 3306으로 고친후 정상
  selector:
    name: mysql
If you inspect the mysql-service in the beta namespace,
you will notice that the targetPort used to create this service is incorrect.
Compare this to the Architecture diagram and change it to 3306. 
Update the mysql-service as per the below YAML:

apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  namespace: beta
spec:
    ports:
    - port: 3306
      targetPort: 3306
    selector:
      name: mysql

 

Troubleshooting Test 3: 

The same 2 tier application is deployed in the gamma namespace. It must display a green web page on success.
Click on the app tab at the top of your terminal to view your application. It is currently failed.
Troubleshoot and fix the issue.

동일한 2계층 애플리케이션이 감마 네임스페이스에 배포됩니다. 성공 시 녹색 웹 페이지를 표시해야 합니다.
 터미널 상단의 앱 탭을 클릭하여 애플리케이션을 봅니다. 현재는 실패했습니다. 문제를 해결하고 수정합니다.

Stick to the given architecture.
Use the same names and port numbers as given in the below architecture diagram.

Feel free to edit, delete or recreate objects as necessary.

주어진 아키텍처를 고수하십시오. 아래 아키텍처 다이어그램에 제공된 것과 동일한 이름과 포트 번호를 사용하십시오. 
필요에 따라 개체를 자유롭게 편집, 삭제 또는 다시 생성할 수 있습니다.

   

 

  • Fix Issue
If you inspect the mysql-service,
you will see that that the selector used does not match the label on the mysql pod.

Service:

root@controlplane:~# kubectl -n gamma describe svc mysql-service | grep -i selector
Selector:          name=sql00001

root@controlplane:~#

Pod:

root@controlplane:~# kubectl -n gamma describe pod mysql | grep -i label   
Labels:       name=mysql

root@controlplane:~#

As you can see the selector used is name=sql001 whereas it should be name=mysql.
Update the mysql-service to use the correct selector as per the below YAML:

apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  namespace: gamma
spec:
    ports:
    - port: 3306
      targetPort: 3306
    selector:
      name: mysql

 

Troubleshooting Test 4:

 The same 2 tier application is deployed in the delta namespace. It must display a green web page on success.
Click on the app tab at the top of your terminal to view your application.
It is currently failed. Troubleshoot and fix the issue.

동일한 2계층 애플리케이션이 델타 네임스페이스에 배포됩니다. 성공 시 녹색 웹 페이지를 표시해야 합니다. 
터미널 상단의 앱 탭을 클릭하여 애플리케이션을 봅니다. 현재는 실패했습니다. 문제를 해결하고 수정합니다.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram.
Feel free to edit, delete or recreate objects as necessary.

주어진 아키텍처를 고수하십시오. 아래 아키텍처 다이어그램에 제공된 것과 동일한 이름과 포트 번호를 사용하십시오.
 필요에 따라 개체를 자유롭게 편집, 삭제 또는 다시 생성할 수 있습니다.
Are the environment variables used by the deployment correct?

Try accessing the web application from the browser using the tab called app.
You will notice that it cannot connect to the MySQL database:

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set;
DB_User=sql-user; DB_Password=paswrd; 

1045 (28000): Access denied for user 'sql-user'@'10.244.1.9' (using password: YES)



According to the architecture diagram, 
the DB_User should be root but it is set to sql-user in the webapp-mysql deployment.

Use the command 
kubectl -n delta edit deployments.apps webapp-mysql 

and 
update the environment variable as follows:

spec:
      containers:
      - env:
        - name: DB_Host
          value: mysql-service
        - name: DB_User
          value: root
        - name: DB_Password
          value: paswrd
          
This will recreate the pod and you should then be able to access the application.
controlplane ~ ➜  k get all -n epsilon
NAME                                READY   STATUS    RESTARTS   AGE
pod/webapp-mysql-67cfc57cbc-cl5s4   1/1     Running   0          14m
pod/mysql                           1/1     Running   0          14m

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/mysql-service   ClusterIP   10.43.87.167   <none>        3306/TCP         14m
service/web-service     NodePort    10.43.90.219   <none>        8080:30081/TCP   14m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/webapp-mysql   1/1     1            1           14m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/webapp-mysql-67cfc57cbc   1         1         1       14m​



note

pod 나 replicaset 에 random string 이 붙는게 신기해서 뭔가 하고 찾아봤는데

디플로이먼트 -> 레플리카셋 -> 파드 순서로 생성되기 때문에
디플로이먼트 -> 레플리카셋 에서 생성되는 이름 [deployment-name] - [random string] 으로 생성이 되고
레플리카셋 -> 파드 에서 생성되는 이름은 [replicaset-name] - [random string] 으로 생성이 되기 때문에
[random string] 이 두번 붙는 것을 볼수 있다.

반면 레플리카셋은 
레플리카셋 -> 파드 순으로 생성이 되기 때문에 
[replicaset - name] - [random string ] 만 붙어 [random string ] 이 한번만 붙은것을 알수 있다.


 

Troubleshooting Test 5:

 The same 2 tier application is deployed in the epsilon namespace.
It must display a green web page on success.
Click on the app tab at the top of your terminal to view your application. It is currently failed.
Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram.
Feel free to edit, delete or recreate objects as necessary.
  • Fix Issue

If you inspect the environment variable called MYSQL_ROOT_PASSWORD, 
you will notice that the value is incorrect as compared 
to the architecture diagram:

root@controlplane:~# kubectl -n epsilon describe pod mysql  | grep MYSQL_ROOT_PASSWORD 
      MYSQL_ROOT_PASSWORD:  passwooooorrddd
      
root@controlplane:~#

Correct this by deleting and recreating the mysql pod with 
the correct environment variable as follows:

spec:
  containers:
  - env:
    - name: MYSQL_ROOT_PASSWORD
      value: paswrd
      
Also edit the webapp-mysql deployment and make sure 
that the DB_User environment variable is set to root as follows:

spec:
      containers:
      - env:
        - name: DB_Host
          value: mysql-service
        - name: DB_User
          value: root
        - name: DB_Password
          value: paswrd
Once the objects are recreated, and you should be able to access the application.

 

Troubleshooting Test 6

 The same 2 tier application is deployed in the zeta namespace. It must display a green web page on success. Click on the app tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

Inspect the ports used by the web-service is it correct?

Are the environment variable used as per the architecture diagram?

 

There are a few things wrong in this setup:

1. If you inspect the web-service, 
you will see that the nodePort used is incorrect.
This service should be exposed on port 30081 and NOT 30088.

root@controlplane:~# kubectl -n zeta get svc web-service 
NAME          TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
web-service   NodePort   10.102.190.212   <none>        8080:30088/TCP   3m1s

root@controlplane:~#

To correct this, delete the service and recreate it using the below YAML file:

```
apiVersion: v1
kind: Service
metadata:
  name: web-service
  namespace: zeta
spec:
  ports:
  - nodePort: 30081
    port: 8080
    targetPort: 8080
  selector:
    name: webapp-mysql
  type: NodePort


2. Also edit the webapp-mysql deployment 
and make sure that the DB_User environment variable is set to root as follows:

spec:
      containers:
      - env:
        - name: DB_Host
          value: mysql-service
        - name: DB_User
          value: root
        - name: DB_Password
          value: paswrd

3. The DB_Password used by the mysql pod is incorrect.
Delete the current pod and recreate with the correct environment
variable as per the snippet below:

spec:
  containers:
  - env:
    - name: MYSQL_ROOT_PASSWORD
      value: paswrd
      
Once the objects are recreated, 
and you should be able to access the application.

Bookmark

https://kubernetes.io/docs/reference/kubectl/cheatsheet/

 

kubectl Cheat Sheet

This page contains a list of commonly used kubectl commands and flags. Kubectl autocomplete BASH source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first. echo "source <(kubect

kubernetes.io

# 먼저 Application에 대해서 접속이 가능한지 확인 

curl http://web-service-ip:node-port 

# 위에 에러가 나면 유저가 접속하기 위한 service를 확인 
## 이때 service의 Selector부분과 EndPoint가 옳바르게 대응되고있는지 확인필요함 
kubectl describe service web-service 

# Pod 확인 
kubectl get pod 
# 어떤 Pod이 떠있는지 확인하고 

kubectl describe pod web 

# 특정 Pod에 대해 정보를 확인하고
kubectl logs web 

# Pod에서 출력되는 로그들을 확인한다

Port, TargetPort, NodePort 의 개념
  • NodePort - 외부에서 접속하기 위해 사용하는 포트
  • Port - Cluster 내부에서 사용할 Service 객체의 포트
  • targetPort - Service 객체로 전달된 요청을 Pod(Deployment)로 전달할때 사용하는 포트
저작자표시

'CKA (Certified Kubernetes Administrator) > Kode Kloud' 카테고리의 다른 글

06.Security - Service Accounts  (0) 2022.01.27
06. Security - ClusterRoles  (0) 2022.01.26
06.Security - Role Based Access Controls  (0) 2022.01.25
06.Security - KubeConfig  (0) 2022.01.25
07.Storage - Storage Class  (0) 2022.01.25

'CKA (Certified Kubernetes Administrator)/Kode Kloud'의 다른글

  • 현재글10.Troubleshooting - Application Failure

관련글

티스토리툴바