쿠버네티스에서 그누보드5 서비스 설정하기.

## 쿠버네티스 이용해서 그누보드5 서비스하기

닭잡는데 소잡는 칼 쓰는거 같긴 하지만, kubernetes 로 그누보드5 를 설치해 봤습니다.
내 PC 는 우분투 리눅스 19.10 버전이고 , 우분투 캐노니컬에서
제공하는 microk8s 라는 쿠버네티스 버전을 사용했습니다.
그누보드는 웹서버와 mysql DB 서버를 분리하고, 데이타 저장을 별도 볼륨으로
해서 서비스 종료시에도 기존 데이타가 사라지지 않게 했습니다.
멀티 노드에서 클러스터를 구축해서 사용할 수도 있다지만, 능력이 안되니 이번에는
내 PC 한대에서 독립형으로 테스트 해 봤습니다.

192.168.219.160 kube-master

제일 하단에 참조 사이트들 읽으며 초보수준으로 이해하고 작업했기 때문에
엉성하거나 틀린 부분 있으면 정정 해주시길.


1. microk8s 설치

일단, 우분투리눅스에서 microk8s 쿠버네티스를 설치합니다.
설치하면, 쿠버네티스 명령어가 microk8s. 가 앞에 붙어 있습니다.
쿠버네티스 데몬은 microk8s.start, microk8s.stop 으로 시작 종료할수 있습니다.

nonots@kube-master:~$ sudo snap install microk8s --classic
microk8s v1.17.3 from Canonical✓ installed

nonots@kube-master:~$ microk8s.
microk8s.add-node     microk8s.disable      microk8s.inspect      microk8s.kubectl      microk8s.reset
microk8s.cilium       microk8s.enable       microk8s.istioctl     microk8s.leave        microk8s.start
microk8s.config       microk8s.helm         microk8s.join         microk8s.linkerd      microk8s.status
microk8s.ctr          microk8s.helm3        microk8s.juju         microk8s.remove-node  microk8s.stop

nonots@kube-master:~$ sudo snap alias microk8s.kubectl kubectl
쿠버네티스 작업에서 가장 많이쓰는 명령어가 kubectl 인데 우분투 kubectl은
microk8s.kubectl 이라는 긴 이름이어서  alias 로 간단하게 kubectl 로 쓸수 있게 만듭니다.
버전은 1.17.3 입니다.

nonots@kube-master:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

그리고, micro8ks.enable 명령어로 유용한 플러그인을 설치할수 있는데,
아래와 같이 dns dashboard 같은걸 추가 해 줍니다.
nonots@kube-master:~$ microk8s.enable dns dashboard ingress
 

2. 사전 작업

우선 mysql 정보와 gnuboard 웹소스가 저장될 저장 장치를 생성합니다.
mysql 컨테이너의 /var/lib/mysql 은 서버의 /home/nonots/mykube/mysql_data 에,
그리고 아파치 웹루트는 /home/nonots/mykube/gnuboard_data 를 사용하게 합니다.

nonots@kube-master:~/mykube$ mkdir /home/nonots/mykube/mysql_data
nonots@kube-master:~/mykube$ mkdir /home/nonots/mykube/gnuboard_data
nonots@kube-master:~/mykube$ mkdir /home/nonots/mykube/gnuboard_data/data
nonots@kube-master:~/mykube$ chmod 707 ./mysql_data/  
nonots@kube-master:~/mykube$ chmod 707 ./gnuboard_data/
nonots@kube-master:~/mykube$ chmod 707 ./gnuboard_data/data
mysql_data 는 초기에 빈디렉토리고,
gnuboard_data 는 그누보드5 전체 소스를 복사해 놓고 /data/ 디렉토리를
퍼미션 707로 생성합니다.

그리고 그누보드가 설치될 웹서버의 도커 이미지는 mysqli 와 gd 가 지원되는 php 모듈을 사용하는
웹서버이고, 웹루트가 빈 디렉토리인 /var/www/html 로 지정되어 있으면 아무거나 되는데. 내 경우

FROM php:7.4-apache
MAINTAINER nonots
RUN apt update && apt -y install libfreetype6-dev  libjpeg62-turbo-dev zlib1g-dev libpng-dev \
 && docker-php-ext-configure gd --with-freetype --with-jpeg \
 && docker-php-ext-install -j$(nproc) gd mysqli

위와 같이 간단한 3 줄로 Dockerfile 을 만들고 
nonots@kube-master:~/mykube$ docker build . -t nonots/gnudocker:v1
와 같이 명령어를 실행해서 간단하게 nonots/gnudocker:v1 라는 이미지 이름으로 별도로 빌드를 했는데,
이걸 제가 hub.docker.com 에 가입후
https://hub.docker.com/r/nonots/gnudocker
에 push 해 두었으므로, 별도 빌드할 필요없이 막바로 사용가능합니다.
물론, 더 상세하게 기능을 넣어 빌드를 하려면 개인적으로 새로 빌드를 해서 사용해도 됩니다.


3. 볼륨 만드는 작업

쿠버네티스에서 볼륨을 쓰는 방법은 다양한데, 영구 데이타 저장을 위해서
persistent volome (pv) 라는걸 사용하겠습니다.
일단, 아래 작업들은 yaml 파일을 만들어 생성하겠습니다.

nonots@kube-master:~/mykube$ cat pv-mysql.yaml 
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-mysql
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/nonots/mykube/mysql_data" 

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysqlclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi 

nonots@kube-master:~/mykube$ kubectl create -f pv-mysql.yaml 
persistentvolume/pv-mysql created
persistentvolumeclaim/mysqlclaim created

위 pv-mysql.yaml 은 mysql 데이타 저장을 mysql_data 디렉토리를 pv-mysql 이라는 볼륨으로 만들고
이 볼륨으로 mysqlclaim 이라는 볼륨claim 으로 mysql 서비스  요청을 받습니다.
쿠버네티스 볼륨 개념은 아직 헷갈리는데, 각자가 알아서 심화학습해 보시길.
참고로, yaml 가운데 하이픈 3개 "---" 는, 2개 이상 yaml 을 하나로 만들어서 
쓸수 있게 구분하는 겁니다. 즉 kubectl create 를 2 번 할걸 한번에 할수 있습니다.
테스트하느라 크기를 1G 로 작게 했는데, 실제 업무에 사용하려면,
SSD 디스크나 NAS 같은 스토리지를 달고 그 마운트 위치에 볼륨을 만들어 사용하면 됩니다.

nonots@kube-master:~/mykube$ cat pv-gnuboard.yaml 
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-gnuboard
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/nonots/mykube/gnuboard_data" 

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gnuboardclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi 

nonots@kube-master:~/mykube$ kubectl create -f pv-gnuboard.yaml 
persistentvolume/pv-gnuboard created
persistentvolumeclaim/gnuboardclaim created

위 내용은 그누보드 서비스가 사용될 볼륨을 gnuboardclaim 이라는 이름으로 만들어
그누보드 데이타를 저장합니다.
이렇게 생성한 볼륨 정보를 아래와 같이 get pv, get pvc 라는 명령어로  확인 가능합니다.

nonots@kube-master:~/mykube$ kubectl get pv
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE
pv-gnuboard   1Gi        RWO            Retain           Bound    default/gnuboardclaim                           32s
pv-mysql      1Gi        RWO            Retain           Bound    default/mysqlclaim                              61s

nonots@kube-master:~/mykube$ kubectl get pvc
NAME            STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
gnuboardclaim   Bound    pv-gnuboard   1Gi        RWO                           34s
mysqlclaim      Bound    pv-mysql      1Gi        RWO                           63s
nonots@kube-master:~/mykube$ 


4. secret 생성
비밀번호 등 민간한 정보를 yaml 파일등에 직접 적어 넣지 않고, 
쿠버네티스에서 secret 오브젝트를 생성해서 불러오는 방법으로 안전하게 사용가능합니다.,

nonots@kube-master:~/mykube$ kubectl create secret generic mysql-root-pwd --from-literal=password=root2349
secret/mysql-root-pwd created
nonots@kube-master:~/mykube$ kubectl create secret generic mysql-nonots-pwd --from-literal=password=kwon2349
secret/mysql-nonots-pwd created

위와 같이 --from-literal 명령어로 직접 password 키워드에 비번을 저장해도 되고,
아니면 --from-file=[파일명] 이라는 옵션으로 비번을 적어둔 text 파일을 만들어서 불러오게 해도 됩니다.
위에서는 mysql root 비번과 그누보드 db  사용자 비번을 mysql-root-pwd 와 mysql-nonots-pwd 라는 
이름으로 만들었습니다.

nonots@kube-master:~/mykube$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-vw2dv   kubernetes.io/service-account-token   3      25h
mysql-nonots-pwd      Opaque                                1      10s
mysql-root-pwd        Opaque                                1      19s


5. mysql DB 서비스 생성

nonots@kube-master:~/mykube$ cat mysql-service.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - image: mysql:latest
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-root-pwd
                  key: password
            - name: MYSQL_DATABASE # 구성할 database명
              value: nonots_db
            - name: MYSQL_USER # database에 권한이 있는 user
              value: nonots
            - name: MYSQL_ROOT_HOST # 접근 호스트
              value: '%'  
            - name: MYSQL_PASSWORD # database에 권한이 있는 user의 패스워드
              valueFrom:
                secretKeyRef:
                  name: mysql-nonots-pwd
                  key: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysqlclaim 

---

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  type: ClusterIP
  ports:
    - port: 3306
  selector:
    app: mysql 

nonots@kube-master:~/mykube$ 

위 mysql-service.yaml 파일은, mysqlclaim 이라는 이름으로 /var/lib/mysql 을 마운트할
볼륨으로 사용하고, mysql 루트 비번과 nonots 사용자 비번은  secret 으로 만든 mysql-root-pwd
와 mysql-nonots-pwd 를 참조해서 안전하게 불러오고
그누보드 사용 db 는 nonots_db, db 사용자는 nonots 로 지정합니다.
그리고 이 mysql 서비스와 컨테이너 이름을 "mysql" 로 했기 때문에,
나중에 그누보드 설치화면에서 mysql 서버의 host 정보란을 기본 localhost 가 아니라
mysql 이라고 입력하면 됩니다.

nonots@kube-master:~/mykube$ kubectl create -f mysql-service.yaml 
deployment.apps/mysql created
service/mysql created

이렇게 mysql 를 디플로이해서 서비스를 생성하면 아래와 같이
mysql-5cc7548b5-nx595 비슷한 이름으로 pod 아이디를 가지는데(매번 다 다름), 이 아이디로  컨테이너 내부에
mysql 명령어로 명령을 내려서 mysql> 프롬프트가 정상적으로 뜨는지 확인합니다.

nonots@kube-master:~/mykube$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
mysql-5cc7548b5-nx595   1/1     Running   0          23s
nonots@kube-master:~/mykube$ kubectl exec -it mysql-5cc7548b5-nx595 -- mysql -unonots -pkwon2349 nonots_db



6. gnuboard 서비스 실행

nonots@kube-master:~/mykube$ cat gnuboard-service.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gnuboard
  labels:
    app: gnuboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gnuboard
  template:
    metadata:
      labels:
        app: gnuboard
    spec:
      containers:
        - image: nonots/gnudocker:v1
          name: gnuboard
          ports:
            - containerPort: 80
              name: gnuboard
          volumeMounts:
            - name: gnuboard-persistent-storage
              mountPath: /var/www/html
      volumes:
        - name: gnuboard-persistent-storage
          persistentVolumeClaim:
            claimName: gnuboardclaim 

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: gnuboard
  name: gnuboard
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: gnuboard

nonots@kube-master:~/mykube$ 

이번에는 gnuboard 를 실행할 아파치 웹서버를 서비스로 실행하는 방법입니다.
위와 같이 gnuboard-service.yaml 을 만듭니다.
gnuboardclaim 이라는 이름으로 웹루트/var/www/html 를 마운트해서 데이타가 영구 저장될
볼륨을 연결합니다.

nonots@kube-master:~/mykube$ kubectl create -f gnuboard-service.yaml 
deployment.apps/gnuboard created
service/gnuboard created

nonots@kube-master:~/mykube$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
gnuboard-55f9fffcb-qh9ff   1/1     Running   0          4s
mysql-5cc7548b5-nx595      1/1     Running   0          3m17s

nonots@kube-master:~/mykube$ kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
gnuboard     NodePort    10.152.183.19            80:31856/TCP   20s
kubernetes   ClusterIP   10.152.183.1             443/TCP        25h
mysql        ClusterIP   10.152.183.110           3306/TCP       3m33s
nonots@kube-master:~/mykube$ 

2개 서비스를 정상 실행화면 위와 같이 get pods, get services, get deployments 로
확인합니다.
kubectl get services 명령어로 보면 gnuboard가 10.152.183.19 사설아이피에
31856 포트로 외부로 사용중임을 알수 있습니다.
이 포트는 서비스 실행시 임의로 매번 다르게 부여됩니다 .
이제 웹브라우저에서 
http://localhost:31856 혹은 http://192.168.219.160:31856 으로 접속합니다.
그누보드 설치 화면이 떠야 정상입니다.
설치화면에서  mysql 정보의 Host 란은 "mysql" 라고 넣습니다.
위에서 mysql 서비스 아이피인 10.152.183.110 을 Host 란에 넣어도 되긴 되는데,
문제는 서비스 삭제시 나중에 다시 서비스를 실행하면 사설 아이피 대역이 변경될 수 있으므로
그냥 Db 연결 서버 정보에 "mysql" 이라고 넣습니다.

정상적으로 그누보드 사용하면서 글을 몇개 등록해 봅니다.

그리고 나서 데이타가 보존되는지 확인하기 위해  아래와 같이 delete 로 2개 서비스를 삭제합니다.
그러면 그누보드 서비스가 사라집니다.

nonots@kube-master:~/mykube$ kubectl  delete service gnuboard mysql
service "gnuboard" deleted
service "mysql" deleted
nonots@kube-master:~/mykube$ kubectl  delete deployments gnuboard mysql
deployment.apps "gnuboard" deleted
deployment.apps "mysql" deleted


그런후 다시 mysql-service.yaml 와 gnuboard-service.yaml 파일을 이용해서
서비스를 생성합니다.

nonots@kube-master:~/mykube$ kubectl create -f mysql-service.yaml 
deployment.apps/mysql created
service/mysql created
nonots@kube-master:~/mykube$ kubectl create -f gnuboard-service.yaml 
deployment.apps/gnuboard created
service/gnuboard created
nonots@kube-master:~/mykube$ kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
gnuboard     NodePort    10.152.183.141           80:30129/TCP   10s
kubernetes   ClusterIP   10.152.183.1             443/TCP        25h
mysql        ClusterIP   10.152.183.122           3306/TCP       16s

그런후 kubectl get services 정보를 보면 이번에는 gnuboard 서비스가 
30129 포트로 실행되었습니다. 위와 달라졌습니다.
외부 노출 포트를 8080 같이 하나로 고정하고 싶은데 이건 kubectl-proxy 를 써야 하는거 같은데
거기 까진 아직 모르겠습니다. 

(수정 :  대충 알아낸거 같습니다.
$ kubectl port-forward svc/gnuboard 8080:80 
와 같이 하니, 컨테이너 내부 80 번 포트를 외부에서 접속할대 8080 으로 됩니다.
만약 80:80 와 같이 80번포트를 열려면, root 권한으로 해야 1024 번 아래 포트를 열수있으므로
일반 계정이라면 1024 번 보다 큰 포트를 열 수 있습니다.
이제 웹브라우저에서 http://localhost:8080 와 같이 접속하면 됩니다.
)

 

암튼 이렇게 바뀐 포트로 
http://localhost:30129 혹은 http://192.168.219.160:30129 으로 접속해보면
이전에 그누보드 설치된 상태, 글등록 정보가 그대로 남아 있을겁니다.
볼륨을 강제로 삭제하기 전에는 정보가 그대로 남아 있습니다.
포트가 달라졌으므로 그누보드관리자에서 캐시와 세션 파일을 일괄 삭제해 주는게 좋습니다.


7. 로드밸런싱 하기
이제 부하분산 기능을 테스트해 보겠습니다.
아래와 같이 gnuboard scale 을 3 으로 복제합니다.
그러면 kubectl get pods  하면 3 개로  늘어나 있습니다.

nonots@kube-master:~/mykube$ kubectl scale deployment gnuboard --replicas=3
deployment.apps/gnuboard scaled
nonots@kube-master:~/mykube$ kubectl get deployments
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
gnuboard   3/3     3            3           13m
mysql      1/1     1            1           13m
nonots@kube-master:~/mykube$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
gnuboard-55f9fffcb-5nk2z   1/1     Running   0          13m
gnuboard-55f9fffcb-c5xrn   1/1     Running   0          24s
gnuboard-55f9fffcb-w8xh4   1/1     Running   0          24s
mysql-5cc7548b5-kpgfd      1/1     Running   0          13m
nonots@kube-master:~/mykube$ 

nonots@kube-master:~/mykube$ kubectl edit services gnuboard 

그런후 위 처럼 gnuboard 서비스 정보를 에디트 모드로 들어가서 (아마 VI 에디터가 뜰겁니다.)
type: NodePort 라는 부분을
type: LoadBalancer 라고 수정하고 저장 후 닫습니다. 그런 후
gnuboard 서비스 상세보기를 하면 아래 처럼 Endpoints  부분에
10.1.9.48:80,10.1.9.49:80,10.1.9.50:80 와 같이 3 개의 사설 아이피가 보일겁니다.

nonots@kube-master:~/mykube$ kubectl describe service gnuboard
Name:                     gnuboard
Namespace:                default
Labels:                   app=gnuboard
Annotations:              
Selector:                 app=gnuboard
Type:                     LoadBalancer
IP:                       10.152.183.141
Port:                       80/TCP
TargetPort:               80/TCP
NodePort:                   30129/TCP
Endpoints:                10.1.9.48:80,10.1.9.49:80,10.1.9.50:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason  Age    From                Message
  ----    ------  ----   ----                -------
  Normal  Type    2m41s  service-controller  NodePort -> LoadBalancer
nonots@kube-master:~/mykube$ 

이제 이 3개 웹서버가 로드밸런싱이 되는지 확인해 보겠습니다. 웹루트 볼륨
/home/nonots/mykube/gnuboard_data 에 lb.php 라는 파일을 아래와 같이 생성한 후에.
웹브라우저에서 
http://localhost:30129/lb.php 혹은 http://192.168.219.160:30129/lb.php 으로 여러번
접속해봅니다.
터미널에서 curl 명령모드에서 실행해 보면 막바로 부하 분산이 되어 접속되는걸 확인할수 있습니다.

nonots@kube-master:~/mykube$ cat gnuboard_data/lb.php 

nonots@kube-master:~/mykube$ curl http://localhost:30129/lb.php
gnuboard-55f9fffcb-5nk2z - localhost - 10.1.9.48
nonots@kube-master:~/mykube$ curl http://localhost:30129/lb.php
gnuboard-55f9fffcb-w8xh4 - localhost - 10.1.9.49
nonots@kube-master:~/mykube$ curl http://localhost:30129/lb.php
gnuboard-55f9fffcb-5nk2z - localhost - 10.1.9.48
nonots@kube-master:~/mykube$ curl http://localhost:30129/lb.php
gnuboard-55f9fffcb-w8xh4 - localhost - 10.1.9.49
nonots@kube-master:~/mykube$ curl http://localhost:30129/lb.php
gnuboard-55f9fffcb-c5xrn - localhost - 10.1.9.50
nonots@kube-master:~/mykube$ 

아래 참조 사이트 등을.. 대충 검색해서 대충 테스트 해 본 것임을 감안하시길.

참고 :
https://cleanupthedesk.tistory.com/16
https://nirsa.tistory.com/category/DevOps/Kubernetes
https://developer.ibm.com/kr/journey/scalable-wordpress-on-kubernetes/