Make simple Kubernetes cluster and deploy own Golang application

Main goal to create simple cluster with one master node and two worker-nodes .

All procedures we will do on 3 VPS with Ubuntu 20.04 Server installation. First step is to create separate user on all VPS, for the test environment let it be ‘kubuntu’. For orchestration we will use Ansible

Lets create directory for Ansible and make inventory file

mkdir kube
cd kube
touch hosts

Write config to hosts file

---
masters:
  hosts:
    kmaster:
      ansible_host: 10.0.1.7
workers:
  hosts:
    kworker1:
      ansible_host: 10.0.1.8
    kworker2:
      ansible_host: 10.0.1.9

Now we want to create playbook initial.yml for Ansible that create user on VPS

- hosts: all
  become: yes
  tasks:
    - name: create the 'kubuntu' user
      user: name=kubuntu append=yes state=present createhome=yes shell=/bin/bash
    - name: allow 'kubuntu' to have passwordless sudo
      lineinfile:
        dest: /etc/sudoers
        line: 'kubuntu ALL=(ALL) NOPASSWD: ALL'
        validate: 'visudo -cf %s'
    - name: set up authorized keys for the ubuntu user
      authorized_key: user=kubuntu key="{{item}}"
      with_file:
        - ~/.ssh/id_rsa.pub

Run playbook

ansible-playbook -i hosts initial.yml

Installing packages for Kubernetes, but first we need create template for configuring Docker. It’s need for disable cgroupfs as default docker driver and set systemd. Please follow the guide at https://kubernetes.io/docs/setup/cri/

Make file templates/docker.daemon.j2

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

Maybe need to edit nano /var/lib/kubelet/kubeadm-flags.env for systemd

Now create playbook kube-dependencies.yml

- hosts: all
  become: yes
  tasks:
   - name: install Docker
     apt:
       name: docker.io
       state: present
       update_cache: true
   - name: Enable systemD cgroups
     template:
       src: templates/docker.daemon.j2
       dest: /etc/docker/daemon.json

   - name: Start Docker
     service:
       name: docker
       state: started
       enabled: yes

   - name: install APT Transport HTTPS
     apt:
       name: apt-transport-https
       state: present

   - name: add Kubernetes apt-key
     apt_key:
       url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
       state: present

   - name: add Kubernetes' APT repository
     apt_repository:
      repo: deb http://apt.kubernetes.io/ kubernetes-xenial main
      state: present
      filename: 'kubernetes'

   - name: install kubelet
     apt:
       name: kubelet=1.18.0-00
       state: present
       update_cache: true

   - name: install kubeadm
     apt:
       name: kubeadm=1.18.0-00
       state: present

   - name: Disable SWAP since kubernetes can't work with swap enabled (1/2)
     become: yes
     shell: |
       swapoff -a

   - name: Disable SWAP in fstab since kubernetes can't work with swap enabled (2/2)
     become: yes
     replace:
       path: /etc/fstab
       regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
       replace: '# \1'



- hosts: master
  become: yes
  tasks:
   - name: install kubectl
     apt:
       name: kubectl=1.18.0-00
       state: present
       force: yes

In this playbook we install latest kubelet, kubead version ( 1.18), disable swap and remove it from fstab. On master node additionaly setup kebectl

Make playbook masters.yml for configuring master node

- hosts: masters
  become: yes
  tasks:
    - name: initialize the cluster
#      shell: kubeadm init --pod-network-cidr=10.244.0.0/16 >> cluster_initialized.txt
      shell: kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all >> cluster_initialized.txt
      args:
        chdir: $HOME
        creates: cluster_initialized.txt

    - name: create .kube directory
      become: yes
      become_user: kubuntu
      file:
        path: $HOME/.kube
        state: directory
        mode: 0755

    - name: copy admin.conf to user's kube config
      copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/kubuntu/.kube/config
        remote_src: yes
        owner: kubuntu

    - name: install Pod network
      become: yes
      become_user: kubuntu
      tags: flannel
      shell: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml >> pod_network_setup.txt
      args:
        chdir: $HOME
        creates: pod_network_setup.txt

Run playbook:

ansible-playbook -i hosts masters.yml

In my case i used option flag ‘-ignore-preflight-errors=all’ because Kubernetes need 2 or more CPU cores and my VPS have only 1.

Go to the master node and check that cluster is up and running

kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
kmaster   Ready    master   8m35s   v1.18.0

If everything OK we can start add worker-node to cluster. Create workers.yml

- hosts: masters
  become: yes
  gather_facts: false
  tasks:
    - name: get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

    - name: set join command
      set_fact:
        join_command: "{{ join_command_raw.stdout_lines[0] }}"


- hosts: workers
  become: yes
  tasks:
    - name: join cluster
      shell: "{{ hostvars['kmaster'].join_command }} >> node_joined.txt"
      args:
        chdir: $HOME
        creates: node_joined.txt

Run playbook

ansible-playbook -i hosts workers.yml

Check workers in cluster

kubectl get nodes
NAME       STATUS   ROLES    AGE    VERSION
kmaster    Ready    master   15m    v1.18.0
kworker1   Ready    <none>   116s   v1.18.0
kworker2   Ready    <none>   117s   v1.18.0

Test our cluster by adding test nginx service that only show “Hello page”

kubuntu@kmaster:~$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
kubuntu@kmaster:~$ kubectl expose deploy nginx --port 80 --target-port 80 --type NodePort
service/nginx exposed
kubuntu@kmaster:~$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        16m
nginx        NodePort    10.106.156.95   <none>        80:32709/TCP   12s

If we are in the same network as nginx than just go to browser and open page http://10.106.156.95 . Or we can use curl

curl -v 10.106.156.95
*   Trying 10.106.156.95:80...
* TCP_NODELAY set
* Connected to 10.106.156.95 (10.106.156.95) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.106.156.95
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.17.10
< Date: Mon, 27 Apr 2020 14:12:48 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 14 Apr 2020 14:19:26 GMT
< Connection: keep-alive
< ETag: "5e95c66e-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Works as expected. Cleaning up

kubectl delete service nginx

Chapter 2. Deploy own Golang app

Create directory for application

mkdir go_app
cd go_app
nano main.go

Code inside

package main

import (
	"fmt"
	"time"

	"github.com/gin-gonic/gin"
	"os"
	"gopkg.in/yaml.v2"
	"io/ioutil"
	"log"
)

type conf struct {
    Text string `yaml:"text"`
    Code int `yaml:"code"`
}

func (c *conf) getConf() *conf {
    filename := "/etc/ping/config.yml"
//    filename := "config.yml"
    _, err := os.Stat(filename)
    if os.IsNotExist(err) {
	c.Text = "Pong"
	c.Code = 200
    } else {
      yamlFile, err := ioutil.ReadFile(filename)
      if err != nil {
        log.Printf("yamlFile.Get err   #%v ", err)
      }
      err = yaml.Unmarshal(yamlFile, c)
      if err != nil {
        log.Fatalf("Unmarshal: %v", err)
      }
    }

    return c
}


func main() {

	var c conf
        c.getConf()
	router := gin.New()

	// LoggerWithFormatter middleware will write the logs to gin.DefaultWriter
	// By default gin.DefaultWriter = os.Stdout
	router.Use(gin.LoggerWithFormatter(func(param gin.LogFormatterParams) string {

		// your custom format
		return fmt.Sprintf("%s - [%s] \"%s %s %s %s %d %s \"%s\" %s BODY: %s\"\n",
			param.ClientIP,
			param.TimeStamp.Format(time.RFC1123),
			param.Method,
			param.Request.Host,
			param.Path,
			param.Request.Proto,
			param.StatusCode,
			param.Latency,
			param.Request.UserAgent(),
			param.ErrorMessage,
			param.Request.Body,
		)
	}))
	router.Use(gin.Recovery())

	router.GET("/ping", func(g *gin.Context) {
		g.String(c.Code, c.Text)
	})

	router.Run(":9991")


}

Thin app get request on endpoint /ping log this request to stdout and send answer to client with text “pong”. If config.yml exist then we override answer and code:

text: pong by config
code: 200

Main code was taken from post on Mediun – https://medium.com/google-cloud/deploy-go-application-to-kubernetes-in-30-seconds-ebff0f51d67b with some changes. So, for start, lets build our app:

go build
go run main.go

If everything ok than start to create Makefile for automation.

cat Makefile
#TAG?=$(shell git rev-list HEAD --max-count=1 --abbrev-commit)
TAG=0.1.2
export TAG

install:
	go get .

build: install
	go build -ldflags  "-X main.version=$(TAG) -w -s -linkmode external -extldflags -static"  -o  ping_app .

pack: build
	GOOS=linux make build
	docker build -t <registry-host>/myproject/ping_app:$(TAG) .

upload: pack
	docker push <registry-host>/myproject/ping_app:$(TAG)
deploy:
	envsubst < k8s/deployment.yml | kubectl apply -f -

If you code tracked by Git – use first line for assign TAG var, or use static version like in second line. make command now have some option that include some actions. For example pack use previous action build and after build Docker image with TAG. If you do not have your private docker registry, start it by docker-compose via manual – https://docs.docker.com/registry/ or https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-top-of-digitalocean-spaces-and-use-it-with-digitalocean-kubernetes

For creating docker image we need to create Dockerfile with actions for start out app

cat Dockerfile
FROM alpine:3.4

RUN apk -U add ca-certificates

EXPOSE 9991

ADD ping_app /bin/ping_app
#RUN chmod +x /bin/ping_app
ADD config.yml /etc/ping/config.yml

CMD ["/bin/ping_app", "-config", "/etc/ping/config.yml"]

This Docker file start/create image with our binary and config file, and expose port 9991

When we run make pack and make upload our image creates and upload to registry, now we need only deploy image/service to kubernetes ( deploy action in Makefile)

cat k8s/deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ping-service
  labels:
    app: ping-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ping-service
  template:
    metadata:
      labels:
        app: ping-service
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - name: ping-service
        image: <registry-host>/myproject/ping_app:0.1.3
        command:
        ports:
          - containerPort: 9991
        volumeMounts:
          - name: ping-config
            mountPath: /etc/ping/
            readOnly: true
      volumes:
        - name: ping-config
          configMap: { name: ping-config }
---
kind: Service
apiVersion: v1
metadata:
  name: ping-service
spec:
  type: LoadBalancer
  selector:
    app: ping-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9991

Some comments…

selector: matchLabels: it is like a link for knowing what servise for what deployment need to use

imagePullSecrets: - name: regcred – i have Docker registry with credentials, so we need somehow to say that kubernetes need to use login/password for pulling images. Simplest way to do this it is a login to Docker-registry docker login my-registry-url:5000 and they will create config to <homeDir>/.docker/config . Add this config to our Kubernetes via command:

kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=/home/kubuntu/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

We register secret with name regcred to cluster and use it in imagePullSecrets

This part of config need to pass our config to cluster, but we need way to update config without rebuild all image in registry.

        volumeMounts:
          - name: ping-config
            mountPath: /etc/ping/
            readOnly: true
      volumes:
        - name: ping-config
          configMap: { name: ping-config }

We will use config-map as volume and mount to /etc/ping/ . How to create config in cluster? simple…

cat k8s/configmap.yml
kind: ConfigMap
apiVersion: v1
metadata:
  name: ping-config
data:
  config.yml: |-
    text: pong by config
    code: 200

run kubectl apply -f configmap.yml

With this map we can update our configmap and file config.yml will be changed in cluster automaticaly.

Last step is to run make deploy

make deploy
envsubst < k8s/deployment.yml | kubectl apply -f -
deployment.apps/ping-service configured
service/ping-service changed

See what we have:

go_app $ kubectl get services
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1       <none>        443/TCP        3d23h
ping-service   LoadBalancer   10.98.133.140   <pending>     80:31842/TCP   25h

go_app $ kubectl get deployments.apps
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
nginx          1/1     1            1           3d23h
ping-service   1/1     1            1           25h

go_app $ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
nginx-f89759699-7j8v6          1/1     Running   1          3d23h
ping-service-d545db97b-496p4   1/1     Running   0          25h

we can do curl -v 10.98.133.140 and cluster answers pong via config

Tutorials used:

Chapter 2. Deploy own Golang app

Залишити відповідь Скасувати коментар