Persistent storage for raspberry pi k8s cluster

As I said in the last post, I installed kubernetes with the k3s tool.

It's cool to deploy stateless applications, but when you need something more complex, you'll need a persistent volume.

K3s comes with local-path-provisioner which is used to create local storages in each node of your cluster. What is the problem? If we are running a simple deployment with multiple replicas and each node needs to store data, we will realize that each node will save its own data. No storage will be shared between nodes and we don't want that behavior.

I tried to mount a GlusterFS directly with k8s drivers but I could't make it work. It seems that is a k3s incompatibility problem. So in this tutorial I am going to try another way.

Cluster info

Ansible

ini
[master]
192.168.1.100
[node]
192.168.1.101
192.168.1.102
192.168.1.103
192.168.1.104
192.168.1.105
[k3s_cluster:children]
master
node

Install glusterfs

INSTALL IN ALL NODES DEPENDENCIES:

bash
ansible all -i hosts.ini -a "sudo modprobe fuse" -b

bash
ansible all -i hosts.ini -a "sudo apt-get install -y xfsprogs glusterfs-server" -b

Add glusterd service to startup services

bash
ansible all -i hosts.ini -a "sudo systemctl enable glusterd" -b

bash
ansible all -i hosts.ini -a "sudo systemctl start glusterd" -b

ONLY ON MASTER NODES:

Ping all slave nodes

bash
sudo gluster peer probe 192.168.1.101

bash
sudo gluster peer probe 192.168.1.102

bash
sudo gluster peer probe 192.168.1.103

bash
sudo gluster peer probe 192.168.1.104

bash
sudo gluster peer probe 192.168.1.105

Check the connection status

bash
sudo gluster peer status

Create folder in all master nodes

bash
sudo mkdir -p /mnt/glusterfs/myvolume/brick1/

Create the volume

bash
sudo gluster volume create brick1 192.168.1.100:/mnt/glusterfs/myvolume/brick1/ force

Start the volume

bash
sudo gluster volume start brick1

Check the volume status

bash
sudo gluster volume status

Final step: mount them

Create the folder in all nodes for mounting (it will create on masters also but wont mount)

bash
ansible all -i hosts.ini -a "mkdir -p /mnt/general-volume" -b

• Mount in all nodes

bash
ansible all -i hosts.ini -a "mount -t glusterfs 192.168.1.100:/brick1 /mnt/general-volume" -b

Mount on reboot

• /root/mount-volumes.sh: create this file in all slaves nodes

bash
#!/bin/sh
function check_glusterd_running()
{
# Explanation of why ps command is used this way:
# https://stackoverflow.com/questions/9117507/linux-unix-command-to-determine-if-process-is-running
        if ! ps cax | grep -w '[g]lusterd' > /dev/null 2>&1
        then
                echo "ERROR: Glusterd is not running"
                exit 1
        fi
}
while [[ !  -z  $(check_glusterd_running)  ]]; do
  sleep 1s;
done
echo "Glusterd is running"
# =====> start volume mounts <======
mount -t glusterfs 192.168.1.100:/brick1 /mnt/general-volume

After creating this file on all slave nodes:

• Create crontab line

bash
sudo crontab -e
@reboot bash /root/mount-volumes.sh
# ^^^^^^^^^^^^ Add this line

k8s Manifest example

yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  volumes:
    - name: general-volume
      hostPath:
        path: "/mnt/general-volume/test"
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
    volumeMounts:
      - mountPath: "/app/data"
        name: general-volume
        readOnly: false
  restartPolicy: Always

Create a file in the container

bash
kubectl exec -it ubuntu -- mkdir -p /app/data && touch /app/data/hello-world

## SSH to any node and check if the new file is there

bash
ssh [email protected]

shell
ubuntu@node-02:~$ ls /mnt/general-volume/test

You are ready to go!