You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
This repo is archived. You can view files and clone it, but cannot push or open issues/pull-requests.
Tobi 05c890cca7 better readme 1 year ago
clis manual way 1 year ago
example manual way 1 year ago
kubernetes manual way 1 year ago better readme 1 year ago
heketi.json manual way 1 year ago
heketi.service init 1 year ago
heketi.yaml manual way 1 year ago
service-account.json ha cli 1 year ago new 1 year ago
storage_class.yaml ha cli 1 year ago

Heketi API

Intented to be used for dynamic provisioning of volumes in Kubernetes


Generate SSH key

ssh-keygen -t rsa -b 4096

Setup Script

chmod +x and run on one of your nodes, this will:

  • install the heketi, heketi-cli
  • setup and run heketi as a service

Joining Nodes

install ssh key on the node (so heketi can manage it)

ssh-copy-id -i /etc/heketi/ root@nodename

remove it from auto mount (delete the according lines)

nano /etc/fstab

wipe the partion so that glusterfs can use it

umount /path/where/its/mountet
wipefs --all --force /dev/partion_name

Join the new node

(you might need to do gluster peer probe NODE_HOSTNAME first - on any of the other nodes/the first one)

heketi-cli -s http://IP_OF_YOUR_HEKETI_INSTALL:8080 --user admin --secret admin-key \
node add --zone=1 --cluster=b9aaaba32491ba8991dfd5fc1c02f88e --management-host-name="mikasa" --storage-host-name=""

IP_OF_YOUR_HEKETI_INSTALL should be localhost for the first 3 nodes, after that you would use the heketi instance that runs in the cluster (more on that in the next steps)

the storage hostname needs to be the ip of the new node

Add disk/partion of node

heketi-cli -s http://localhost:8080 --user admin --secret admin-key \
device add --name=/dev/nvme0n1p3 --node 9249aa5ed1d85281189343752d9bb320

Init cluster via topology

Alternatively you can add a whole bunch of nodes and thier devices via a topology (see topology.json for an example)

heketi-cli -s http://localhost:8080 --user admin --secret admin-key \
topology load --json=/path/to/topology.json

Mount a volume on a dev machine

sudo mount -t glusterfs ONE_OF_THE_NODES_HOSTNAME:/volume_name /data

Heketi Deployment and service

Heketi is just as important as the rest of the kubernetes core features if not more. We want it HA aswell so we need to heave it up into the cluster.

Kubernetes Config

  • create service account kubectl apply -f service-account.json

  • give service account the needed rights kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account

  • create config secret (wrong word, its just a simple storage, configmap seems more appropriate to use) kubectl create secret generic heketi-config-secret --from-file=/etc/heketi/heketi.json

Heketi Deployment

create an volume unknown to the cluster for your heketi configuration

heketi-cli -s http://localhost:8080 --user admin --secret admin-key create volume --size=1 --name=heketi-db-storage

copy over your heketi.db file

mkdir /mnt/heketi_db_transfer
mount -t glusterfs localhost:/heketi-db-storage /mnt/heketi_db_transfer
cp /var/lib/heketi/heketi.db /mnt/heketi_db_transfer/

create secrets to mount later

kubectl create secret generic heketi-config --from-file=/etc/heketi/heketi.json
kubectl create secret generic heketi-private-key --from-file=/etc/heketi/heketi_key
kubectl create secret generic heketi-public-key --from-file=/etc/heketi/

create the deployment

kubectl apply -f endpoints.yaml
kubectl apply -f heketi.yaml

!! if you use dns hostnames like (“mikasa” and not something like “”) in your topology or when joining nodes, you need to specify them in the heketi.yaml file under “hostAliases”. Otherwise the pod cant resolve the hostname, since its using coredns, and will thus fail.

Any other easy way I have discovered so far either involves using a enterprise solution like openshift with all thier priced operating systems or running glusterfs in a DaemonSet. This solution seems very sloppy to me since we are running a system level service in a container. The trickery to get this to work is astonishing, looking at the respective yaml files will make you feel completely lost. For now this seems to be the best solution for a home lab. Feel free to code something to automate the node addressing yourself.

accessing the service

=> now you can access heketi via heketi-cli -s --user admin --secret admin-key aswell

! use that url in the storage class