Kubernetes Homelab

[toadie]

Updated: 2 August, 8.44 am Central European Time

Today I’d like to introduce my little home lab.

This is just the initial post and will be updated regularly

The aim of the home lab is to deepen my knowledge and skills in Kubernetes. My goal is to get as close as possible to real-world scenarios.

Of course, I also experiment with things that have nothing to do with business or IT.

---

Setup

Hardware

Device	Hostname	Function	IP-WIFI	IP-LAN
Raspberry Pi 4 / 8GB	winterfell	Control Plane	210	211
Raspberry Pi 4 / 1GB	worker01	Worker Node	212	213
Raspberry Pi 4 / 1GB	worker02	Worker Node	214	215

Network
192.168.178.0/24

Storage
USB-SSD with XFS filesystem

Software

Operating System

• Raspberry Pi OS

Kubernetes distribution

• k3s

Administration Tools

• k9s
• ku
• helm
• Ansible

K3S Cluster

What is running on the cluster?

Monitoring

• Prometheus / Grafana Stack
• Uptime-Kuma

Development

• Forgejo (not yet implemented)
• Code-Server (not yet implemented)

Production / Private

• Audiobookshelf (not yet implemented)
• Vaultwarden (not yet implemented)
• Linkding (not yet implemented)

Testing

• many

Apply an deployment

Here you can see how to deploy an deployment with kubernetes.
In this case it is a minimal Ubuntu Container.
Scaling an container is a bit like RAID. It depents on the settings but basicly it works similar.
Examble: Scaling the Ubuntu Container to 10 means 10 instances of the same container. If 1 or 2 (depents on the settings) are failling, the users won’t notice.

[toadie]

Audiobookshelf is a more complex project.

I’ve been running it with Docker for about two years. Now I’d like to migrate it to the Kubernetes cluster. Audiobookshelf itself isn’t a problem at all, but migrating the data will be interesting.

[hydn (Hayden James)]

@toadie

The aim of the home lab is to deepen my knowledge and skills in Kubernetes

This is an important part of home labs. A safe place to fail, learn, fail again and learn some more.

[toadie]

Yes, absolutely!

The great thing about Kubernetes is that you can use namespaces.

When I’m testing a new app, I create a new namespace to which everything is linked.

If my test goes wrong, I delete the namespace and everything related to the app is deleted as well.

If everything went well and is working, I do the same thing. I then adjust the desired namespace in the underlying YAML file (e.g. fun) and recreate the app in the new, correct namespace.

Of course, only if I actually intend to use the app after my tests

At the same time, I then have ‘production-ready’ YAML so I can deploy the app to a cluster at any time.

[Brian_Masinick]

Thanks for sharing your expertise with us @toadie !

[toadie]

Ansible Playbooks

Git Sync

To sync my Git repo over all devices, I use this Ansible Playbook.

---
- name: Synchronize Git repository on all nodes
  hosts: pi_cluster
  become: false
  vars:
    repo_url: "https://github.com/mrtoadie/lab.git"
    repo_dest: "/home/toadie/Projects/lab"
    repo_branch: "main"
    git_user: "toadie"
    
  tasks:
    # 1. Clone repository (if not present)
    - name: Check repository exists
      ansible.builtin.stat:
        path: "{{ repo_dest }}"
      register: repo_stat

    - name: Clone Repository
      ansible.builtin.git:
        repo: "{{ repo_url }}"
        dest: "{{ repo_dest }}"
        version: "{{ repo_branch }}"
        accept_hostkey: yes
      when: not repo_stat.stat.exists
      register: clone_result

    # 2. Pull repository (if already exists)
    - name: Pull Repository
      ansible.builtin.command:
        cmd: git pull origin {{ repo_branch }}
        chdir: "{{ repo_dest }}"
      when: repo_stat.stat.exists
      register: pull_result
      changed_when: "'Already up to date' not in pull_result.stdout"

    # 3. Show status
    - name: Log pull result
      ansible.builtin.debug:
        msg: |
          Host: {{ inventory_hostname }}
          Status: {{ 'Cloned' if clone_result.changed else 'Pulled' }}
          Changes: {{ pull_result.stdout | default('None') }}

Update Operating System

---
- name: Update Raspberry Pi Cluster
  hosts: pi_cluster
  become: yes
  serial: 1                       # one node at a time!
  gather_facts: yes
  
  tasks:
    - name: Update package cache
      ansible.builtin.package:
        update_cache: yes
        
    - name: Upgrade all packages
      ansible.builtin.package:
        name: "*"
        state: latest
        
    - name: Check if reboot is needed
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_required
      ignore_errors: yes
      
    - name: Reboot if necessary
      ansible.builtin.reboot:
        reboot_timeout: 300
      when: ansible_distribution == "Debian" or ansible_os_family == "RedHat"
      ignore_errors: yes
      
    - name: Wait for node to come back
      ansible.builtin.wait_for_connection:
        timeout: 300

Inventory.ini

The inventory.ini looks like this:

[pi_cluster]
winterfell ansible_host=192.168.178.210
worker01 ansible_host=192.168.178.212
worker02 ansible_host=192.168.178.214

[k3s_master]
winterfell

[k3s_workers]
worker01
worker02

[Furqan_Ashraf]

Nice setup. Using Ansible for the Pi cluster updates is a smart move, especially the serial: 1 and reboot check that avoids taking down all nodes at once. One thing I do in similar playbooks is add a health check task after the reboot, just a simple curl or ping loop, to confirm the node actually rejoined the cluster before moving to the next one. Saves some manual checking later if a node gets stuck.

[toadie]

Thank you. Great idea with the health check!

[omar]

Hi @toadie I’ve a question (I’m still new to Kubernetes and learning it and plan to make a homelab for it just like you)

Shouldn’t we let the worker nodes to have more specs because of the running applications? so it’s kind of replacing the control plane with one of the worker nodes?

[toadie]

Hello @omar,
I’m afraid I don’t understand your question.

Do you mean whether the worker nodes should have more RAM?

Basically, the control plane (master) should be the strongest link in the chain – it handles the ‘thinking’ and the administration.

The worker nodes (slaves) don’t need to think, just work, and don’t necessarily need to have the same processing power as the master.

Ideally, of course, that would be better.

In my case, the two workers only have 1 GB of RAM because the Pis with more RAM have become extremely expensive.

So far, though, that’s been enough. The great thing is, you can add new worker nodes at any time or set up a second cluster for redundancy – whatever takes your fancy.

My plan is to add another Pi with a bit more RAM as a third worker node and set up a second cluster to experiment with high availability (HA).

[Bombilla (John)]

Welcome to the community @omar, good to have you here!

[toadie]

Some Grafana dashboards I playing around with.

Some Stats of the Raspberrys

The memory usage isn’t richght and I don’t know why at this time…

Velero Backup

Kubernetes Stats

[hydn (Hayden James)]

Cool! @toadie Whats happening here?:

Dark mode has been increasingly growing on me. I think especially at night I try to use dark mode for everything now.

[toadie]

I don’t know
I can see it’s worker node 2

and the node is using get

that happens all the time. what it was in this case - i don’t know. pretty sure i can find it in the logs or in grafana.

Monitoring with Grafana and Kubernetes are very new to me

[Brian_Masinick]

@omar

Hi @toadie I’ve a question (I’m still new to Kubernetes and learning it and plan to make a homelab for it just like you)

Best wishes on that and I also send you my warm forum community greetings following your first post!

[toadie]

I was lucky enough to win a bid for a Pi 4 with 2 GB on eBay today. It should arrive by the weekend. Then I’ll expand the cluster.

[omar]

Hi folks! Thanks for the warm welcoming and sorry for the late response.

Do you mean whether the worker nodes should have more RAM?

Yes. In fact, I’ve read about that somewhere in an article on the internet but I’m not able to get it right now. But also I’ve seen someone creating a kubeadm multi-node cluster and he did give the nodes more hardware specs. (Also Kelsey Hightower Kubernetes cluster from scratch did give all nodes equal resources) I thought this is the way most people handle their clusters.

so that was my point, I hope you did understand me now. And keep going with great stuff. I plan to start mine ASAP ~ 1 month.

[toadie]

Yes, it’s certainly better if all the devices have the same hardware. If it had been cost-effective, I would have done that too.
But – and that’s the beauty of it – basically you can use anything to start with to learn and understand Kubernetes.

BTW: my new Pi from eBay has already arrived (bloody quick delivery!)
Unfortunately, the right power supply hasn’t arrived yet

[toadie]

I use velero for Kubernetes backups. Velero backups all the metadata (or selective) from clusters.
Very nice and easy to use app. But deleting backups is a pain in the as.

So I decided to code a little helper with go as always.

The first version lists all the backups; you can select several backups, and deleting them works too.

Basically, it’s everything I wanted. Let’s see what else I can do with it.

[toadie]

Quick and dirty Github Repo

GitHub - mrtoadie/go-velero-manager