k3s on RPis

Posted on 2021-02-17

An account of setting up a k3s cluster on RPi 4Bs.

Context

I’ve wanted to do this for a while and now I finally am! I went as far as purchasing two RPi 4B (4GB), with the promise to myself that I can extend to this (only) if I make use of it.

This post is about k3s: advertized as being awesome, simple to setup, and suitable for embedded. Fingers crossed.

I also wanted to try a 64bit OS. For raspbian, these are apparently still in beta(?). Anyway, the latest at the time of writing seems to be available from here.

There is no lite version, so that’s where I’ll start.

Setting up an OS

Note: It’s perhaps advantageous, if there are a large number of devices, to make a custom image with much of the setup completed, saving on repeating the same steps multiple times.

Download, check the hash, and extract. Then flash to the SD cards (use lsblk to aid finding SD card). (Ensure that the card is not mounted before running the following1 .)

export MY_IMG=~/Downloads/2020-02-13-raspbian-buster-lite.img
export MY_DEVICE=/dev/mmcblk0
sudo dd bs=4M if=$MY_IMG of=$MY_DEVICE

This will take a bit of time to complete.

Once flashed, mount both partitions of the SD card in order to begin modifying things.

In the boot partition, do the following. Add a blank file entitled ssh to the boot partition to enable ssh.

There are several edits to be done to the cmdline.txt file. k3s needs cgroups enabled. To do this, append at the end of the line three additonal declarations: cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory. (Note that if you make this a new line, it won’t work and you’ll find out about it by trawling through the journalctl -xe.)

If you want to make a custom image in order to not repeat the same steps for each sd card, then also in the cmdline.txt remove the declaration init=/usr/lib/raspi-config/init_resize.sh. Otherwise the resulting images will be larger than is comfortable.

In the rootfs partition do the following:

Setup ssh keys as the only way to authorize ssh: Add ssh key to home/pi/.ssh/authorized_keys. In etc/ssh/sshd_config, find, uncomment and set PasswordAuthentication no. Using wifi? Configure the wpa_supplicant (and good luck … 2)

Custom steps: maybe add bashrc, install vim etc for a more comfortable life down the road.

On the pi

Since the device runs headless, purge X: 3

sudo apt purge -y x11-common bluez gnome-menus gnome-icon-theme gnome-themes-standard
sudo apt purge -y hicolor-icon-theme gnome-themes-extra-data bluealsa cifs-utils
sudo apt purge -y desktop-base desktop-file-utils
sudo apt autoremove -y

In the raspi-config, reduce the memory set aside for graphics:

8 Advanced Options > Memory split > 16 > OK

Disable swap:

sudo systemctl disable dphys-swapfile.service

Run 4

sudo apt update
sudo apt upgrade

Either via the command line tools, or the raspi-config, set password and rename the hostdevice, eg 5

sudo hostnamectl set-hostname k3s-0

Reboot.

Pre-reqs for k3s

Read the instructions from source.

There are special steps needed for the raspberry pi, particularly running raspbian’s buster.

Re-enable iptables over the newer default nftables.

sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot

Enable cgroups, if you haven’t already. The RPi mounts the boot partition at /boot/. Thus in the file /boot/cmdline.txt append to the end of the line:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Reboot.

Note: Apart from without this, it doesn’t work, I’m really not sure what this is about. In particular, cgroup_enable=cpuset doesn’t appear on rancher, but does appear in Alex Ellis’s blog post.

This is a good place to stop, and create a custom image. The installation of k3s is handled differently between master and worker nodes.

Creating a custom image

Shutdown the pi. Pause. Power-down. Plug the sd card back into a linux machine and do the following.

Mount the boot partition. In cmdline.txt, reinsert the option removed init=/usr/lib/raspi-config/init_resize.sh.Then unmount.

Point fdisk at the sd card - it (usually) requires privilege. For example

sudo fdisk /dev/mmcblk0

Use the p command to learn the sector [^sector-block] size, and the last used sector.

Output should look something like this:

$sudo fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/mmcblk0: 58.25 GiB, 62534975488 bytes, 122138624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xad09722e

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        8192  532479  524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 7380991 6848512  3.3G 83 Linux

Then, setting bs to the block size ie sector size of 512 (in this case this is in fact the default value), and set count to be 1 more than the end sector, ie 7380992.

sudo dd bs=512 count=7380992 if=/dev/mmcblk0 of=2020-08-20-raspios-buster-arm64-for-k3s.img

If this is not being used immediately, this can be zipped for later… etc.

Edit: I’ve subsequently had to fix the images I made with this. The card works, but the rest of the rootfs partition is “unallocated”. This is fixed by running ‘check’ on the partition in gparted.

NOTE: k3s relies on hostnames being distinct. So either each hostname needs to be modified after flashing, or k3s will need explicitly telling to sort this on its own.

Installing k3s

On the master RPi, here called k3s-0, run

curl -sfL https://get.k3s.io | sh -

Ensure service is running by, for example:

$ sudo systemctl status k3s.service 
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
   Active: active (running) ...

If the installation script finds the environment variable K3S_URL set, then it will setup k3s agent service, and connect to the master.

On k3s-0, copy the token output by the command

sudo cat /var/lib/rancher/k3s/server/node-token

The output will be a line of mostly random characters, which will be referenced here by <token>. The address of the k3s-0 is also needed, which will be referenced as <k3s-0-ip>

On all worker devices, run

export K3S_URL=https://<k3s-0-ip>:6443
export K3S_TOKEN=<token>
curl -sfL https://get.k3s.io | sh -

Note that this may not be as easy as it looks.6

And voila!

[0 || k3s-0] [~]  
$sudo k3s kubectl get nodes
NAME    STATUS   ROLES                  AGE     VERSION
k3s-1   Ready    <none>                 4m39s   v1.20.2+k3s1
k3s-0   Ready    control-plane,master   10m     v1.20.2+k3s1

One final note: in order to not run everything in root/ have to type sudo k3s kubectl every time, we can make the config accessible to the user pi. There are numerous ways to do this. One is to make it readable to the user.

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s.yaml ~/.kube/config.yaml
sudo chown pi:pi ~/.kube/config.yaml

Then

export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config.yaml

Append this to the .bashrc for future sessions.

Note that if you need to do a reinstall, k3s helpfully ships with an uninstall tool, k3s-uninstall.sh or k3s-agent-uninstall.sh. The above steps will then need to be repeated to ensure the user’s file is in agreement with the one at /etc/rancher/k3s.yaml.

Backmatter

Appendix: How big a difference is purging X?

After update & upgrade free -mh had the following output

              total        used        free      shared  buff/cache   available
Mem:          3.7Gi       100Mi       2.1Gi        16Mi       1.6Gi       3.5Gi
Swap:          99Mi          0B        99Mi

After purge of X and related

              total        used        free      shared  buff/cache   available
Mem:          3.7Gi        98Mi       2.4Gi        16Mi       1.2Gi       3.5Gi
Swap:          99Mi          0B        99Mi

On rebooting

              total        used        free      shared  buff/cache   available
Mem:          3.8Gi        73Mi       3.6Gi       8.0Mi       106Mi       3.6Gi
Swap:            0B          0B          0B

I won’t claim I fully understand all these numbers just yet.

  1. I have a bad track record of trying to do initial connection over wifi, so opted for a cable. My first card did not work. The second one did. The second SD card required playing with the partition using fdisk. After cocking this up twice, following these instructions eventually fixed this.↩︎

  2. I’ve screwed this up so many times: trailing whitespaces, badly formatted country code, etc↩︎

  3. based on a post in this thread↩︎

  4. Don’t forget like I did.↩︎

  5. This worked on one, and I screwed the other up, and then needed to manually correct the /etc/hosts as directed to here. This can also be done before booting the RPi, by manually editing the /etc/hostname and /etc/hosts files. Details in the link.↩︎

  6. Github is awash with open issues, with people with all sorts of problems. Some seem to be down to the OS or architecture, others that the very simple setup script is very simple, because it tries to guess - and sometimes it guesses wrong. The most helpful comment I read seemed to be: don’t forget to run update && upgrade↩︎