Introduction
I built a Kubernetes cluster for running parallel Jenkins builds for Swift and realized that the disk size I had selected for the worker nodes was woefully insufficient; all my jobs would be evicted simply because of disk pressure.
I used Ubuntu 20.04 for the builders and so by default an LVM volume is created which is nice because then I can extend its size instead of dealing with mounting a regular disk and dealing with setting up the directory and all that in the /etc/fstab file.
Here are the steps. I'm using Ubuntu 20.04 but I presume the info is relevant regardless of Linux flavor.
Step 1 - Add another disk to the machine
In my case my machines are running under VMWare ESXi. I thought to include a bunch of screenshots but in the end this step is: install the new disk somehow. Good luck with that.
Step 2 - Log into the machine and go root
Standard sudo su - here.
Step 3 - Scan for the new disk
The new disk won't be visible until you scan for it, which is easily done via:
for host in /sys/class/scsi_host/*; do echo "- - -" | sudo tee $host/scan; ls /dev/sd* ; done
This will roll through the SCSI buses (I had 25? WTF?) and eventually it found /dev/sdb. If there is already a /dev/sdb on your machine it'll be /dev/sdc etc.
Step 4 - Create a new Physical Volume on the disk
Enter:
pvcreate /dev/sdb
and if you run lvmdiskscan -l you'll see the new disk in the list.
Step 5 - Add the Physical Volume to the existing Logical Volume
The name of my Volume Group is called ubuntu-vg. You can find out what the VG is called by running lvdisplay and taking note of the LV Name parameter.
So now you can add the PV to the LV by entering:
vgextend ubuntu-vg /dev/sdb
And you should see a message that the volume group was extended.
Step 6 - Extend the Logical Volume
Now we're gonna tell the logical volume to use all that extra space we just gave it:
lvm lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
Step 7 - Now actually let the LV use the space we just allocated
Ah, but here's the critical last step, the filesystem (in my case EXT4) needs to be updated to include all the new space:
resize2fs -p /dev/ubuntu-vg/ubuntu-lv
And we're done
You should be able to run df -h and see the LVM now has all that additional space; in my case I added 50G to each node and now I really do have the space for my Swift builders. Yay!