Configure and use LVM thin provisioning.
Thin volumes are useful for overprovisioning especially for containers.
Preparations
Install LVM utilities.
$ sudo apt install lvm2
Initialize physical volume.
$ sudo pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
Create a volume group.
$ sudo vgcreate vg01 /dev/sdb
Volume group "vg01" successfully created
Create thin pool
Create thin pool.
$ sudo lvcreate --thin --size 5G --chunksize 256K --poolmetadatasize 1G vg01/thin_pool
Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. Logical volume "thin_pool" created.
Display information about logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-a-tz-- 5.00g 0.00 1.57 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g
Create thin volumes
Create thin volumes.
$ sudo lvcreate --thinpool vg01/thin_pool --name volume_cerberus --virtualsize 3G
Logical volume "volume_cerberus" created.
$ sudo lvcreate --thinpool vg01/thin_pool --name volume_kraken --virtualsize 3G
WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool vg01/thin_pool (5.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "volume_kraken" created.
$ sudo lvcreate --thinpool vg01/thin_pool --name volume_hydra --virtualsize 3G
WARNING: Sum of all thin volume sizes (9.00 GiB) exceeds the size of thin pool vg01/thin_pool (5.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "volume_hydra" created
Create an ext4 filesystem.
$ sudo mkfs.ext4 /dev/mapper/vg01-volume_cerberus
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 075ebaeb-36ef-4532-9826-7b926d3a2186
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mkfs.ext4 /dev/mapper/vg01-volume_kraken
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: d6fa7e01-5262-43d7-9718-d10a0e109cff
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mkfs.ext4 /dev/mapper/vg01-volume_hydra
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: fc9f511f-4310-4318-9b97-07b594e32dfb
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
Create mount directories.
$ sudo mkdir /opt/{cerberus,kraken,hydra}
Mount thin volumes.
$ sudo mount /dev/mapper/vg01-volume_cerberus /opt/cerberus/
$ sudo mount /dev/mapper/vg01-volume_kraken /opt/kraken/
$ sudo mount /dev/mapper/vg01-volume_hydra /opt/hydra/
Inspect disk space usage.
$ df -h /dev/mapper/vg01-volume_*
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg01-volume_cerberus 2.9G 24K 2.8G 1% /opt/cerberus /dev/mapper/vg01-volume_hydra 2.9G 24K 2.8G 1% /opt/hydra /dev/mapper/vg01-volume_kraken 2.9G 24K 2.8G 1% /opt/kraken
Trigger out of space situation
Create files to trigger out of space situation.
$ sudo dd if=/dev/zero of=/opt/cerberus/file bs=1M count=2048
2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.2239 s, 966 MB/s
$ sudo dd if=/dev/zero of=/opt/kraken/file bs=1M count=2048
2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.01049 s, 713 MB/s
$ sudo dd if=/dev/zero of=/opt/hydra/file bs=1M count=2048
^C Killed
Inspect disk space usage.
$ df -h /dev/mapper/vg01-volume_*
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg01-volume_cerberus 2.9G 2.1G 734M 74% /opt/cerberus /dev/mapper/vg01-volume_kraken 2.9G 2.1G 734M 74% /opt/kraken /dev/mapper/vg01-volume_hydra 2.9G 745M 2.0G 27% /opt/hydra
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotzD- 5.00g 100.00 1.63 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 25.80 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43
Reclaim used space
Delete created files.
$ sudo rm /opt/{cerberus,kraken,hydra}/file
Inspect disk space usage.
$ df -h /dev/mapper/vg01-volume_*
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg01-volume_cerberus 2.9G 24K 2.8G 1% /opt/cerberus /dev/mapper/vg01-volume_hydra 2.9G 24K 2.8G 1% /opt/hydra /dev/mapper/vg01-volume_kraken 2.9G 24K 2.8G 1% /opt/kraken
Notice, volume usage does not change.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotzD- 5.00g 100.00 1.63 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 25.80 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43 vagrant@bullseye:~$
Discard unused blocks on mounted filesystems.
$ sudo fstrim /opt/cerberus
$ sudo fstrim /opt/kraken
$ sudo fstrim /opt/hydra
Inspect volume usage.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 5.00g 6.76 1.58 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 3.76 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 3.75 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 3.76
Create snaphot
Create sample file.
$ sudo dd if=/dev/zero of=/opt/cerberus/file bs=1M count=2048
2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.7677 s, 776 MB/s
Create a snapshot.
$ sudo lvcreate --name cerberus_snapshot --snapshot vg01/volume_cerberus
WARNING: Sum of all thin volume sizes (12.00 GiB) exceeds the size of thin pool vg01/thin_pool (5.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "cerberus_snapshot" created.
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert cerberus_snapshot vg01 Vwi---tz-k 3.00g thin_pool volume_cerberus [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 5.00g 46.77 1.60 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 3.75 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 3.76
Delete created file.
$ sudo rm /opt/cerberus/file
Discard unused blocks on mounted filesystems.
$ sudo fstrim /opt/cerberus
Activate the snapshot.
$ sudo lvchange --activate y --ignoreactivationskip vg01/cerberus_snapshot
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert cerberus_snapshot vg01 Vwi-a-tz-k 3.00g thin_pool volume_cerberus 70.43 [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 5.00g 46.79 1.60 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 3.75 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 3.76
Create mount point.
$ sudo mkdir /opt/cerberus_snapshot
Mount snapshot.
$ sudo mount /dev/mapper/vg01-cerberus_snapshot /opt/cerberus_snapshot/
Inspect snapshot.
$ ls /opt/cerberus_snapshot/
file lost+found
Autoextend thin pool
Inspect activation/thin_pool_autoextend_threshold option.
$ sudo lvmconfig ---withcomment activation/thin_pool_autoextend_threshold
# Configuration option activation/thin_pool_autoextend_threshold. # Auto-extend a thin pool when its usage exceeds this percent. # Setting this to 100 disables automatic extension. # The minimum value is 50 (a smaller value is treated as 50.) # Also see thin_pool_autoextend_percent. # Automatic extension requires dmeventd to be monitoring the LV. # Example # Using 70% autoextend threshold and 20% autoextend size, when a 1G # thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds # 840M, it is extended to 1.44G: # thin_pool_autoextend_threshold = 70 thin_pool_autoextend_threshold=100
Inspect activation/thin_pool_autoextend_percent option.
$ sudo lvmconfig ---withcomment activation/thin_pool_autoextend_percent
# Configuration option activation/thin_pool_autoextend_percent. # Auto-extending a thin pool adds this percent extra space. # The amount of additional space added to a thin pool is this # percent of its current size. # Example # Using 70% autoextend threshold and 20% autoextend size, when a 1G # thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds # 840M, it is extended to 1.44G: # thin_pool_autoextend_percent = 20 thin_pool_autoextend_percent=20
Create a custom profile.
$ sudo lvmconfig --file /etc/lvm/profile/autoextend.profile --withcomments --config "activation/thin_pool_autoextend_threshold=60 activation/thin_pool_autoextend_percent=20"
Inspect created profile.
$ sudo cat /etc/lvm/profile/autoextend.profile
# Configuration section activation.
activation {
# Configuration option activation/thin_pool_autoextend_threshold.
# Auto-extend a thin pool when its usage exceeds this percent.
# Setting this to 100 disables automatic extension.
# The minimum value is 50 (a smaller value is treated as 50.)
# Also see thin_pool_autoextend_percent.
# Automatic extension requires dmeventd to be monitoring the LV.
# Example
# Using 70% autoextend threshold and 20% autoextend size, when a 1G
# thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
# 840M, it is extended to 1.44G:
# thin_pool_autoextend_threshold = 70
thin_pool_autoextend_threshold=60
# Configuration option activation/thin_pool_autoextend_percent.
# Auto-extending a thin pool adds this percent extra space.
# The amount of additional space added to a thin pool is this
# percent of its current size.
# Example
# Using 70% autoextend threshold and 20% autoextend size, when a 1G
# thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
# 840M, it is extended to 1.44G:
# thin_pool_autoextend_percent = 20
thin_pool_autoextend_percent=20
}
Apply profile to existing thin pool.
$ sudo lvchange --metadataprofile autoextend vg01/thin_pool
Logical volume vg01/thin_pool changed.
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 5.00g 6.77 1.58 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 3.76 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 3.76 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 3.76
Create a file.
$ sudo dd if=/dev/zero of=/opt/cerberus/file bs=1M count=2048
2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.76261 s, 777 MB/s
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 5.00g 46.18 1.60 [thin_pool_tdata] vg01 Twi-ao---- 5.00g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 69.45 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 3.76 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 3.76
Create a file.
$ sudo dd if=/dev/zero of=/opt/kraken/file bs=1M count=2048
2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.27238 s, 656 MB/s
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 7.62g 56.68 1.63 [thin_pool_tdata] vg01 Twi-ao---- 7.62g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 3.76 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 69.86
Create a file.
$ sudo dd if=/dev/zero of=/opt/hydra/file bs=1M count=2048
2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.94788 s, 728 MB/s
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 10.98g 57.71 1.65 [thin_pool_tdata] vg01 Twi-ao---- 10.98g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43
Extend thin pool or volume
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 1.00g thin_pool vg01 twi-aotz-- 10.98g 57.71 1.65 [thin_pool_tdata] vg01 Twi-ao---- 10.98g [thin_pool_tmeta] vg01 ewi-ao---- 1.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43
Extend pool metadata.
lvextend --poolmetadatasize +1G vg01/thin_pool
Size of logical volume vg01/thin_pool_tmeta changed from 1.00 GiB (256 extents) to 2.00 GiB (512 extents). Logical volume vg01/thin_pool_tmeta successfully resized.
Inspect logical volumes.
vagrant@bullseye:~$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 2.00g thin_pool vg01 twi-aotz-- 10.98g 57.71 0.83 [thin_pool_tdata] vg01 Twi-ao---- 10.98g [thin_pool_tmeta] vg01 ewi-ao---- 2.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43
Extend thin pool.
$ sudo lvextend --size +1G vg01/thin_pool
Size of logical volume vg01/thin_pool_tdata changed from 10.98 GiB (2812 extents) to 11.98 GiB (3068 extents). Logical volume vg01/thin_pool_tdata successfully resized.
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 2.00g thin_pool vg01 twi-aotz-- 11.98g 52.89 0.83 [thin_pool_tdata] vg01 Twi-ao---- 11.98g [thin_pool_tmeta] vg01 ewi-ao---- 2.00g volume_cerberus vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43
Extend thin volume.
$ sudo lvextend --size +1G vg01/volume_cerberus
Size of logical volume vg01/volume_cerberus changed from 3.00 GiB (768 extents) to 4.00 GiB (1024 extents). Logical volume vg01/volume_cerberus successfully resized.
Inspect logical volumes.
$ sudo lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] vg01 ewi------- 2.00g thin_pool vg01 twi-aotz-- 11.98g 52.89 0.83 [thin_pool_tdata] vg01 Twi-ao---- 11.98g [thin_pool_tmeta] vg01 ewi-ao---- 2.00g volume_cerberus vg01 Vwi-aotz-- 4.00g thin_pool 52.83 volume_hydra vg01 Vwi-aotz-- 3.00g thin_pool 70.43 volume_kraken vg01 Vwi-aotz-- 3.00g thin_pool 70.43
Extend filesystem on volume.
$ sudo resize2fs /dev/mapper/vg01-volume_cerberus
resize2fs 1.46.1 (9-Feb-2021) Filesystem at /dev/mapper/vg01-volume_cerberus is mounted on /opt/cerberus; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1 The filesystem on /dev/mapper/vg01-volume_cerberus is now 1048576 (4k) blocks long.
Inspect used space.
$ df -h /dev/mapper/vg01-volume_cerberus
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg01-volume_cerberus 3.9G 2.1G 1.7G 55% /opt/cerberu
Additional notes
Read lvm, lvmconfig,lvmthin manual pages.
Update
Thanks to Walter Dvorak for an update on this topic.
You can avoid the use of the dedicated fstrim command to free up unused blocks after the deletion of some data on the LVM-thin volumes simply by mount the thin volume with the option "discard": $ sudo mount -o discard /dev/mapper/vg01-volume_cerberus /opt/cerberus/ The option "discard" works also perfect if the physical device is a HDD (not a SSD). In this case the option "discard" is related to the specific features of thin LVM volume. it use just the same mechanism as for SSD.