Categories
SysOps

How to use LVM thin provisioning

Configure and use LVM thin provisioning.

Thin volumes are useful for overprovisioning especially for containers.

Preparations

Install LVM utilities.

$ sudo apt install lvm2

Initialize physical volume.

$ sudo pvcreate  /dev/sdb
  Physical volume "/dev/sdb" successfully created.

Create a volume group.

$ sudo vgcreate vg01 /dev/sdb
  Volume group "vg01" successfully created

Create thin pool

Create thin pool.

$ sudo lvcreate --thin --size 5G --chunksize 256K --poolmetadatasize 1G vg01/thin_pool
  Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
  Logical volume "thin_pool" created.

Display information about logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                    
  thin_pool         vg01 twi-a-tz-- 5.00g             0.00   1.57                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                    
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g 

Create thin volumes

Create thin volumes.

$ sudo lvcreate --thinpool vg01/thin_pool --name volume_cerberus  --virtualsize 3G
  Logical volume "volume_cerberus" created.
$ sudo lvcreate --thinpool vg01/thin_pool --name volume_kraken  --virtualsize 3G
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool vg01/thin_pool (5.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "volume_kraken" created.
$ sudo lvcreate --thinpool vg01/thin_pool --name volume_hydra  --virtualsize 3G
  WARNING: Sum of all thin volume sizes (9.00 GiB) exceeds the size of thin pool vg01/thin_pool (5.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "volume_hydra" created

Create an ext4 filesystem.

$ sudo mkfs.ext4 /dev/mapper/vg01-volume_cerberus 
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done                            
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 075ebaeb-36ef-4532-9826-7b926d3a2186
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 
$ sudo mkfs.ext4 /dev/mapper/vg01-volume_kraken
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done                            
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: d6fa7e01-5262-43d7-9718-d10a0e109cff
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 
$ sudo mkfs.ext4 /dev/mapper/vg01-volume_hydra
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done                            
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: fc9f511f-4310-4318-9b97-07b594e32dfb
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

Create mount directories.

$ sudo mkdir /opt/{cerberus,kraken,hydra}

Mount thin volumes.

$ sudo mount /dev/mapper/vg01-volume_cerberus /opt/cerberus/
$ sudo mount /dev/mapper/vg01-volume_kraken /opt/kraken/
$ sudo mount /dev/mapper/vg01-volume_hydra /opt/hydra/

Inspect disk space usage.

$ df -h /dev/mapper/vg01-volume_*
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg01-volume_cerberus  2.9G   24K  2.8G   1% /opt/cerberus
/dev/mapper/vg01-volume_hydra     2.9G   24K  2.8G   1% /opt/hydra
/dev/mapper/vg01-volume_kraken    2.9G   24K  2.8G   1% /opt/kraken

Trigger out of space situation

Create files to trigger out of space situation.

$ sudo dd if=/dev/zero of=/opt/cerberus/file bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.2239 s, 966 MB/s
$ sudo dd if=/dev/zero of=/opt/kraken/file bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.01049 s, 713 MB/s
$ sudo dd if=/dev/zero of=/opt/hydra/file bs=1M count=2048
^C
Killed

Inspect disk space usage.

$ df -h /dev/mapper/vg01-volume_*
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg01-volume_cerberus  2.9G  2.1G  734M  74% /opt/cerberus
/dev/mapper/vg01-volume_kraken    2.9G  2.1G  734M  74% /opt/kraken
/dev/mapper/vg01-volume_hydra     2.9G  745M  2.0G  27% /opt/hydra
$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                         
  thin_pool         vg01 twi-aotzD- 5.00g                  100.00 1.63                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                         
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool        25.80                                  
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool        70.43 

Reclaim used space

Delete created files.

$ sudo rm /opt/{cerberus,kraken,hydra}/file

Inspect disk space usage.

$ df -h /dev/mapper/vg01-volume_*
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg01-volume_cerberus  2.9G   24K  2.8G   1% /opt/cerberus
/dev/mapper/vg01-volume_hydra     2.9G   24K  2.8G   1% /opt/hydra
/dev/mapper/vg01-volume_kraken    2.9G   24K  2.8G   1% /opt/kraken

Notice, volume usage does not change.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                         
  thin_pool         vg01 twi-aotzD- 5.00g                  100.00 1.63                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                         
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool        25.80                                  
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool        70.43                                  
vagrant@bullseye:~$ 

Discard unused blocks on mounted filesystems.

$ sudo fstrim /opt/cerberus 
$ sudo fstrim /opt/kraken 
$ sudo fstrim /opt/hydra 

Inspect volume usage.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                         
  thin_pool         vg01 twi-aotz-- 5.00g                  6.76   1.58                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                         
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool        3.76                                   
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool        3.75                                   
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool        3.76  

Create snaphot

Create sample file.

$ sudo dd if=/dev/zero of=/opt/cerberus/file bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.7677 s, 776 MB/s

Create a snapshot.

$ sudo lvcreate --name cerberus_snapshot --snapshot vg01/volume_cerberus
  WARNING: Sum of all thin volume sizes (12.00 GiB) exceeds the size of thin pool vg01/thin_pool (5.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "cerberus_snapshot" created.

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin          Data%  Meta%  Move Log Cpy%Sync Convert
  cerberus_snapshot vg01 Vwi---tz-k 3.00g thin_pool volume_cerberus                                        
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                                  
  thin_pool         vg01 twi-aotz-- 5.00g                           46.77  1.60                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                                  
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                                  
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool                 70.43                                  
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool                 3.75                                   
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool                 3.76   

Delete created file.

$ sudo rm /opt/cerberus/file 

Discard unused blocks on mounted filesystems.

$ sudo fstrim /opt/cerberus 

Activate the snapshot.

$ sudo lvchange --activate y --ignoreactivationskip vg01/cerberus_snapshot

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin          Data%  Meta%  Move Log Cpy%Sync Convert
  cerberus_snapshot vg01 Vwi-a-tz-k 3.00g thin_pool volume_cerberus 70.43                                  
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                                  
  thin_pool         vg01 twi-aotz-- 5.00g                           46.79  1.60                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                                  
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                                  
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool                 70.43                                  
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool                 3.75                                   
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool                 3.76

Create mount point.

$ sudo mkdir /opt/cerberus_snapshot

Mount snapshot.

$ sudo mount /dev/mapper/vg01-cerberus_snapshot /opt/cerberus_snapshot/

Inspect snapshot.

$ ls /opt/cerberus_snapshot/
file  lost+found

Autoextend thin pool

Inspect activation/thin_pool_autoextend_threshold option.

$ sudo lvmconfig ---withcomment activation/thin_pool_autoextend_threshold
# Configuration option activation/thin_pool_autoextend_threshold.
# Auto-extend a thin pool when its usage exceeds this percent.
# Setting this to 100 disables automatic extension.
# The minimum value is 50 (a smaller value is treated as 50.)
# Also see thin_pool_autoextend_percent.
# Automatic extension requires dmeventd to be monitoring the LV.
# Example
# Using 70% autoextend threshold and 20% autoextend size, when a 1G
# thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
# 840M, it is extended to 1.44G:
# thin_pool_autoextend_threshold = 70
thin_pool_autoextend_threshold=100

Inspect activation/thin_pool_autoextend_percent option.

$ sudo lvmconfig ---withcomment activation/thin_pool_autoextend_percent
# Configuration option activation/thin_pool_autoextend_percent.
# Auto-extending a thin pool adds this percent extra space.
# The amount of additional space added to a thin pool is this
# percent of its current size.
# Example
# Using 70% autoextend threshold and 20% autoextend size, when a 1G
# thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
# 840M, it is extended to 1.44G:
# thin_pool_autoextend_percent = 20
thin_pool_autoextend_percent=20

Create a custom profile.

$ sudo lvmconfig --file /etc/lvm/profile/autoextend.profile --withcomments --config "activation/thin_pool_autoextend_threshold=60 activation/thin_pool_autoextend_percent=20"

Inspect created profile.

$ sudo cat /etc/lvm/profile/autoextend.profile 
# Configuration section activation.
activation {

        # Configuration option activation/thin_pool_autoextend_threshold.
        # Auto-extend a thin pool when its usage exceeds this percent.
        # Setting this to 100 disables automatic extension.
        # The minimum value is 50 (a smaller value is treated as 50.)
        # Also see thin_pool_autoextend_percent.
        # Automatic extension requires dmeventd to be monitoring the LV.
        # Example
        # Using 70% autoextend threshold and 20% autoextend size, when a 1G
        # thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
        # 840M, it is extended to 1.44G:
        # thin_pool_autoextend_threshold = 70
        thin_pool_autoextend_threshold=60

        # Configuration option activation/thin_pool_autoextend_percent.
        # Auto-extending a thin pool adds this percent extra space.
        # The amount of additional space added to a thin pool is this
        # percent of its current size.
        # Example
        # Using 70% autoextend threshold and 20% autoextend size, when a 1G
        # thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
        # 840M, it is extended to 1.44G:
        # thin_pool_autoextend_percent = 20
        thin_pool_autoextend_percent=20
}

Apply profile to existing thin pool.

$ sudo lvchange --metadataprofile autoextend vg01/thin_pool
  Logical volume vg01/thin_pool changed.

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                         
  thin_pool         vg01 twi-aotz-- 5.00g                  6.77   1.58                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                         
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool        3.76                                   
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool        3.76                                   
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool        3.76

Create a file.

$ sudo dd if=/dev/zero of=/opt/cerberus/file bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.76261 s, 777 MB/s

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                         
  thin_pool         vg01 twi-aotz-- 5.00g                  46.18  1.60                            
  [thin_pool_tdata] vg01 Twi-ao---- 5.00g                                                         
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool        69.45                                  
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool        3.76                                   
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool        3.76 

Create a file.

$ sudo dd if=/dev/zero of=/opt/kraken/file bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.27238 s, 656 MB/s

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi------- 1.00g                                                         
  thin_pool         vg01 twi-aotz-- 7.62g                  56.68  1.63                            
  [thin_pool_tdata] vg01 Twi-ao---- 7.62g                                                         
  [thin_pool_tmeta] vg01 ewi-ao---- 1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz-- 3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz-- 3.00g thin_pool        3.76                                   
  volume_kraken     vg01 Vwi-aotz-- 3.00g thin_pool        69.86 

Create a file.

$ sudo dd if=/dev/zero of=/opt/hydra/file bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.94788 s, 728 MB/s

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize  Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi-------  1.00g                                                         
  thin_pool         vg01 twi-aotz-- 10.98g                  57.71  1.65                            
  [thin_pool_tdata] vg01 Twi-ao---- 10.98g                                                         
  [thin_pool_tmeta] vg01 ewi-ao----  1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_kraken     vg01 Vwi-aotz--  3.00g thin_pool        70.43

Extend thin pool or volume

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize  Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi-------  1.00g                                                         
  thin_pool         vg01 twi-aotz-- 10.98g                  57.71  1.65                            
  [thin_pool_tdata] vg01 Twi-ao---- 10.98g                                                         
  [thin_pool_tmeta] vg01 ewi-ao----  1.00g                                                         
  volume_cerberus   vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_kraken     vg01 Vwi-aotz--  3.00g thin_pool        70.43  

Extend pool metadata.

lvextend --poolmetadatasize +1G vg01/thin_pool
  Size of logical volume vg01/thin_pool_tmeta changed from 1.00 GiB (256 extents) to 2.00 GiB (512 extents).
  Logical volume vg01/thin_pool_tmeta successfully resized.

Inspect logical volumes.

vagrant@bullseye:~$ sudo lvs -a
  LV                VG   Attr       LSize  Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi-------  2.00g                                                         
  thin_pool         vg01 twi-aotz-- 10.98g                  57.71  0.83                            
  [thin_pool_tdata] vg01 Twi-ao---- 10.98g                                                         
  [thin_pool_tmeta] vg01 ewi-ao----  2.00g                                                         
  volume_cerberus   vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_kraken     vg01 Vwi-aotz--  3.00g thin_pool        70.43 

Extend thin pool.

$ sudo lvextend --size +1G vg01/thin_pool
  Size of logical volume vg01/thin_pool_tdata changed from 10.98 GiB (2812 extents) to 11.98 GiB (3068 extents).
  Logical volume vg01/thin_pool_tdata successfully resized.

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize  Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi-------  2.00g                                                         
  thin_pool         vg01 twi-aotz-- 11.98g                  52.89  0.83                            
  [thin_pool_tdata] vg01 Twi-ao---- 11.98g                                                         
  [thin_pool_tmeta] vg01 ewi-ao----  2.00g                                                         
  volume_cerberus   vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_hydra      vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_kraken     vg01 Vwi-aotz--  3.00g thin_pool        70.43 

Extend thin volume.

sudo lvextend --size +1G vg01/volume_cerberus
  Size of logical volume vg01/volume_cerberus changed from 3.00 GiB (768 extents) to 4.00 GiB (1024 extents).
  Logical volume vg01/volume_cerberus successfully resized.

Inspect logical volumes.

$ sudo lvs -a
  LV                VG   Attr       LSize  Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0_pmspare]   vg01 ewi-------  2.00g                                                         
  thin_pool         vg01 twi-aotz-- 11.98g                  52.89  0.83                            
  [thin_pool_tdata] vg01 Twi-ao---- 11.98g                                                         
  [thin_pool_tmeta] vg01 ewi-ao----  2.00g                                                         
  volume_cerberus   vg01 Vwi-aotz--  4.00g thin_pool        52.83                                  
  volume_hydra      vg01 Vwi-aotz--  3.00g thin_pool        70.43                                  
  volume_kraken     vg01 Vwi-aotz--  3.00g thin_pool        70.43

Extend filesystem on volume.

$ sudo resize2fs /dev/mapper/vg01-volume_cerberus 
resize2fs 1.46.1 (9-Feb-2021)
Filesystem at /dev/mapper/vg01-volume_cerberus is mounted on /opt/cerberus; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/vg01-volume_cerberus is now 1048576 (4k) blocks long.

Inspect used space.

$ df -h /dev/mapper/vg01-volume_cerberus 
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg01-volume_cerberus  3.9G  2.1G  1.7G  55% /opt/cerberu

Additional notes

Read lvm, lvmconfig,lvmthin manual pages.