Inspect and solve non-obvious lack of free space when using lxd snap.
Display LXD snap version.
$ snap list lxd
Name Version Rev Tracking Publisher Notes lxd 4.0.4 18150 4.0/stable/… canonical✓ -
The symptoms
Identify the problem during the export operation.
$ sudo lxc export laboratory laboratory.tar.gz
Error: Create backup: Backup create: Error adding "/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/laboratory/dns/rootfs/var/opt/gitlab/backups/1596300891_2020_08_01_13.2.1_gitlab_backup.tar" as "backup/snapshots/dns/rootfs/var/opt/gitlab/backups/1596300891_2020_08_01_13.2.1_gitlab_backup.tar" to tarball: Failed to copy file content "/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/laboratory/dns/rootfs/var/opt/gitlab/backups/1596300891_2020_08_01_13.2.1_gitlab_backup.tar": io: read/write on closed pipe
Identify the problem during the import operation.
$ sudo lxc import laboratory.tar.gz
Importing instance: 39% (43.36MB/s)Error: write /var/snap/lxd/common/lxd/backups/lxd_backup_084016629: no space left on device
The issue
The issue is that the /var/snap/lxd/common/lxd/backups directory that is used during export/import operations is by default on the root filesystem.
$ sudo df -h /var/snap/lxd/common/lxd/
Filesystem Size Used Avail Use% Mounted on /dev/md0 46G 36G 7.8G 83% /
This is an issue when you want to perform the mentioned operations on virtual machines that use more disk space than a small amount available on the root filesystem.
The solution
The solution is to create a new pool. I will stick with ZFS but use LVM or whatever is available.
$ sudo zfs create tank/lxd_backup
Ensure that it will mount at boot. Use a proper solution or a bind mount.
$ echo "/tank/lxd_backup /var/snap/lxd/common/lxd/backups none bind,nofail 0 0" | sudo tee -a /etc/fstab
/tank/lxd_backup /var/snap/lxd/common/lxd/backups none bind,nofail 0 0
Mount it afterward.
$ mount /var/snap/lxd/common/lxd/backups
Restart lxd service, to ensure that it will take advantage of the new directory.
$ sudo snap restart lxd
Restarted.
Problem solved.
