It is time to mention about simple sequential or simultaneous upgrade process on lxd guests. It is adapted to Debian-like operating systems using apt as I am using these daily.
Distribution-specific information.
$ lsb_release -a
No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial
The container hypervisor client/daemon version.
$ lxc --version
2.0.10
As you can see, the following operations are performed on the Ubuntu 16.04 using the container hypervisor lxc 2.0.10.
Upgrade guests sequentially
Update package index.
for guest in $(sudo lxc list -c ns | awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do
sudo lxc exec $guest -- /bin/bash -c "echo $guest; apt-get update; echo;";
done | tee /tmp/guest_update.log
Inspect the /tmp/guest_update.log log file.
$ tail /tmp/guest_update.log
[...] dokuwiki-tests Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease Reading package lists... [...]
This step is purely optional, but you can list packages to be upgraded.
for guest in $(sudo lxc list -c ns | awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do
sudo lxc exec $guest -- /bin/bash -c "echo $guest; apt-get --dry-run dist-upgrade | grep ^Inst | cut -d -f2 | sort | tr 'n' ' '; echo;echo;";
done | tee /tmp/guest_upgrade_list.log
Inspect the /tmp/guest_upgrade_list.log log file.
$ tail /tmp/guest_upgrade_list.log
[...] dokuwiki-tests apt apt-transport-https apt-utils base-files binutils bsdutils btrfs-tools cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf coreutils iproute2 isc-dhcp-client isc-dhcp-common kmod less libapt-inst2.0 libapt-pkg5.0 libblkid1 libdrm2 libfdisk1 libkmod2 liblxc1 libmount1 libpam-systemd libphp7.0-embed libsmartcols1 libssl1.0.0 libsystemd0 libudev1 libuuid1 linux-libc-dev lxc-common lxcfs lxd lxd-client mount open-iscsi openssl overlayroot php7.0-common php7.0-intl php7.0-phpdbg php7.0-readline php7.0-sqlite3 php7.0-xml python3-distupgrade python3-software-properties python3-update-manager snap-confine snapd software-properties-common sudo systemd systemd-sysv ubuntu-core-launcher ubuntu-release-upgrader-core udev unattended-upgrades update-manager-core util-linux uuid-runtime vlan [...]
Perform the upgrade process.
for guest in $(sudo lxc list -c ns | awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do
sudo lxc exec $guest -- /bin/bash -c "export DEBIAN_FRONTEND=noninteractive; echo $guest; apt-get upgrade -y; echo;";
done | tee /tmp/guest_upgrade.log
Inspect the /tmp/guest_upgrade.log log file.
$ tail /tmp/guest_upgrade.log
[...] dokuwiki-tests [...] Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... The following package was automatically installed and is no longer required: snap-confine Use 'apt autoremove' to remove it. The following packages will be upgraded: apt apt-transport-https apt-utils base-files binutils bsdutils btrfs-tools cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf coreutils iproute2 isc-dhcp-client isc-dhcp-common kmod less libapt-inst2.0 libapt-pkg5.0 libblkid1 libdrm2 libfdisk1 libkmod2 liblxc1 libmount1 libpam-systemd libphp7.0-embed libsmartcols1 libssl1.0.0 libsystemd0 libudev1 libuuid1 linux-libc-dev lxc-common lxcfs lxd lxd-client mount open-iscsi openssl overlayroot php7.0-common php7.0-intl php7.0-phpdbg php7.0-readline php7.0-sqlite3 php7.0-xml python3-distupgrade python3-software-properties python3-update-manager snap-confine snapd software-properties-common sudo systemd systemd-sysv ubuntu-core-launcher ubuntu-release-upgrader-core udev unattended-upgrades update-manager-core util-linux uuid-runtime vlan 63 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 36.6 MB of archives. After this operation, 1794 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 base-files amd64 9.4ubuntu4.5 [68.4 kB] Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 bsdutils amd64 1:2.27.1-6ubuntu3.3 [52.0 kB] Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 coreutils amd64 8.25-2ubuntu3~16.04 [1174 kB] Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 util-linux amd64 2.27.1-6ubuntu3.3 [849 kB] [...] Hint: Some lines were ellipsized, use -l to show in full. Setting up snapd (2.25) ... Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ... Setting up ubuntu-core-launcher (2.25) ... Setting up snap-confine (2.25) ... Setting up iproute2 (4.3.0-1ubuntu3.16.04.1) ... Setting up isc-dhcp-client (4.3.3-5ubuntu12.7) ... Setting up isc-dhcp-common (4.3.3-5ubuntu12.7) ... [...]
You can use additional conditions inside awk program line to match only specific guests.
Upgrade guests in parallel
You can speed up a whole process by using the amazing parallel utility.
$ sudo apt-get install parallel
Update package index simultaneously on four guests.
sudo lxc list -c ns | \
awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}' | \
tr -d ' ' | \
parallel --jobs 4 --progress --no-run-if-empty \
--results /tmp/parallel_update/ --joblog /tmp/parallel_update_status.log \
sudo lxc exec {1} -- apt-get updatePerform the upgrade process simultaneously on four guests.
export DEBIAN_FRONTEND="noninteractive"; \
sudo lxc list -c ns | \
awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}' | \
tr -d ' ' | \
parallel --jobs 4 --progress --no-run-if-empty \
--results /tmp/parallel_upgrade/ --joblog /tmp/parallel_upgrade_status.log \
--env DEBIAN_FRONTEND \
sudo lxc exec {1} -- apt-get upgrade -yA list of the executed jobs will be stored in /tmp/parallel_update_status.log and /tmp/parallel_update_status.log log files.
$ cat /tmp/parallel_update_status.log
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command 1 : 1501795764.764 4.626 0 0 0 0 sudo lxc exec dokuwiki-tests -- apt-get update 4 : 1501795764.793 4.599 0 0 0 0 sudo lxc exec jira-tests -- apt-get update 5 : 1501795769.392 4.840 0 0 0 0 sudo lxc exec nextcloud-tests -- apt-get update 6 : 1501795769.395 4.841 0 0 0 0 sudo lxc exec wallabag-tests -- apt-get update 7 : 1501795774.236 4.055 0 0 0 0 sudo lxc exec wekan-tests -- apt-get update 2 : 1501795764.767 20.569 0 0 0 0 sudo lxc exec draw-tests -- apt-get update 3 : 1501795764.775 20.561 0 0 0 0 sudo lxc exec foswiki-tests -- apt-get update
Detailed logs will be located in the specified /tmp/parallel_update/ and /tmp/parallel_upgrade/ directories.
$ cat /tmp/parallel_update/1/jira-tests/std{err,out}
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB] Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Fetched 306 kB in 1s (239 kB/s) Reading package lists...