Categories
SysOps

How to concatenate regular and gzip compressed files

Concatenate regular and gzip-compressed files at the same time.

Assume that we want to feed regular and gzip-compressed files to an unspecified applications.

$ ls /var/log/apt/history.log*
/var/log/apt/history.log  /var/log/apt/history.log.1.gz

It does not work at first.

$ zcat  /var/log/apt/history.log*
gzip: /var/log/apt/history.log: unexpected end of file

Inspect zcat help information.

$ zcat --help
Usage: /usr/bin/zcat [OPTION]... [FILE]...
Uncompress FILEs to standard output.

  -f, --force       force; read compressed data even from a terminal
  -l, --list        list compressed file contents
  -q, --quiet       suppress all warnings
  -r, --recursive   operate recursively on directories
  -S, --suffix=SUF  use suffix SUF on compressed files
      --synchronous synchronous output (safer if system crashes, but slower)
  -t, --test        test compressed file integrity
  -v, --verbose     verbose mode
      --help        display this help and exit
      --version     display version information and exit

With no FILE, or when FILE is -, read standard input.

Report bugs to <bug-gzip@gnu.org>.

Force execution, so the uncompressed data will be copied to the standard output.

$ zcat --force  /var/log/apt/history.log*
Start-Date: 2021-04-19  19:24:18
Commandline: apt install mariadb-server mariadb-client
Requested-By: vagrant (1000)
Install: libtimedate-perl:amd64 (2.3300-2, automatic), libhtml-tagset-perl:amd64 (3.20-4, automatic), mariadb-server:amd64 (1:10.5.9-1), libdbi-perl:amd64 (1.643-3+b1, automatic), libfcgi-perl:amd64 (0.79+ds-2, automatic), libdbd-mariadb-perl:amd64 (1.21-3, automatic), galera-4:amd64 (26.4.7-3, automatic), mysql-common:amd64 (5.8+1.0.7, automatic), libencode-locale-perl:amd64 (1.05-1.1, automatic), libsigsegv2:amd64 (2.13-1, automatic), liblwp-mediatypes-perl:amd64 (6.04-1, automatic), libmpfr6:amd64 (4.1.0-3, automatic), psmisc:amd64 (23.4-2, automatic), mariadb-server-core-10.5:amd64 (1:10.5.9-1, automatic), libhtml-template-perl:amd64 (2.97-1.1, automatic), gawk:amd64 (1:5.1.0-1, automatic), liburi-perl:amd64 (5.08-1, automatic), libfcgi0ldbl:amd64 (2.4.2-2, automatic), libio-html-perl:amd64 (1.004-2, automatic), libsnappy1v5:amd64 (1.1.8-1, automatic), libterm-readkey-perl:amd64 (2.38-1+b2, automatic), libhttp-date-perl:amd64 (6.05-1, automatic), rsync:amd64 (3.2.3-4, automatic), mariadb-client-10.5:amd64 (1:10.5.9-1, automatic), libhtml-parser-perl:amd64 (3.75-1+b1, automatic), libcgi-fast-perl:amd64 (1:2.15-1, automatic), libconfig-inifiles-perl:amd64 (3.000003-1, automatic), mariadb-common:amd64 (1:10.5.9-1, automatic), libmariadb3:amd64 (1:10.5.9-1, automatic), mariadb-client:amd64 (1:10.5.9-1), libclone-perl:amd64 (0.45-1+b1, automatic), socat:amd64 (1.7.4.1-3, automatic), mariadb-server-10.5:amd64 (1:10.5.9-1, automatic), libhttp-message-perl:amd64 (6.28-1, automatic), libfcgi-bin:amd64 (2.4.2-2, automatic), libcgi-pm-perl:amd64 (4.51-1, automatic), mariadb-client-core-10.5:amd64 (1:10.5.9-1, automatic)
End-Date: 2021-04-19  19:24:26

[...]

Start-Date: 2021-04-24  06:54:25
Commandline: /usr/bin/unattended-upgrade
Upgrade: dpkg:amd64 (1.20.7.1, 1.20.9)
End-Date: 2021-04-24  06:54:26

Start-Date: 2021-04-24  13:53:19
Commandline: apt install vim
Requested-By: vagrant (1000)
Install: vim:amd64 (2:8.2.2434-3), libgpm2:amd64 (1.20.7-8, automatic), vim-runtime:amd64 (2:8.2.2434-3, automatic)
End-Date: 2021-04-24  13:53:20

It works, but let’s not stop there and check this command more closely.

$ which zcat
/usr/bin/zcat

It is a shell script, which is interesting.

$ file  /usr/bin/zcat 
/usr/bin/zcat: POSIX shell script, ASCII text executable

Inspect this shell script.

$ cat /usr/bin/zcat
#!/bin/sh
# Uncompress files to standard output.

# Copyright (C) 2007, 2010-2018 Free Software Foundation, Inc.

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

version="zcat (gzip) 1.10
Copyright (C) 2007, 2011-2018 Free Software Foundation, Inc.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License .
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Eggert."

usage="Usage: $0 [OPTION]... [FILE]...
Uncompress FILEs to standard output.

  -f, --force       force; read compressed data even from a terminal
  -l, --list        list compressed file contents
  -q, --quiet       suppress all warnings
  -r, --recursive   operate recursively on directories
  -S, --suffix=SUF  use suffix SUF on compressed files
      --synchronous synchronous output (safer if system crashes, but slower)
  -t, --test        test compressed file integrity
  -v, --verbose     verbose mode
      --help        display this help and exit
      --version     display version information and exit

With no FILE, or when FILE is -, read standard input.

Report bugs to <bug-gzip@gnu.org>."

case $1 in
--help)    printf '%s\n' "$usage"   || exit 1; exit;;
--version) printf '%s\n' "$version" || exit 1; exit;;
esac

exec gzip -cd "$@"

It is simply executing gzip utility, so you use it directly to get the same effect.

$ gzip --stdout --decompress --force /var/log/apt/history.log*
Start-Date: 2021-04-19  19:24:18
Commandline: apt install mariadb-server mariadb-client
Requested-By: vagrant (1000)
Install: libtimedate-perl:amd64 (2.3300-2, automatic), libhtml-tagset-perl:amd64 (3.20-4, automatic), mariadb-server:amd64 (1:10.5.9-1), libdbi-perl:amd64 (1.643-3+b1, automatic), libfcgi-perl:amd64 (0.79+ds-2, automatic), libdbd-mariadb-perl:amd64 (1.21-3, automatic), galera-4:amd64 (26.4.7-3, automatic), mysql-common:amd64 (5.8+1.0.7, automatic), libencode-locale-perl:amd64 (1.05-1.1, automatic), libsigsegv2:amd64 (2.13-1, automatic), liblwp-mediatypes-perl:amd64 (6.04-1, automatic), libmpfr6:amd64 (4.1.0-3, automatic), psmisc:amd64 (23.4-2, automatic), mariadb-server-core-10.5:amd64 (1:10.5.9-1, automatic), libhtml-template-perl:amd64 (2.97-1.1, automatic), gawk:amd64 (1:5.1.0-1, automatic), liburi-perl:amd64 (5.08-1, automatic), libfcgi0ldbl:amd64 (2.4.2-2, automatic), libio-html-perl:amd64 (1.004-2, automatic), libsnappy1v5:amd64 (1.1.8-1, automatic), libterm-readkey-perl:amd64 (2.38-1+b2, automatic), libhttp-date-perl:amd64 (6.05-1, automatic), rsync:amd64 (3.2.3-4, automatic), mariadb-client-10.5:amd64 (1:10.5.9-1, automatic), libhtml-parser-perl:amd64 (3.75-1+b1, automatic), libcgi-fast-perl:amd64 (1:2.15-1, automatic), libconfig-inifiles-perl:amd64 (3.000003-1, automatic), mariadb-common:amd64 (1:10.5.9-1, automatic), libmariadb3:amd64 (1:10.5.9-1, automatic), mariadb-client:amd64 (1:10.5.9-1), libclone-perl:amd64 (0.45-1+b1, automatic), socat:amd64 (1.7.4.1-3, automatic), mariadb-server-10.5:amd64 (1:10.5.9-1, automatic), libhttp-message-perl:amd64 (6.28-1, automatic), libfcgi-bin:amd64 (2.4.2-2, automatic), libcgi-pm-perl:amd64 (4.51-1, automatic), mariadb-client-core-10.5:amd64 (1:10.5.9-1, automatic)
End-Date: 2021-04-19  19:24:26

[...]

Start-Date: 2021-04-24  06:54:25
Commandline: /usr/bin/unattended-upgrade
Upgrade: dpkg:amd64 (1.20.7.1, 1.20.9)
End-Date: 2021-04-24  06:54:26

Start-Date: 2021-04-24  13:53:19
Commandline: apt install vim
Requested-By: vagrant (1000)
Install: vim:amd64 (2:8.2.2434-3), libgpm2:amd64 (1.20.7-8, automatic), vim-runtime:amd64 (2:8.2.2434-3, automatic)
End-Date: 2021-04-24  13:53:20

It is a small thing, but an interesting one.