Trace packets as they pass through the iptables firewall.

Initial configuration

Configure rsyslog to use /var/log/firewall_trace.log log file for firewall trace.

$ cat << EOF | sudo tee /etc/rsyslog.d/01-firewall_trace.conf
# Log messages generated by iptables firewall to file
if \$syslogfacility-text == 'kern' and \$msg contains 'TRACE' then /var/log/firewall_trace.log
# stop processing it further
& stop
EOF

Apply rsyslog configuration.

$ sudo systemctl restart rsyslog

Rotate log file to conserve disk space.

$ cat << EOF | sudo tee /etc/logrotate.d/firewall_trace.conf
/var/log/firewall_trace.log
{
  rotate 7
  daily
  missingok
  notifempty
  delaycompress
  compress
  postrotate
  invoke-rc.d rsyslog rotate > /dev/null
  endscript
}
EOF
You should definitely rotate this log file hourly by size or push to an external logging service, which I strongly suggest.

Trace incoming packets

Use raw table and PREROUTING to trace packets arriving via any network interface.

$ sudo iptables -t raw -A PREROUTING -p tcp --destination 1.2.3.4 --dport 443 -j TRACE

Inspect raw table.

$ sudo iptables -t raw -L -v -n --line-numbers

Chain PREROUTING (policy ACCEPT 3501 packets, 946K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      468 28159 TRACE      tcp  --  *      *       0.0.0.0/0            1.2.3.4       tcp dpt:443
Chain OUTPUT (policy ACCEPT 885 packets, 695K bytes)
num   pkts bytes target     prot opt in     out     source               destination

Trace to the internal network will look like this.

[...]
Jul 18 18:33:27 cerberus kernel: [68907.892027] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64783 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477568 WINDOW=82 RES=0x00 ACK URGP=0
Jul 18 18:33:27 cerberus kernel: [68907.892093] TRACE: mangle:INPUT:policy:1 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64783 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477568 WINDOW=82 RES=0x00 ACK URGP=0
Jul 18 18:33:27 cerberus kernel: [68907.892113] TRACE: filter:INPUT:rule:6 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64783 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477568 WINDOW=82 RES=0x00 ACK URGP=0
Jul 18 18:33:27 cerberus kernel: [68907.892150] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64784 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477569 WINDOW=82 RES=0x00 ACK RST URGP=0
[...]

Display table filter, chain INPUT, rule number 6 that will accept related and established connections.

$ sudo iptables -t filter -L INPUT 6 -v -n --line-numbers

6     979K  851M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Part of initial rule set */ ctstate RELATED,ESTABLISHED

Delete first rule in raw table, PREROUTING chain.

$ sudo iptables -t raw -D PREROUTING 1

Trace outgoing packets

Use raw table and OUTPUT to trace locally generated packets.

$ sudo iptables -t raw -A OUTPUT -p tcp --destination 8.8.8.8 --dport 53 -j TRACE

$ sudo iptables -t raw -A OUTPUT -p udp --destination 8.8.8.8 --dport 53 -j TRACE

Inspect raw table.

$ sudo iptables -t raw -L -v -n --line-numbers

Chain PREROUTING (policy ACCEPT 1281 packets, 422K bytes)
num   pkts bytes target     prot opt in     out     source               destination
Chain OUTPUT (policy ACCEPT 379 packets, 324K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 TRACE      tcp  --  *      *       0.0.0.0/0            8.8.8.8              tcp dpt:53
2        0     0 TRACE      udp  --  *      *       0.0.0.0/0            8.8.8.8              udp dpt:53

Trace to the external DNS server will look like this.

[...]
Jul 18 18:48:38 cerberus kernel: [69819.286907] TRACE: raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
Jul 18 18:48:38 cerberus kernel: [69819.286922] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
Jul 18 18:48:38 cerberus kernel: [69819.286929] TRACE: filter:OUTPUT:rule:7 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
Jul 18 18:48:38 cerberus kernel: [69819.286939] TRACE: nat:POSTROUTING:policy:2 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
[...]

Display table filter, chain OUTPUT, rule number 7 that will accept outgoing udp connections to port 53 everywhere.

$ sudo iptables -t filter -L OUTPUT 7 -v -n --line-numbers

7     2982  223K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* Part of initial rule set */

Delete first and second rule in raw table, OUTPUT chain.

$ sudo iptables -t raw -D PREROUTING 1

$ sudo iptables -t raw -D PREROUTING 2
ko-fi