How to secure Consul communication

Secure Consul communication.

Generate encryption key

Generate encryption key that will be used to encrypt gossip protocol.

$ consul keygen

wZHOCKakzCeAesu7HK07tqmc3PwJojN/jNfbXEDqplI=

Generate certificates

Create ssl directory.

$ sudo install --directory --group consul --owner consul --mode 700 /opt/consul/ssl

Create CA.

$ consul tls ca create

==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

$ cat consul-agent-ca.pem

$ cat consul-agent-ca-key.pem

Create server certificate.

$ consul tls cert create -server -dc dc-lab-1

==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc-lab-1-server-consul-0.pem
==> Saved dc-lab-1-server-consul-0-key.pem

$ cat dc-lab-1-server-consul-0.pem

$ cat dc-lab-1-server-consul-0-key.pem

Create client certificate.

$ consul tls cert create -client -dc dc-lab-1

==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc-lab-1-client-consul-0.pem
==> Saved dc-lab-1-client-consul-0-key.pem

$ cat dc-lab-1-client-consul-0.pem

$ cat dc-lab-1-client-consul-0-key.pem

Create cli certificate.

$ consul tls cert create -cli -dc dc-lab-1

==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc-lab-1-cli-consul-0.pem
==> Saved dc-lab-1-cli-consul-0-key.pem

$ cat dc-lab-1-cli-consul-0.pem

$ cat dc-lab-1-cli-consul-0-key.pem

Enable encryption on server

Store CA and server certificate.

$ sudo -u consul tee /opt/consul/ssl/consul-agent-ca.pem << EOF
-----BEGIN CERTIFICATE-----
MIIC7DCCApOgAwIBAgIQb+uO2QIbIaeohiZetp1DhTAKBggqhkjOPQQDAjCBuTEL
MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRowGAYDVQQJExExMDEgU2Vjb25kIFN0cmVldDEOMAwGA1UEERMFOTQxMDUxFzAV
BgNVBAoTDkhhc2hpQ29ycCBJbmMuMUAwPgYDVQQDEzdDb25zdWwgQWdlbnQgQ0Eg
MTQ4NzY3Mzk0NTg5MjA4MTAwNDY1MjA0NjI0NDAzNjQyMjcwNTk3MB4XDTIyMDcx
NDIwNTY0N1oXDTI3MDcxMzIwNTY0N1owgbkxCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJDQTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNjbzEaMBgGA1UECRMRMTAxIFNlY29u
ZCBTdHJlZXQxDjAMBgNVBBETBTk0MTA1MRcwFQYDVQQKEw5IYXNoaUNvcnAgSW5j
LjFAMD4GA1UEAxM3Q29uc3VsIEFnZW50IENBIDE0ODc2NzM5NDU4OTIwODEwMDQ2
NTIwNDYyNDQwMzY0MjI3MDU5NzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKJe
6FzEXbOGS4HoNOlEj8WhytPdkILGjRNPk7B5jsB/dpO9JgL2RFI5SJJBwXHug2kI
gEzh5NEjcVmtsJhAEy6jezB5MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTAD
AQH/MCkGA1UdDgQiBCD+9xE/gAdmCws+5CUq5595mEeBLFIsS2WFr5UniPGIUjAr
BgNVHSMEJDAigCD+9xE/gAdmCws+5CUq5595mEeBLFIsS2WFr5UniPGIUjAKBggq
hkjOPQQDAgNHADBEAiB6zHbLgTWX5elXaua65zOHclb5FUTqDsPtVjLd1F7+lgIg
WhGwFk9pnBhreN+joMdVf3aWw3zSlrbm0qDKinDryHs=
-----END CERTIFICATE-----
EOF

$ sudo -u consul tee /opt/consul/ssl/dc-lab-1-server-consul-0.pem << EOF
-----BEGIN CERTIFICATE-----
MIICpzCCAkygAwIBAgIQBDwlHs1ohymMKvFSs1ORpjAKBggqhkjOPQQDAjCBuTEL
MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRowGAYDVQQJExExMDEgU2Vjb25kIFN0cmVldDEOMAwGA1UEERMFOTQxMDUxFzAV
BgNVBAoTDkhhc2hpQ29ycCBJbmMuMUAwPgYDVQQDEzdDb25zdWwgQWdlbnQgQ0Eg
MTQ4NzY3Mzk0NTg5MjA4MTAwNDY1MjA0NjI0NDAzNjQyMjcwNTk3MB4XDTIyMDcx
NDIxMDIxN1oXDTIzMDcxNDIxMDIxN1owITEfMB0GA1UEAxMWc2VydmVyLmRjLWxh
Yi0xLmNvbnN1bDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABOlJiHyw7RfMD4Ov
hFZxKlpdQdCawAyL41BBiukjKqi7bG1t1bYUX4ltBlA0j2XpBlKps6wg9gPFi0jg
ywfnQkGjgcwwgckwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMB
BggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMCkGA1UdDgQiBCAhVpOfcmtkv726+NmJ
yY9LUSczBdlNR/TxDxoXeJOTSzArBgNVHSMEJDAigCD+9xE/gAdmCws+5CUq5595
mEeBLFIsS2WFr5UniPGIUjAyBgNVHREEKzApghZzZXJ2ZXIuZGMtbGFiLTEuY29u
c3Vsgglsb2NhbGhvc3SHBH8AAAEwCgYIKoZIzj0EAwIDSQAwRgIhAOJ4TK1ljy/x
4FIe7TnpgHwxQ58C/NWdwXANhmfyFYwYAiEA+8U57t2UsN7RwYwKIDj4MaNgedcN
CCXW9/oP9wSAW0Y=
-----END CERTIFICATE-----
EOF

$ sudo -u consul tee /opt/consul/ssl/dc-lab-1-server-consul-0-key.pem << EOF
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIO8Cxt63TIL2SFUpVuq+eQkedX2uXkPorJrEtBm9Z/YJoAoGCCqGSM49
AwEHoUQDQgAE6UmIfLDtF8wPg6+EVnEqWl1B0JrADIvjUEGK6SMqqLtsbW3VthRf
iW0GUDSPZekGUqmzrCD2A8WLSODLB+dCQQ==
-----END EC PRIVATE KEY-----
EOF

Try to not store certificates in /etc/consul.d as it will create unnecessary noise.

[...]
skipping file /etc/consul.d/consul-agent-ca.pem, extension must be .hcl or .json, or config format must be set
skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
skipping file /etc/consul.d/dc-lab-1-server-consul-0-key.pem, extension must be .hcl or .json, or config format must be set
skipping file /etc/consul.d/dc-lab-1-server-consul-0.pem, extension must be .hcl or .json, or config format must be set
[...]

Update server configuration to include encryption.

$ sudo -u consul tee /etc/consul.d/encryption.hcl << EOF
# encrypt gossip protocol
encrypt = "AcHUV+z4kLDJiQeVqLAh2sG25SH4K4WYU6oIru29lSM="

# tls encryption
tls {
  defaults{
    verify_incoming = true
    verify_outgoing = true
    ca_file = "/opt/consul/ssl/consul-agent-ca.pem"
    cert_file = "/opt/consul/ssl/dc-lab-1-server-consul-0.pem"
    key_file = "/opt/consul/ssl/dc-lab-1-server-consul-0-key.pem"
  }
}
EOF

Validate configuration.

$ sudo -u consul consul validate /etc/consul.d

skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
bootstrap = true: do not enable unless necessary
Configuration is valid!

Restart service.

$ sudo systemctl restart consul

You can later use consul reload command, but you cannot enable or disable TLS with reloading.

Enable encryption on agent

Store CA and client certificate.

$ sudo -u consul tee /opt/consul/ssl/consul-agent-ca.pem << EOF
-----BEGIN CERTIFICATE-----
MIIC7DCCApOgAwIBAgIQb+uO2QIbIaeohiZetp1DhTAKBggqhkjOPQQDAjCBuTEL
MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRowGAYDVQQJExExMDEgU2Vjb25kIFN0cmVldDEOMAwGA1UEERMFOTQxMDUxFzAV
BgNVBAoTDkhhc2hpQ29ycCBJbmMuMUAwPgYDVQQDEzdDb25zdWwgQWdlbnQgQ0Eg
MTQ4NzY3Mzk0NTg5MjA4MTAwNDY1MjA0NjI0NDAzNjQyMjcwNTk3MB4XDTIyMDcx
NDIwNTY0N1oXDTI3MDcxMzIwNTY0N1owgbkxCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJDQTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNjbzEaMBgGA1UECRMRMTAxIFNlY29u
ZCBTdHJlZXQxDjAMBgNVBBETBTk0MTA1MRcwFQYDVQQKEw5IYXNoaUNvcnAgSW5j
LjFAMD4GA1UEAxM3Q29uc3VsIEFnZW50IENBIDE0ODc2NzM5NDU4OTIwODEwMDQ2
NTIwNDYyNDQwMzY0MjI3MDU5NzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKJe
6FzEXbOGS4HoNOlEj8WhytPdkILGjRNPk7B5jsB/dpO9JgL2RFI5SJJBwXHug2kI
gEzh5NEjcVmtsJhAEy6jezB5MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTAD
AQH/MCkGA1UdDgQiBCD+9xE/gAdmCws+5CUq5595mEeBLFIsS2WFr5UniPGIUjAr
BgNVHSMEJDAigCD+9xE/gAdmCws+5CUq5595mEeBLFIsS2WFr5UniPGIUjAKBggq
hkjOPQQDAgNHADBEAiB6zHbLgTWX5elXaua65zOHclb5FUTqDsPtVjLd1F7+lgIg
WhGwFk9pnBhreN+joMdVf3aWw3zSlrbm0qDKinDryHs=
-----END CERTIFICATE-----
EOF

$ sudo -u consul tee /opt/consul/ssl/dc-lab-1-client-consul-0.pem << EOF
-----BEGIN CERTIFICATE-----
MIICpjCCAkygAwIBAgIQK+uPhvt9xtpiPopnwx6yWjAKBggqhkjOPQQDAjCBuTEL
MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRowGAYDVQQJExExMDEgU2Vjb25kIFN0cmVldDEOMAwGA1UEERMFOTQxMDUxFzAV
BgNVBAoTDkhhc2hpQ29ycCBJbmMuMUAwPgYDVQQDEzdDb25zdWwgQWdlbnQgQ0Eg
MTQ4NzY3Mzk0NTg5MjA4MTAwNDY1MjA0NjI0NDAzNjQyMjcwNTk3MB4XDTIyMDcx
NDIxMDIzOVoXDTIzMDcxNDIxMDIzOVowITEfMB0GA1UEAxMWY2xpZW50LmRjLWxh
Yi0xLmNvbnN1bDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABMjDA/gRuyqJTTkJ
fg09JorVQYxRWUCfK3nrv5eeo0HmY/QjsFDwQhD38XGyJ1mqsaDOOjfZRe1HZMIH
JZ/g7O6jgcwwgckwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMC
BggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCkGA1UdDgQiBCDOSYswbdIIcconfobw
SyT8dpIaTfi6fQcG3vSe3XrYfDArBgNVHSMEJDAigCD+9xE/gAdmCws+5CUq5595
mEeBLFIsS2WFr5UniPGIUjAyBgNVHREEKzApghZjbGllbnQuZGMtbGFiLTEuY29u
c3Vsgglsb2NhbGhvc3SHBH8AAAEwCgYIKoZIzj0EAwIDSAAwRQIhALGk2J5al51i
gcWll0QqESuai19Tq2U24ZxQmjOMA5TfAiAeSaiqX37uSrGbZaPKUVrGvX7P74Hx
37XXD/9WqUU0qg==
-----END CERTIFICATE-----
EOF

$ sudo -u consul tee /opt/consul/ssl/dc-lab-1-client-consul-0-key.pem << EOF
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEII3kkrtfjELw5sdtaQSuxR29xCkKdH43XFRbpCeKZZVwoAoGCCqGSM49
AwEHoUQDQgAEyMMD+BG7KolNOQl+DT0mitVBjFFZQJ8reeu/l56jQeZj9COwUPBC
EPfxcbInWaqxoM46N9lF7Udkwgcln+Ds7g==
-----END EC PRIVATE KEY-----
EOF

Update agent configuration.

$ sudo -u consul tee /etc/consul.d/encryption.hcl << EOF
# encrypt gossip protocol
encrypt = "AcHUV+z4kLDJiQeVqLAh2sG25SH4K4WYU6oIru29lSM="

# tls encryption
tls {
  defaults{
    verify_incoming = true
    verify_outgoing = true
    ca_file = "/opt/consul/ssl/consul-agent-ca.pem"
    cert_file = "/opt/consul/ssl/dc-lab-1-client-consul-0.pem"
    key_file = "/opt/consul/ssl/dc-lab-1-client-consul-0-key.pem"    
  }
}
EOF

Validate configuration.

$ sudo -u consul consul validate /etc/consul.d

skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Configuration is valid!

Restart service.

$ sudo systemctl restart consul

Configuration reload triggered

List cluster members.

$ consul members

Node   Address              Status  Type    Build   Protocol  DC        Partition  Segment
jammy  172.16.151.111:8301  alive   server  1.12.3  2         dc-lab-1  default    <all>
mgmt   172.16.151.120:8301  alive   client  1.12.3  2         dc-lab-1  default    <default>

Use auto-encryption to simplify certificate provisioning

Use auto-encryption to automatically distribute client certificates to agents.

Update server configuration.

$ sudo -u consul tee /etc/consul.d/encryption.hcl << EOF
# encrypt gossip protocol
encrypt = "AcHUV+z4kLDJiQeVqLAh2sG25SH4K4WYU6oIru29lSM="

# tls encryption
tls {
  defaults{
    verify_incoming = true
    verify_outgoing = true
    ca_file = "/opt/consul/ssl/consul-agent-ca.pem"
    cert_file = "/opt/consul/ssl/dc-lab-1-server-consul-0.pem"
    key_file = "/opt/consul/ssl/dc-lab-1-server-consul-0-key.pem"
  }
}

# Auto-Encrypt-TLS
auto_encrypt {
  allow_tls = true
}
EOF

Update agent configuration.

$ sudo -u consul tee /etc/consul.d/encryption.hcl << EOF
# encrypt gossip protocol
encrypt = "AcHUV+z4kLDJiQeVqLAh2sG25SH4K4WYU6oIru29lSM="

# tls encryption
tls {
  defaults{
    verify_incoming = true
    verify_outgoing = true
    ca_file = "/opt/consul/ssl/consul-agent-ca.pem"
  }
}

# Auto-Encrypt-TLS
auto_encrypt = {
  tls = true
}
EOF

CLI over encrypted connection

You can enable HTTPS protocol inside consul configuration file.

[...]

ports {
  http = -1
  https = 8501
}

[...]

Use additional command-line parameters to specify ca, certificates and address.

$ consul members -ca-file /opt/consul/ssl/consul-agent-ca.pem \
                 -client-cert /opt/consul/ssl/dc-lab-1-cli-consul-0.pem \
                 -client-key /opt/consul/ssl/dc-lab-1-cli-consul-0-key.pem \
                 -http-addr https://127.0.0.1:8501

Node   Address              Status  Type    Build   Protocol  DC        Partition  Segment
jammy  172.16.151.111:8301  alive   server  1.12.3  2         dc-lab-1  default    <all>
mgmt   172.16.151.120:8301  alive   client  1.12.3  2         dc-lab-1  default    <default>

$ export CONSUL_HTTP_SSL=true
$ export CONSUL_HTTP_ADDR=https://127.0.0.1:8501
$ export CONSUL_CACERT=/opt/ssl/consul-agent-ca.pem 
$ export CONSUL_CLIENT_CERT=/opt/ssl/dc-lab-1-cli-consul-0.pem
$ export CONSUL_CLIENT_KEY=/opt/ssl/dc-lab-1-cli-consul-0-key.pem
$ consul rtt mgmt
Estimated mgmt <-> jammy rtt: 0.632 ms (using LAN coordinates)

Additional notes

Inspect to determine if encryption is enabled or not.

Jul 16 22:59:53 vault-1 consul[1788]: ==> Starting Consul agent...
Jul 16 22:59:53 vault-1 consul[1788]:            Version: '1.12.3'
Jul 16 22:59:53 vault-1 consul[1788]:            Node ID: '806c8cef-77b0-585e-9d6b-0d7e120e7f76'
Jul 16 22:59:53 vault-1 consul[1788]:          Node name: 'vault-1'
Jul 16 22:59:53 vault-1 consul[1788]:         Datacenter: 'dc-lab-1' (Segment: '')
Jul 16 22:59:53 vault-1 consul[1788]:             Server: false (Bootstrap: false)
Jul 16 22:59:53 vault-1 consul[1788]:        Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Jul 16 22:59:53 vault-1 consul[1788]:       Cluster Addr: 172.16.148.3 (LAN: 8301, WAN: 8302)
Jul 16 22:59:53 vault-1 consul[1788]:            Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

Jul 16 23:02:40 vault-1 consul[1910]: ==> Starting Consul agent...
Jul 16 23:02:40 vault-1 consul[1910]:            Version: '1.12.3'
Jul 16 23:02:40 vault-1 consul[1910]:            Node ID: '806c8cef-77b0-585e-9d6b-0d7e120e7f76'
Jul 16 23:02:40 vault-1 consul[1910]:          Node name: 'vault-1'
Jul 16 23:02:40 vault-1 consul[1910]:         Datacenter: 'dc-lab-1' (Segment: '')
Jul 16 23:02:40 vault-1 consul[1910]:             Server: false (Bootstrap: false)
Jul 16 23:02:40 vault-1 consul[1910]:        Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Jul 16 23:02:40 vault-1 consul[1910]:       Cluster Addr: 172.16.148.3 (LAN: 8301, WAN: 8302)
Jul 16 23:02:40 vault-1 consul[1910]:            Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true, Auto-Encrypt-TLS: true

Generate encryption key#

Generate certificates#

Enable encryption on server#

Enable encryption on agent#

Use auto-encryption to simplify certificate provisioning#

CLI over encrypted connection#

Additional notes#