Categories
DevOps

How to secure the ZooKeeper quorum communication

Secure the ZooKeeper quorum communication using SSL certificates.

Certification authority

Generate certification authority for testing purposes.

Create simplified CA configuration.

$ cat <<EOF | tee ca.config
# OpenSSL configuration file for certification authority.

[ req ]
prompt = no
distinguished_name = req_distinguished_name
x509_extensions    = v3_ca

[ req_distinguished_name ]
O               = ZooKeeper
OU              = Backend
emailAddress    = admin@example.org

[ v3_ca ]
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints       = critical,CA:true
EOF

Generate CA certificate and private key.

$ openssl req -new -x509 -days 3650 -keyout ca.key -out ca.pem -passout "pass:capassword" -config ca.config
Generating a RSA private key
..........................................................................+++++
.+++++
writing new private key to 'ca.key'
-----

Inspect created CA certificate.

$ openssl x509 -in ca.pem -text --noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            75:af:15:b2:5f:1c:6d:67:08:59:38:11:f4:1d:54:7b:0b:3d:f6:e5
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Validity
            Not Before: Aug  8 19:14:09 2021 GMT
            Not After : Aug  6 19:14:09 2031 GMT
        Subject: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b7:08:4b:ba:bf:bb:55:96:b9:cf:02:f5:11:38:
                    7c:70:f9:58:a4:c3:fc:88:70:23:6e:ce:35:03:10:
                    83:ff:ba:5c:ad:1e:e1:55:ce:8e:53:c2:e5:6b:12:
                    5b:93:39:b3:ad:7a:d8:4c:73:d1:68:5e:67:98:6b:
                    ba:2a:0e:64:af:49:11:77:23:9d:74:fe:5a:5c:a8:
                    59:97:80:ff:80:73:49:67:b1:60:b7:ae:57:30:42:
                    e3:71:aa:48:84:e8:30:4d:8c:d4:18:52:63:75:37:
                    21:d6:e9:61:35:b6:31:d7:60:a9:9c:56:ad:9c:b7:
                    60:f6:7c:9b:bb:ee:a6:f4:14:8c:87:81:dc:5a:69:
                    7f:f5:99:aa:57:c3:fb:29:2a:bb:dd:b8:e8:2b:8e:
                    ff:f1:5e:2b:6e:4f:a9:11:a3:4b:e5:d7:0c:3a:ef:
                    0d:ff:4a:4d:e8:c1:e8:e1:0d:2f:4a:da:3c:32:9f:
                    55:d3:9f:6f:40:5b:14:dc:00:84:30:7a:b3:55:3e:
                    70:a1:f8:6f:92:7f:41:51:93:62:00:e4:72:19:dc:
                    a2:31:99:30:ac:72:a2:d9:50:4c:11:17:fc:10:44:
                    48:99:41:3c:22:81:6d:42:1c:2b:fc:6e:49:52:01:
                    db:99:94:e4:16:a9:51:bd:40:55:54:71:94:dc:a3:
                    83:17
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                E8:58:8D:F9:A3:9A:67:5A:09:3D:0E:72:3A:EE:1E:FF:51:76:1E:63
            X509v3 Authority Key Identifier: 
                keyid:E8:58:8D:F9:A3:9A:67:5A:09:3D:0E:72:3A:EE:1E:FF:51:76:1E:63

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         1e:51:83:61:51:63:6c:47:8f:ac:21:ce:84:a7:ee:dd:77:68:
         28:63:a5:16:c4:ad:b2:30:5a:94:5b:52:63:82:6b:9d:29:2e:
         f9:91:79:6c:92:bc:63:df:e9:a0:21:5d:ad:81:37:5e:ff:d2:
         06:a1:5d:6c:2e:37:e9:f9:38:8e:13:ce:c7:a0:45:ff:fc:7c:
         e5:7a:9c:33:69:44:4b:01:88:83:03:ed:d3:49:26:bc:4b:43:
         73:5f:67:1e:3f:eb:24:b1:98:41:3e:66:fb:2a:78:83:35:c9:
         68:9b:80:67:89:d5:1a:ee:50:9e:b0:78:36:45:31:02:c7:3e:
         56:09:dd:2c:15:47:0a:e1:e4:45:ce:67:f7:c0:bc:87:6e:f5:
         4f:43:1e:8e:d7:c1:f5:d9:5f:6d:b4:70:a4:31:16:96:e4:9a:
         04:a5:b0:ac:a0:a6:44:6e:91:48:ae:59:0a:51:a0:77:9d:dc:
         66:64:47:5a:38:e1:e6:e0:51:7a:88:b9:ce:ca:e4:71:26:f7:
         e1:ea:46:19:7e:ae:b6:bb:50:a2:6f:6a:1a:2d:73:c7:44:7e:
         3f:64:8f:1e:a2:5c:5b:80:2f:5d:7d:30:ca:8a:6f:2c:8e:1a:
         46:8a:85:48:b7:db:cc:49:f2:3a:fd:f0:fd:7b:d1:7b:22:6e:
         31:c7:e4:a3

Server certificates

The best solution would be to use Subject Alternative Names extension to generate a single certificate and use it on each server.

$ cat <<EOF | tee san_certificate.config
# OpenSSL configuration file for SAN certificate.

[ req ]
req_extensions     = req_ext
distinguished_name = req_distinguished_name
prompt             = no

[ req_distinguished_name ]
commonName=zookeeper.backend.example.org

[ req_ext ]
subjectAltName   = @alt_names

[ alt_names ]
DNS.1  = zookeeper1.example.org
DNS.2  = zookeeper2.example.org
DNS.3  = zookeeper3.example.org
EOF

Generate certification request.

$ openssl req -new -config san_certificate.config -extensions req_ext -nodes -keyout san_certificate.key -out san_certificate.csr
Generating a RSA private key
.....+++++
...................................................................................+++++
writing new private key to 'san_certificate.key'
-----

Create SAN certificate.

$ openssl x509 -req -CA ca.pem -CAkey ca.key -in san_certificate.csr -out san_certificate.pem -days 365 -CAcreateserial -extfile san_certificate.config -extensions req_ext -passin pass:capassword
Signature ok
subject=CN = zookeeper.backend.example.org
Getting CA Private Key

Inspect created certificate.

$ openssl x509 -in san_certificate.pem -text -noout
Certificate:
    Data:                                                          
        Version: 3 (0x2)
        Serial Number:
            05:be:44:95:b8:4d:c3:1f:ac:90:d4:ef:d4:d9:43:ef:2c:49:1a:42
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Validity
            Not Before: Aug  8 19:16:12 2021 GMT
            Not After : Aug  8 19:16:12 2022 GMT
        Subject: CN = zookeeper.backend.example.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c3:99:c5:40:0d:49:09:9d:73:95:5e:a2:04:ba:
                    97:c1:15:8d:54:d4:a2:72:8d:aa:b5:4b:5c:ab:1c:
                    c1:81:fb:57:d6:2a:9f:df:d0:01:15:aa:40:c7:36:
                    59:8c:93:3a:70:28:de:ad:c9:a0:29:1c:05:3d:c3:
                    95:2d:44:79:8b:7a:7a:aa:fb:20:78:2c:73:ba:c9:
                    a8:d9:5c:48:ee:b6:39:a9:f3:5e:09:b5:32:6a:81:
                    a8:c3:82:57:8c:96:8f:b1:80:e8:9b:b7:4c:6a:a4:
                    9a:f7:a1:dc:4b:9d:60:6a:d7:fa:5b:97:84:f1:78:
                    60:f2:e9:ea:30:28:be:9b:fa:d9:b2:87:ae:05:22:
                    39:84:c7:4d:67:b1:23:f3:6c:35:74:3a:03:a1:a6:
                    06:e0:8a:16:58:79:7b:df:12:4a:1e:d1:35:e3:7f:
                    11:03:94:8a:5c:ac:07:4b:d9:b4:e9:0d:b5:f4:e5:
                    36:eb:f7:f3:55:aa:0e:8f:64:61:23:f8:dd:c3:77:
                    06:96:6f:2d:c5:81:56:e1:06:02:cf:fa:82:e3:fc:
                    d4:a6:80:7c:3d:44:77:25:85:45:b3:fd:b9:49:5a:
                    7d:d8:87:75:b1:f5:f0:5a:6e:56:f0:80:b5:30:74:
                    0b:d6:9f:c3:ca:b1:a7:e5:6b:87:2b:81:7d:91:d0:
                    3d:3b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:zookeeper1.example.org, DNS:zookeeper2.example.org, DNS:zookeeper3.example.org
    Signature Algorithm: sha256WithRSAEncryption
         3d:05:a5:90:a7:a5:ff:d5:c2:61:08:c0:42:15:d6:a1:20:ce:
         18:b5:c7:8b:c2:0c:fb:a6:a5:35:a1:7d:07:96:32:8f:6e:84:
         81:01:82:9a:94:86:63:b9:20:00:90:52:a0:ff:52:c4:9a:e8:
         b2:57:63:cd:5a:a3:26:67:39:c1:f8:54:0c:b4:4d:d6:9c:e5:
         4c:37:1d:99:70:06:52:65:f1:5f:f1:20:6d:59:a2:cf:db:09:
         8d:2a:98:a7:b4:b2:f2:cc:82:80:fd:b7:5a:99:c4:18:02:6e:
         22:16:94:e4:1d:d3:49:e6:ad:5f:c2:c0:fc:5e:18:48:7b:f1:
         0d:e7:33:fd:a2:08:0b:6f:83:2d:64:67:f3:d2:ce:1b:b4:72:
         f0:23:07:70:7b:d2:0a:b3:56:10:b9:65:02:b8:b6:0a:20:50:
         76:96:78:e7:e2:e1:f9:18:45:17:2f:93:7e:79:12:ae:ad:89:
         7d:41:89:5f:33:f1:fe:4c:76:01:10:fb:f4:07:b8:c7:49:a7:
         07:f9:18:06:06:62:f2:29:bf:cf:79:02:0c:9c:10:e8:07:7a:
         a8:d2:25:94:d2:9a:b1:46:d1:62:b4:60:2e:84:08:99:e8:59:
         40:e1:49:71:24:21:06:7c:ea:d2:d4:26:71:b1:1e:33:4f:1d:
         9b:2f:f2:d0

Create PKCS12 keystore.

$ openssl pkcs12 -export -in san_certificate.pem -inkey san_certificate.key -out server.keystore.p12 -name "zookeeper.backend" -password pass:certpassword

To add multiple certificates simply concatenate these including keys.

$ cat zookeeper_servers/zookeeper1.{pem,enckey} zookeeper_servers/zookeeper2.{pem,enckey} zookeeper_servers/zookeeper3.{pem,enckey} | tee multiple_certificates.pem
$ openssl pkcs12 -export -in multiple_certificates.pem -out server.keystore.p12 -name "zookeeper.backend" -password pass:certpassword

Inspect PKCS12 files.

$ openssl pkcs12 -in server.keystore.p12 -info -password pass:certpassword -nokeys
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: 34 E5 5B 82 D8 07 C6 3A 42 76 29 A8 54 3C 29 65 27 CB 33 D2 
    friendlyName: zookeeper.backend
subject=CN = zookeeper.backend.example.org

issuer=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org

-----BEGIN CERTIFICATE-----
MIIDUzCCAjugAwIBAgIUBb5ElbhNwx+skNTv1NlD7yxJGkIwDQYJKoZIhvcNAQEL
BQAwSDESMBAGA1UECgwJWm9vS2VlcGVyMRAwDgYDVQQLDAdCYWNrZW5kMSAwHgYJ
KoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLm9yZzAeFw0yMTA4MDgxOTE2MTJaFw0y
MjA4MDgxOTE2MTJaMCgxJjAkBgNVBAMMHXpvb2tlZXBlci5iYWNrZW5kLmV4YW1w
bGUub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw5nFQA1JCZ1z
lV6iBLqXwRWNVNSico2qtUtcqxzBgftX1iqf39ABFapAxzZZjJM6cCjercmgKRwF
PcOVLUR5i3p6qvsgeCxzusmo2VxI7rY5qfNeCbUyaoGow4JXjJaPsYDom7dMaqSa
96HcS51gatf6W5eE8Xhg8unqMCi+m/rZsoeuBSI5hMdNZ7Ej82w1dDoDoaYG4IoW
WHl73xJKHtE1438RA5SKXKwHS9m06Q219OU26/fzVaoOj2RhI/jdw3cGlm8txYFW
4QYCz/qC4/zUpoB8PUR3JYVFs/25SVp92Id1sfXwWm5W8IC1MHQL1p/DyrGn5WuH
K4F9kdA9OwIDAQABo1UwUzBRBgNVHREESjBIghZ6b29rZWVwZXIxLmV4YW1wbGUu
b3JnghZ6b29rZWVwZXIyLmV4YW1wbGUub3JnghZ6b29rZWVwZXIzLmV4YW1wbGUu
b3JnMA0GCSqGSIb3DQEBCwUAA4IBAQA9BaWQp6X/1cJhCMBCFdahIM4YtceLwgz7
pqU1oX0HljKPboSBAYKalIZjuSAAkFKg/1LEmuiyV2PNWqMmZznB+FQMtE3WnOVM
Nx2ZcAZSZfFf8SBtWaLP2wmNKpintLLyzIKA/bdamcQYAm4iFpTkHdNJ5q1fwsD8
XhhIe/EN5zP9oggLb4MtZGfz0s4btHLwIwdwe9IKs1YQuWUCuLYKIFB2lnjn4uH5
GEUXL5N+eRKurYl9QYlfM/H+THYBEPv0B7jHSacH+RgGBmLyKb/PeQIMnBDoB3qo
0iWU0pqxRtFitGAuhAiZ6FlA4UlxJCEGfOrS1CZxsR4zTx2bL/LQ
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Configure ZooKeeper

Inspect default configuration.

$ cat conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/zookeeper/data
clientPort=2181
server.1=zookeeper1.example.org:2888:3888
server.2=zookeeper2.example.org:2888:3888
server.3=zookeeper3.example.org:2888:3888
4lw.commands.whitelist=*

Secure the ZooKeeper quorum communication.

$ cat conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/zookeeper/data
clientPort=2181
server.1=zookeeper1.example.org:2888:3888
server.2=zookeeper2.example.org:2888:3888
server.3=zookeeper3.example.org:2888:3888
4lw.commands.whitelist=*

sslQuorum=true
ssl.quorum.trustStore.location=/opt/zookeeper/ca.pem
ssl.quorum.trustStore.password=capassword
ssl.quorum.keyStore.location=/opt/zookeeper/server.keystore.p12
ssl.quorum.keyStore.password=certpassword
ssl.quorum.hostnameVerification=true

Additional notes

Using PKCS12 format for truststore requires additional steps, as PKCS12 needs to contain a special attribute which is not easily added using OpenSSL utilities, at least for now.

Export CA to the JKS using keytool utility.

$ keytool -noprompt -storepass "capassword" -keystore server.truststore.jks -alias CARoot -import -file ca.pem
Certificate was added to keystore

Export it to PKCS12 format using keytool utility.

$ keytool -importkeystore -srckeystore server.truststore.jks -destkeystore cacertificate.p12 -srcstoretype jks -deststoretype pkcs12 -srcstorepass capassword -deststorepass capassword
Importing keystore server.truststore.jks to cacertificate.p12...
Entry for alias caroot successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Inspect PKCS12 file, notice additional 2.16.840.1.113894.746875.1.1 OID.

$ openssl pkcs12 -in cacertificate.p12  -info -password pass:capassword
MAC: sha1, Iteration 100000
MAC length: 20, salt length: 20
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 50000
Certificate bag
Bag Attributes
    friendlyName: caroot
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org

issuer=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org

-----BEGIN CERTIFICATE-----
MIIDcTCCAlmgAwIBAgIUda8Vsl8cbWcIWTgR9B1Uews99uUwDQYJKoZIhvcNAQEL
BQAwSDESMBAGA1UECgwJWm9vS2VlcGVyMRAwDgYDVQQLDAdCYWNrZW5kMSAwHgYJ
KoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLm9yZzAeFw0yMTA4MDgxOTE0MDlaFw0z
MTA4MDYxOTE0MDlaMEgxEjAQBgNVBAoMCVpvb0tlZXBlcjEQMA4GA1UECwwHQmFj
a2VuZDEgMB4GCSqGSIb3DQEJARYRYWRtaW5AZXhhbXBsZS5vcmcwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3CEu6v7tVlrnPAvUROHxw+Vikw/yIcCNu
zjUDEIP/ulytHuFVzo5TwuVrEluTObOtethMc9FoXmeYa7oqDmSvSRF3I510/lpc
qFmXgP+Ac0lnsWC3rlcwQuNxqkiE6DBNjNQYUmN1NyHW6WE1tjHXYKmcVq2ct2D2
fJu77qb0FIyHgdxaaX/1mapXw/spKrvduOgrjv/xXituT6kRo0vl1ww67w3/Sk3o
wejhDS9K2jwyn1XTn29AWxTcAIQwerNVPnCh+G+Sf0FRk2IA5HIZ3KIxmTCscqLZ
UEwRF/wQREiZQTwigW1CHCv8bklSAduZlOQWqVG9QFVUcZTco4MXAgMBAAGjUzBR
MB0GA1UdDgQWBBToWI35o5pnWgk9DnI67h7/UXYeYzAfBgNVHSMEGDAWgBToWI35
o5pnWgk9DnI67h7/UXYeYzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUA
A4IBAQAeUYNhUWNsR4+sIc6Ep+7dd2goY6UWxK2yMFqUW1JjgmudKS75kXlskrxj
3+mgIV2tgTde/9IGoV1sLjfp+TiOE87HoEX//HzlepwzaURLAYiDA+3TSSa8S0Nz
X2ceP+sksZhBPmb7KniDNclom4BnidUa7lCesHg2RTECxz5WCd0sFUcK4eRFzmf3
wLyHbvVPQx6O18H12V9ttHCkMRaW5JoEpbCsoKZEbpFIrlkKUaB3ndxmZEdaOOHm
4FF6iLnOyuRxJvfh6kYZfq62u1Cib2oaLXPHRH4/ZI8eolxbgC9dfTDKim8sjhpG
ioVIt9vMSfI6/fD9e9F7Im4xx+Sj
-----END CERTIFICATE-----

Remember, you can also use a PEM file as keystore, just remember to encrypt private keys.