Secure the ZooKeeper quorum communication using SSL certificates.
Certification authority
Generate certification authority for testing purposes.
Create simplified CA configuration.
$ cat <<EOF | tee ca.config # OpenSSL configuration file for certification authority. [ req ] prompt = no distinguished_name = req_distinguished_name x509_extensions = v3_ca [ req_distinguished_name ] O = ZooKeeper OU = Backend emailAddress = admin@example.org [ v3_ca ] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical,CA:true EOF
Generate CA certificate and private key.
$ openssl req -new -x509 -days 3650 -keyout ca.key -out ca.pem -passout "pass:capassword" -config ca.config
Generating a RSA private key ..........................................................................+++++ .+++++ writing new private key to 'ca.key' -----
Inspect created CA certificate.
$ openssl x509 -in ca.pem -text --noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
75:af:15:b2:5f:1c:6d:67:08:59:38:11:f4:1d:54:7b:0b:3d:f6:e5
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
Validity
Not Before: Aug 8 19:14:09 2021 GMT
Not After : Aug 6 19:14:09 2031 GMT
Subject: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:b7:08:4b:ba:bf:bb:55:96:b9:cf:02:f5:11:38:
7c:70:f9:58:a4:c3:fc:88:70:23:6e:ce:35:03:10:
83:ff:ba:5c:ad:1e:e1:55:ce:8e:53:c2:e5:6b:12:
5b:93:39:b3:ad:7a:d8:4c:73:d1:68:5e:67:98:6b:
ba:2a:0e:64:af:49:11:77:23:9d:74:fe:5a:5c:a8:
59:97:80:ff:80:73:49:67:b1:60:b7:ae:57:30:42:
e3:71:aa:48:84:e8:30:4d:8c:d4:18:52:63:75:37:
21:d6:e9:61:35:b6:31:d7:60:a9:9c:56:ad:9c:b7:
60:f6:7c:9b:bb:ee:a6:f4:14:8c:87:81:dc:5a:69:
7f:f5:99:aa:57:c3:fb:29:2a:bb:dd:b8:e8:2b:8e:
ff:f1:5e:2b:6e:4f:a9:11:a3:4b:e5:d7:0c:3a:ef:
0d:ff:4a:4d:e8:c1:e8:e1:0d:2f:4a:da:3c:32:9f:
55:d3:9f:6f:40:5b:14:dc:00:84:30:7a:b3:55:3e:
70:a1:f8:6f:92:7f:41:51:93:62:00:e4:72:19:dc:
a2:31:99:30:ac:72:a2:d9:50:4c:11:17:fc:10:44:
48:99:41:3c:22:81:6d:42:1c:2b:fc:6e:49:52:01:
db:99:94:e4:16:a9:51:bd:40:55:54:71:94:dc:a3:
83:17
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
E8:58:8D:F9:A3:9A:67:5A:09:3D:0E:72:3A:EE:1E:FF:51:76:1E:63
X509v3 Authority Key Identifier:
keyid:E8:58:8D:F9:A3:9A:67:5A:09:3D:0E:72:3A:EE:1E:FF:51:76:1E:63
X509v3 Basic Constraints: critical
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
1e:51:83:61:51:63:6c:47:8f:ac:21:ce:84:a7:ee:dd:77:68:
28:63:a5:16:c4:ad:b2:30:5a:94:5b:52:63:82:6b:9d:29:2e:
f9:91:79:6c:92:bc:63:df:e9:a0:21:5d:ad:81:37:5e:ff:d2:
06:a1:5d:6c:2e:37:e9:f9:38:8e:13:ce:c7:a0:45:ff:fc:7c:
e5:7a:9c:33:69:44:4b:01:88:83:03:ed:d3:49:26:bc:4b:43:
73:5f:67:1e:3f:eb:24:b1:98:41:3e:66:fb:2a:78:83:35:c9:
68:9b:80:67:89:d5:1a:ee:50:9e:b0:78:36:45:31:02:c7:3e:
56:09:dd:2c:15:47:0a:e1:e4:45:ce:67:f7:c0:bc:87:6e:f5:
4f:43:1e:8e:d7:c1:f5:d9:5f:6d:b4:70:a4:31:16:96:e4:9a:
04:a5:b0:ac:a0:a6:44:6e:91:48:ae:59:0a:51:a0:77:9d:dc:
66:64:47:5a:38:e1:e6:e0:51:7a:88:b9:ce:ca:e4:71:26:f7:
e1:ea:46:19:7e:ae:b6:bb:50:a2:6f:6a:1a:2d:73:c7:44:7e:
3f:64:8f:1e:a2:5c:5b:80:2f:5d:7d:30:ca:8a:6f:2c:8e:1a:
46:8a:85:48:b7:db:cc:49:f2:3a:fd:f0:fd:7b:d1:7b:22:6e:
31:c7:e4:a3
Server certificates
The best solution would be to use Subject Alternative Names extension to generate a single certificate and use it on each server.
$ cat <<EOF | tee san_certificate.config # OpenSSL configuration file for SAN certificate. [ req ] req_extensions = req_ext distinguished_name = req_distinguished_name prompt = no [ req_distinguished_name ] commonName=zookeeper.backend.example.org [ req_ext ] subjectAltName = @alt_names [ alt_names ] DNS.1 = zookeeper1.example.org DNS.2 = zookeeper2.example.org DNS.3 = zookeeper3.example.org EOF
Generate certification request.
$ openssl req -new -config san_certificate.config -extensions req_ext -nodes -keyout san_certificate.key -out san_certificate.csr
Generating a RSA private key .....+++++ ...................................................................................+++++ writing new private key to 'san_certificate.key' -----
Create SAN certificate.
$ openssl x509 -req -CA ca.pem -CAkey ca.key -in san_certificate.csr -out san_certificate.pem -days 365 -CAcreateserial -extfile san_certificate.config -extensions req_ext -passin pass:capassword
Signature ok subject=CN = zookeeper.backend.example.org Getting CA Private Key
Inspect created certificate.
$ openssl x509 -in san_certificate.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
05:be:44:95:b8:4d:c3:1f:ac:90:d4:ef:d4:d9:43:ef:2c:49:1a:42
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
Validity
Not Before: Aug 8 19:16:12 2021 GMT
Not After : Aug 8 19:16:12 2022 GMT
Subject: CN = zookeeper.backend.example.org
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c3:99:c5:40:0d:49:09:9d:73:95:5e:a2:04:ba:
97:c1:15:8d:54:d4:a2:72:8d:aa:b5:4b:5c:ab:1c:
c1:81:fb:57:d6:2a:9f:df:d0:01:15:aa:40:c7:36:
59:8c:93:3a:70:28:de:ad:c9:a0:29:1c:05:3d:c3:
95:2d:44:79:8b:7a:7a:aa:fb:20:78:2c:73:ba:c9:
a8:d9:5c:48:ee:b6:39:a9:f3:5e:09:b5:32:6a:81:
a8:c3:82:57:8c:96:8f:b1:80:e8:9b:b7:4c:6a:a4:
9a:f7:a1:dc:4b:9d:60:6a:d7:fa:5b:97:84:f1:78:
60:f2:e9:ea:30:28:be:9b:fa:d9:b2:87:ae:05:22:
39:84:c7:4d:67:b1:23:f3:6c:35:74:3a:03:a1:a6:
06:e0:8a:16:58:79:7b:df:12:4a:1e:d1:35:e3:7f:
11:03:94:8a:5c:ac:07:4b:d9:b4:e9:0d:b5:f4:e5:
36:eb:f7:f3:55:aa:0e:8f:64:61:23:f8:dd:c3:77:
06:96:6f:2d:c5:81:56:e1:06:02:cf:fa:82:e3:fc:
d4:a6:80:7c:3d:44:77:25:85:45:b3:fd:b9:49:5a:
7d:d8:87:75:b1:f5:f0:5a:6e:56:f0:80:b5:30:74:
0b:d6:9f:c3:ca:b1:a7:e5:6b:87:2b:81:7d:91:d0:
3d:3b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:zookeeper1.example.org, DNS:zookeeper2.example.org, DNS:zookeeper3.example.org
Signature Algorithm: sha256WithRSAEncryption
3d:05:a5:90:a7:a5:ff:d5:c2:61:08:c0:42:15:d6:a1:20:ce:
18:b5:c7:8b:c2:0c:fb:a6:a5:35:a1:7d:07:96:32:8f:6e:84:
81:01:82:9a:94:86:63:b9:20:00:90:52:a0:ff:52:c4:9a:e8:
b2:57:63:cd:5a:a3:26:67:39:c1:f8:54:0c:b4:4d:d6:9c:e5:
4c:37:1d:99:70:06:52:65:f1:5f:f1:20:6d:59:a2:cf:db:09:
8d:2a:98:a7:b4:b2:f2:cc:82:80:fd:b7:5a:99:c4:18:02:6e:
22:16:94:e4:1d:d3:49:e6:ad:5f:c2:c0:fc:5e:18:48:7b:f1:
0d:e7:33:fd:a2:08:0b:6f:83:2d:64:67:f3:d2:ce:1b:b4:72:
f0:23:07:70:7b:d2:0a:b3:56:10:b9:65:02:b8:b6:0a:20:50:
76:96:78:e7:e2:e1:f9:18:45:17:2f:93:7e:79:12:ae:ad:89:
7d:41:89:5f:33:f1:fe:4c:76:01:10:fb:f4:07:b8:c7:49:a7:
07:f9:18:06:06:62:f2:29:bf:cf:79:02:0c:9c:10:e8:07:7a:
a8:d2:25:94:d2:9a:b1:46:d1:62:b4:60:2e:84:08:99:e8:59:
40:e1:49:71:24:21:06:7c:ea:d2:d4:26:71:b1:1e:33:4f:1d:
9b:2f:f2:d0
Create PKCS12 keystore.
$ openssl pkcs12 -export -in san_certificate.pem -inkey san_certificate.key -out server.keystore.p12 -name "zookeeper.backend" -password pass:certpassword
To add multiple certificates simply concatenate these including keys.
$ cat zookeeper_servers/zookeeper1.{pem,enckey} zookeeper_servers/zookeeper2.{pem,enckey} zookeeper_servers/zookeeper3.{pem,enckey} | tee multiple_certificates.pem
$ openssl pkcs12 -export -in multiple_certificates.pem -out server.keystore.p12 -name "zookeeper.backend" -password pass:certpassword
Inspect PKCS12 files.
$ openssl pkcs12 -in server.keystore.p12 -info -password pass:certpassword -nokeys
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
localKeyID: 34 E5 5B 82 D8 07 C6 3A 42 76 29 A8 54 3C 29 65 27 CB 33 D2
friendlyName: zookeeper.backend
subject=CN = zookeeper.backend.example.org
issuer=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
-----BEGIN CERTIFICATE-----
MIIDUzCCAjugAwIBAgIUBb5ElbhNwx+skNTv1NlD7yxJGkIwDQYJKoZIhvcNAQEL
BQAwSDESMBAGA1UECgwJWm9vS2VlcGVyMRAwDgYDVQQLDAdCYWNrZW5kMSAwHgYJ
KoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLm9yZzAeFw0yMTA4MDgxOTE2MTJaFw0y
MjA4MDgxOTE2MTJaMCgxJjAkBgNVBAMMHXpvb2tlZXBlci5iYWNrZW5kLmV4YW1w
bGUub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw5nFQA1JCZ1z
lV6iBLqXwRWNVNSico2qtUtcqxzBgftX1iqf39ABFapAxzZZjJM6cCjercmgKRwF
PcOVLUR5i3p6qvsgeCxzusmo2VxI7rY5qfNeCbUyaoGow4JXjJaPsYDom7dMaqSa
96HcS51gatf6W5eE8Xhg8unqMCi+m/rZsoeuBSI5hMdNZ7Ej82w1dDoDoaYG4IoW
WHl73xJKHtE1438RA5SKXKwHS9m06Q219OU26/fzVaoOj2RhI/jdw3cGlm8txYFW
4QYCz/qC4/zUpoB8PUR3JYVFs/25SVp92Id1sfXwWm5W8IC1MHQL1p/DyrGn5WuH
K4F9kdA9OwIDAQABo1UwUzBRBgNVHREESjBIghZ6b29rZWVwZXIxLmV4YW1wbGUu
b3JnghZ6b29rZWVwZXIyLmV4YW1wbGUub3JnghZ6b29rZWVwZXIzLmV4YW1wbGUu
b3JnMA0GCSqGSIb3DQEBCwUAA4IBAQA9BaWQp6X/1cJhCMBCFdahIM4YtceLwgz7
pqU1oX0HljKPboSBAYKalIZjuSAAkFKg/1LEmuiyV2PNWqMmZznB+FQMtE3WnOVM
Nx2ZcAZSZfFf8SBtWaLP2wmNKpintLLyzIKA/bdamcQYAm4iFpTkHdNJ5q1fwsD8
XhhIe/EN5zP9oggLb4MtZGfz0s4btHLwIwdwe9IKs1YQuWUCuLYKIFB2lnjn4uH5
GEUXL5N+eRKurYl9QYlfM/H+THYBEPv0B7jHSacH+RgGBmLyKb/PeQIMnBDoB3qo
0iWU0pqxRtFitGAuhAiZ6FlA4UlxJCEGfOrS1CZxsR4zTx2bL/LQ
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Configure ZooKeeper
Inspect default configuration.
$ cat conf/zoo.cfg
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/zookeeper/zookeeper/data clientPort=2181 server.1=zookeeper1.example.org:2888:3888 server.2=zookeeper2.example.org:2888:3888 server.3=zookeeper3.example.org:2888:3888 4lw.commands.whitelist=*
Secure the ZooKeeper quorum communication.
$ cat conf/zoo.cfg
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/zookeeper/zookeeper/data clientPort=2181 server.1=zookeeper1.example.org:2888:3888 server.2=zookeeper2.example.org:2888:3888 server.3=zookeeper3.example.org:2888:3888 4lw.commands.whitelist=* sslQuorum=true ssl.quorum.trustStore.location=/opt/zookeeper/ca.pem ssl.quorum.trustStore.password=capassword ssl.quorum.keyStore.location=/opt/zookeeper/server.keystore.p12 ssl.quorum.keyStore.password=certpassword ssl.quorum.hostnameVerification=true
Additional notes
Using PKCS12 format for truststore requires additional steps, as PKCS12 needs to contain a special attribute which is not easily added using OpenSSL utilities, at least for now.
Export CA to the JKS using keytool utility.
$ keytool -noprompt -storepass "capassword" -keystore server.truststore.jks -alias CARoot -import -file ca.pem
Certificate was added to keystore
Export it to PKCS12 format using keytool utility.
$ keytool -importkeystore -srckeystore server.truststore.jks -destkeystore cacertificate.p12 -srcstoretype jks -deststoretype pkcs12 -srcstorepass capassword -deststorepass capassword
Importing keystore server.truststore.jks to cacertificate.p12... Entry for alias caroot successfully imported. Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
Inspect PKCS12 file, notice additional 2.16.840.1.113894.746875.1.1 OID.
$ openssl pkcs12 -in cacertificate.p12 -info -password pass:capassword
MAC: sha1, Iteration 100000
MAC length: 20, salt length: 20
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 50000
Certificate bag
Bag Attributes
friendlyName: caroot
2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
issuer=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
-----BEGIN CERTIFICATE-----
MIIDcTCCAlmgAwIBAgIUda8Vsl8cbWcIWTgR9B1Uews99uUwDQYJKoZIhvcNAQEL
BQAwSDESMBAGA1UECgwJWm9vS2VlcGVyMRAwDgYDVQQLDAdCYWNrZW5kMSAwHgYJ
KoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLm9yZzAeFw0yMTA4MDgxOTE0MDlaFw0z
MTA4MDYxOTE0MDlaMEgxEjAQBgNVBAoMCVpvb0tlZXBlcjEQMA4GA1UECwwHQmFj
a2VuZDEgMB4GCSqGSIb3DQEJARYRYWRtaW5AZXhhbXBsZS5vcmcwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3CEu6v7tVlrnPAvUROHxw+Vikw/yIcCNu
zjUDEIP/ulytHuFVzo5TwuVrEluTObOtethMc9FoXmeYa7oqDmSvSRF3I510/lpc
qFmXgP+Ac0lnsWC3rlcwQuNxqkiE6DBNjNQYUmN1NyHW6WE1tjHXYKmcVq2ct2D2
fJu77qb0FIyHgdxaaX/1mapXw/spKrvduOgrjv/xXituT6kRo0vl1ww67w3/Sk3o
wejhDS9K2jwyn1XTn29AWxTcAIQwerNVPnCh+G+Sf0FRk2IA5HIZ3KIxmTCscqLZ
UEwRF/wQREiZQTwigW1CHCv8bklSAduZlOQWqVG9QFVUcZTco4MXAgMBAAGjUzBR
MB0GA1UdDgQWBBToWI35o5pnWgk9DnI67h7/UXYeYzAfBgNVHSMEGDAWgBToWI35
o5pnWgk9DnI67h7/UXYeYzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUA
A4IBAQAeUYNhUWNsR4+sIc6Ep+7dd2goY6UWxK2yMFqUW1JjgmudKS75kXlskrxj
3+mgIV2tgTde/9IGoV1sLjfp+TiOE87HoEX//HzlepwzaURLAYiDA+3TSSa8S0Nz
X2ceP+sksZhBPmb7KniDNclom4BnidUa7lCesHg2RTECxz5WCd0sFUcK4eRFzmf3
wLyHbvVPQx6O18H12V9ttHCkMRaW5JoEpbCsoKZEbpFIrlkKUaB3ndxmZEdaOOHm
4FF6iLnOyuRxJvfh6kYZfq62u1Cib2oaLXPHRH4/ZI8eolxbgC9dfTDKim8sjhpG
ioVIt9vMSfI6/fD9e9F7Im4xx+Sj
-----END CERTIFICATE-----
Remember, you can also use a PEM file as keystore, just remember to encrypt private keys.