Categories
DevOps

How to secure the ZooKeeper client communication

Secure the ZooKeeper client communication.

Certification authority

Generate certification authority for testing purposes.

Create a simplified CA configuration.

$ cat <<EOF | tee ca.config
# OpenSSL configuration file for certification authority.

[ req ]
prompt = no
distinguished_name = req_distinguished_name
x509_extensions    = v3_ca

[ req_distinguished_name ]
O               = ZooKeeper
OU              = Backend
emailAddress    = admin@example.org

[ v3_ca ]
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints       = critical,CA:true
EOF

Generate CA certificate and private key.

$ openssl req -new -x509 -days 3650 -keyout ca.key -out ca.pem -passout "pass:capassword" -config ca.config
Generating a RSA private key
........+++++
...............................................................................................................................................+++++
writing new private key to 'ca.key'
-----

Inspect created CA certificate.

$ openssl x509 -in ca.pem -text --noout
certificate:                                                                     
    Data:                                                                        
        Version: 3 (0x2)                                                         
        Serial Number:                                                           
            0d:fe:d0:e6:2f:90:b7:3d:e8:62:d2:36:5a:13:d2:d6:ae:45:2e:58
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Validity                                                                 
            Not Before: Aug 28 07:52:58 2021 GMT
            Not After : Aug 26 07:52:58 2031 GMT
        Subject: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:d0:bb:78:73:1a:04:52:a5:d9:61:73:e8:f9:e8:
                    aa:dd:9b:81:79:6b:df:07:32:81:f1:2f:d0:0b:6c:
                    28:c2:0f:8d:5a:99:7c:fd:2e:0a:b0:7d:ec:af:07:
                    d8:51:f9:54:1a:f6:27:06:5c:d9:d5:97:d5:21:f2:
                    94:61:e6:12:6f:46:98:16:18:6b:66:5c:c3:89:b3:
                    f5:12:01:93:7a:8d:d7:46:2b:6e:cf:74:00:54:9b:
                    ae:60:be:d0:53:b0:c4:da:b6:17:fd:42:e1:f1:ec:
                    9f:8d:43:9b:7d:f3:73:95:cd:a6:ec:90:13:a3:43:
                    24:48:f1:54:e5:07:49:21:98:9c:89:6f:54:18:46:
                    f0:56:81:5c:9d:88:b4:b4:72:e1:9b:07:6f:60:3e:
                    59:b7:76:7f:d7:ac:eb:c4:d0:20:eb:9e:97:ab:18:
                    2b:a4:64:0a:55:ac:ac:dd:49:7c:8a:c4:1b:ea:67:
                    da:2e:10:c6:66:81:de:86:97:9b:5e:20:64:cc:bb:
                    cc:ca:ac:a8:6f:79:cc:62:91:3d:b3:dd:44:e4:01:
                    cd:b9:2b:02:25:fd:f2:93:f8:27:5c:99:8e:9d:04:
                    7c:09:32:0b:f4:7d:ab:72:61:87:56:78:e6:31:f7:
                    14:32:d1:44:65:9d:37:47:65:91:01:b6:bf:93:2b:
                    f9:9f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                C7:23:90:57:0D:7D:FD:99:21:7C:7D:35:C4:87:A6:55:C6:F5:54:DC
            X509v3 Authority Key Identifier: 
                keyid:C7:23:90:57:0D:7D:FD:99:21:7C:7D:35:C4:87:A6:55:C6:F5:54:DC 

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         33:73:6f:88:08:96:5b:f7:5d:9e:84:ec:4c:5b:0e:2c:73:61:
         de:e2:86:2e:f1:4d:eb:ca:77:cd:7c:89:8a:68:45:7d:92:3f:
         69:c6:fe:32:33:24:57:77:88:8f:be:ef:11:b3:eb:3c:dc:be:
         02:9a:20:cf:ea:b1:d2:33:5c:bf:24:de:42:81:f6:fa:9f:92:
         cc:dd:7a:f7:6d:39:87:e3:93:5d:84:cc:a4:21:f9:56:d9:64:
         dd:07:d6:66:84:4e:52:b5:4e:48:cc:ad:32:63:f1:05:62:4e:
         88:f4:93:e1:a3:9c:37:67:d8:6d:a9:6e:76:3e:88:07:43:fe:
         33:57:57:ff:78:74:74:3e:ec:5d:7a:51:a8:9a:22:82:43:a1:
         7c:a7:7c:bf:9b:58:82:1e:b0:12:b2:bb:8c:e7:50:6d:dc:21:
         b5:fd:38:83:09:67:54:bc:91:53:9b:e2:f4:4b:b5:14:99:98:
         c3:b5:80:5e:b6:30:01:40:c3:e8:15:f3:a9:c5:9f:21:ca:b0:
         51:a5:34:f9:59:ed:8b:83:2c:e4:6d:dd:62:c4:6f:29:25:1c:
         58:91:84:94:36:4a:f9:8f:18:eb:be:e1:ee:b4:02:9f:d2:a1:
         fc:9d:1b:07:e2:b4:1f:a1:c3:d4:2b:76:bd:39:f6:ac:65:22:
         b9:b0:6b:9d

Server certificates

The easiest solution, in this case, is to use Subject Alternative Names extension to generate a single certificate and use it on each server. Add localhost to use local ZooKeeper utilities.

$ cat <<EOF | tee san_certificate.config
# OpenSSL configuration file for SAN certificate.

[ req ]
req_extensions     = req_ext
distinguished_name = req_distinguished_name
prompt             = no

[ req_distinguished_name ]
commonName=zookeeper.backend.example.org

[ req_ext ]
subjectAltName   = @alt_names

[ alt_names ]
DNS.1 = zookeeper1.example.org
DNS.2 = zookeeper2.example.org
DNS.3 = zookeeper3.example.org
DNS.4 = localhost
EOF

Generate certification request.

$ openssl req -new -config san_certificate.config -extensions req_ext -nodes -keyout san_certificate.key -out san_certificate.csr
Generating a RSA private key
........................+++++
..................................................................................................................+++++
writing new private key to 'san_certificate.key'
-----

Create SAN certificate.

$ openssl x509 -req -CA ca.pem -CAkey ca.key -in san_certificate.csr -out san_certificate.pem -days 365 -CAcreateserial -extfile san_certificate.config -extensions req_ext -passin pass:capassword
Signature ok
subject=CN = zookeeper.backend.example.org
Getting CA Private Key

Inspect created certificate.

$ openssl x509 -in san_certificate.pem -text -noout
Certificate:                                                                                                                                                       
    Data:                                                                                                                                                          
        Version: 3 (0x2)                                                                                                                                           
        Serial Number:
            36:40:ff:09:9f:63:5b:29:6c:a1:9a:fa:73:79:92:bd:79:d1:2d:f1
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Validity
            Not Before: Aug 28 08:00:14 2021 GMT
            Not After : Aug 28 08:00:14 2022 GMT
        Subject: CN = zookeeper.backend.example.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:df:af:10:16:d9:27:c9:2f:89:27:8d:d4:14:db:
                    24:ba:b4:eb:a8:19:b8:ea:d9:e6:98:58:13:3b:0d:
                    78:d8:ab:7f:e1:28:89:ba:b8:dc:7d:9a:02:c9:d1:
                    30:68:fd:f1:19:36:55:a4:d7:2d:9c:0f:2b:7b:7a:
                    95:98:e9:c6:ad:0a:68:fb:24:2b:de:c0:f1:fc:87:
                    bf:9a:16:2d:c9:28:e8:6a:a6:44:e9:0f:d8:50:d1:
                    e6:13:d3:7a:7f:5a:47:06:c7:d7:ae:e2:a9:58:70:
                    b9:51:68:4c:f8:94:85:3e:e8:73:cf:9b:df:af:ee:
                    e6:86:24:41:46:f4:e6:67:5c:e4:95:2f:b8:df:6f:
                    09:19:67:8d:09:dd:a6:8e:28:c4:f7:ff:d1:1d:d1:
                    0d:27:e3:53:91:93:fe:bf:78:a5:55:85:c5:6d:c7:
                    46:8f:69:b8:fd:49:3d:c8:8b:c5:e3:82:0e:dd:7f:
                    44:71:98:5c:5f:0c:09:0d:ba:b2:3b:67:32:8a:40:
                    5e:8d:ec:91:e8:ba:5f:53:8c:60:83:0a:85:58:59:
                    6c:5e:39:3f:5b:dd:72:a2:e3:60:c1:72:dd:18:7b:
                    81:8d:73:b0:66:05:8f:70:73:dc:29:89:16:9e:4a:
                    64:d7:67:27:66:bd:45:9b:37:11:3a:41:21:89:05:
                    2e:93
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:zookeeper1.example.org, DNS:zookeeper2.example.org, DNS:zookeeper3.example.org, DNS:localhost
    Signature Algorithm: sha256WithRSAEncryption
         7b:ae:bf:42:b6:44:a4:0a:91:6c:b9:ec:42:be:90:e5:63:2e:
         71:a9:3b:d7:96:f6:51:c8:1e:34:12:4e:0f:d4:3b:b0:43:7c:
         a0:d4:f0:1f:aa:23:58:04:18:a4:8a:08:a3:55:3f:6f:ed:95:
         70:f4:b4:ed:a9:16:d6:60:f7:a6:71:af:a0:5f:ec:93:4f:4e:
         79:46:95:e1:21:96:4b:a4:27:6c:51:76:b0:2f:be:d7:2d:99:
         46:1f:74:08:ca:54:85:61:64:94:fc:8f:28:50:43:51:06:72:
         36:f5:e2:b8:a8:c5:c0:99:28:03:3c:2e:7d:f2:b3:9e:aa:61:
         da:c1:0d:be:83:21:b4:48:9b:b0:06:3c:5b:d3:a5:4b:6c:9e:
         b0:c2:5b:22:65:d5:08:67:ca:11:d3:05:33:7a:12:79:7d:35:
         57:ef:8d:88:56:f1:da:d2:82:06:71:fe:3e:f6:5f:1a:f5:78:
         6c:28:ac:ae:ec:15:6c:80:4a:e9:b2:9b:4c:5a:7b:c4:f2:13:
         fb:fb:fe:fe:f0:7d:8d:f0:ee:3b:24:69:b0:ad:64:e6:77:e8:
         9c:87:94:a5:15:65:85:b3:7b:45:d4:ff:1b:b2:4e:74:95:0e:
         35:4c:59:c6:b8:ab:56:be:73:58:87:48:2f:eb:50:e2:42:37:
         34:42:79:78

Create PKCS12 keystore.

$ openssl pkcs12 -export -in san_certificate.pem -inkey san_certificate.key -out server.keystore.p12 -name "zookeeper.backend" -password pass:certpassword

Inspect PKCS12 files.

$ openssl pkcs12 -in server.keystore.p12 -info -password pass:certpassword -nokeys
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: 1A 8B 34 C4 87 36 AD 45 3E CE 2B D4 6D 94 78 20 48 90 44 A7 
    friendlyName: zookeeper.backend
subject=CN = zookeeper.backend.example.org

issuer=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org

-----BEGIN CERTIFICATE-----
MIIDXjCCAkagAwIBAgIUNkD/CZ9jWylsoZr6c3mSvXnRLfEwDQYJKoZIhvcNAQEL
BQAwSDESMBAGA1UECgwJWm9vS2VlcGVyMRAwDgYDVQQLDAdCYWNrZW5kMSAwHgYJ
KoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLm9yZzAeFw0yMTA4MjgwODAwMTRaFw0y
MjA4MjgwODAwMTRaMCgxJjAkBgNVBAMMHXpvb2tlZXBlci5iYWNrZW5kLmV4YW1w
bGUub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA368QFtknyS+J
J43UFNskurTrqBm46tnmmFgTOw142Kt/4SiJurjcfZoCydEwaP3xGTZVpNctnA8r
e3qVmOnGrQpo+yQr3sDx/Ie/mhYtySjoaqZE6Q/YUNHmE9N6f1pHBsfXruKpWHC5
UWhM+JSFPuhzz5vfr+7mhiRBRvTmZ1zklS+4328JGWeNCd2mjijE9//RHdENJ+NT
kZP+v3ilVYXFbcdGj2m4/Uk9yIvF44IO3X9EcZhcXwwJDbqyO2cyikBejeyR6Lpf
U4xggwqFWFlsXjk/W91youNgwXLdGHuBjXOwZgWPcHPcKYkWnkpk12cnZr1FmzcR
OkEhiQUukwIDAQABo2AwXjBcBgNVHREEVTBTghZ6b29rZWVwZXIxLmV4YW1wbGUu
b3JnghZ6b29rZWVwZXIyLmV4YW1wbGUub3JnghZ6b29rZWVwZXIzLmV4YW1wbGUu
b3Jngglsb2NhbGhvc3QwDQYJKoZIhvcNAQELBQADggEBAHuuv0K2RKQKkWy57EK+
kOVjLnGpO9eW9lHIHjQSTg/UO7BDfKDU8B+qI1gEGKSKCKNVP2/tlXD0tO2pFtZg
96Zxr6Bf7JNPTnlGleEhlkukJ2xRdrAvvtctmUYfdAjKVIVhZJT8jyhQQ1EGcjb1
4rioxcCZKAM8Ln3ys56qYdrBDb6DIbRIm7AGPFvTpUtsnrDCWyJl1QhnyhHTBTN6
Enl9NVfvjYhW8drSggZx/j72Xxr1eGworK7sFWyASumym0xae8TyE/v7/v7wfY3w
7jskabCtZOZ36JyHlKUVZYWze0XU/xuyTnSVDjVMWca4q1a+c1iHSC/rUOJCNzRC
eXg=
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Client certificates

Create a client certificate for the specific user, use commonName to specify client name instead of domain name.

$ cat <<EOF | tee client_milosz_certificate.config
# OpenSSL configuration file for SAN certificate.

[ req ]
distinguished_name = req_distinguished_name
prompt             = no

[ req_distinguished_name ]
commonName=client_milosz
EOF

Generate certification request.

$ openssl req -new -config client_milosz_certificate.config -nodes -keyout client_milosz_certificate.key -out client_milosz_certificate.csr
Generating a RSA private key
.........+++++
........................+++++
writing new private key to 'client_milosz_certificate.key'
-----

Create client certificate.

$ openssl x509 -req -CA ca.pem -CAkey ca.key -in client_milosz_certificate.csr -out client_milosz_certificate.pem -days 365 -CAcreateserial -passin pass:capassword
Signature ok
subject=CN = client_milosz
Getting CA Private Key

Inspect created certificate.

$ openssl x509 -in client_milosz_certificate.pem -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            36:40:ff:09:9f:63:5b:29:6c:a1:9a:fa:73:79:92:bd:79:d1:2d:f2
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = ZooKeeper, OU = Backend, emailAddress = admin@example.org
        Validity
            Not Before: Aug 28 08:05:27 2021 GMT
            Not After : Aug 28 08:05:27 2022 GMT
        Subject: CN = client_milosz
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:ba:6b:85:37:59:2a:06:53:28:bb:7c:3e:4c:d8:
                    a2:b1:c0:ca:8b:30:2e:19:bd:2c:c5:06:c3:79:b1:
                    95:0c:ac:41:5d:c6:06:76:7f:02:38:36:74:43:96:
                    b4:ac:d6:a4:df:cb:f3:87:b8:72:1b:e2:c1:34:7f:
                    97:bb:cf:5f:be:c2:10:dd:69:e4:88:31:a5:68:3c:
                    d7:f6:9b:aa:02:c5:71:e0:b4:0d:86:e9:73:03:5a:
                    00:a0:58:d6:8e:e0:eb:a1:5c:2d:04:a2:45:0c:8d:
                    17:4f:15:5f:f2:00:4d:fa:fe:82:d8:84:23:d7:0b:
                    ba:22:04:cd:75:aa:d2:0a:40:6b:cb:8b:28:21:09:
                    ef:99:b5:ab:f8:c0:2d:7e:05:ec:18:80:7d:0a:26:
                    d1:88:bd:60:9e:c0:c6:70:c4:60:c8:a3:b2:7c:10:
                    86:ef:06:6c:64:9b:71:25:54:4e:82:1e:1e:15:3d:
                    17:d3:ff:4f:14:ab:82:31:eb:8c:43:40:66:e2:db:
                    7e:bf:60:eb:ac:e6:c9:85:fa:e4:2d:d0:46:17:61:
                    ba:3d:c1:f8:ca:27:88:08:12:e2:d7:ec:66:63:6e:
                    d4:10:06:7c:64:88:b5:01:d9:85:00:b4:16:b0:fd:
                    9a:f0:38:65:ce:45:57:53:ad:e0:dc:f9:46:9e:62:
                    9d:2d
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         5b:4d:bf:8b:91:c5:16:02:8f:63:fc:69:c6:f5:df:c3:94:4a:
         1d:5a:b8:03:b0:8b:7e:2d:9a:05:92:8c:12:ab:bf:f1:d5:35:
         b9:97:2f:6d:b2:52:4f:25:9d:90:a7:6d:12:31:68:5b:11:6d:
         df:05:0f:df:fc:e9:f0:17:df:58:c1:57:e9:3b:89:56:8d:0a:
         b0:8a:fc:0e:5b:bd:ac:95:47:e0:46:05:ac:df:f5:24:6e:73:
         7f:90:74:fc:4b:1b:8f:99:ec:d7:2a:00:f0:0b:db:09:50:a2:
         98:7d:31:6c:49:21:05:c4:39:f5:69:3d:75:71:85:9f:91:d9:
         7d:1a:c6:76:2e:31:4f:ab:1e:ec:a3:0a:de:97:bf:21:ee:1f:
         6b:50:f4:c5:6d:46:ad:51:f2:61:55:66:85:75:af:34:54:d3:
         de:c4:2e:69:0c:4a:8d:6a:1b:20:da:2a:26:bd:22:1a:45:96:
         33:86:28:21:72:c6:89:41:2a:d3:af:1a:b5:33:88:17:34:10:
         dc:88:0f:2d:3d:50:11:39:e8:a3:21:54:c8:5d:61:3c:93:d1:
         dd:cf:bf:b8:96:3a:02:28:89:db:61:59:a4:b6:6f:7e:9e:9c:
         7e:05:81:84:b1:93:c6:0f:a0:7e:da:5c:07:85:17:82:42:05:
         da:2a:f3:a6

Create PKCS12 keystore.

$ openssl pkcs12 -export -in client_milosz_certificate.pem -inkey client_milosz_certificate.key -out client.keystore.p12 -name "client_milosz" -password pass:clientcertpassword

Inspect PKCS12 files.

$ openssl pkcs12 -in client.keystore.p12 -info -password pass:clientcertpassword -nokeys
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: 8F 8B 48 87 16 96 DA 8A 4B D3 FA 88 DA 14 98 78 B3 CF 9A C0 
    friendlyName: client_milosz
subject=CN = client_milosz

issuer=O = ZooKeeper, OU = Backend, emailAddress = admin@example.org

-----BEGIN CERTIFICATE-----
MIIC5zCCAc8CFDZA/wmfY1spbKGa+nN5kr150S3yMA0GCSqGSIb3DQEBCwUAMEgx
EjAQBgNVBAoMCVpvb0tlZXBlcjEQMA4GA1UECwwHQmFja2VuZDEgMB4GCSqGSIb3
DQEJARYRYWRtaW5AZXhhbXBsZS5vcmcwHhcNMjEwODI4MDgwNTI3WhcNMjIwODI4
MDgwNTI3WjAYMRYwFAYDVQQDDA1jbGllbnRfbWlsb3N6MIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEAumuFN1kqBlMou3w+TNiiscDKizAuGb0sxQbDebGV
DKxBXcYGdn8CODZ0Q5a0rNak38vzh7hyG+LBNH+Xu89fvsIQ3WnkiDGlaDzX9puq
AsVx4LQNhulzA1oAoFjWjuDroVwtBKJFDI0XTxVf8gBN+v6C2IQj1wu6IgTNdarS
CkBry4soIQnvmbWr+MAtfgXsGIB9CibRiL1gnsDGcMRgyKOyfBCG7wZsZJtxJVRO
gh4eFT0X0/9PFKuCMeuMQ0Bm4tt+v2DrrObJhfrkLdBGF2G6PcH4yieICBLi1+xm
Y27UEAZ8ZIi1AdmFALQWsP2a8DhlzkVXU63g3PlGnmKdLQIDAQABMA0GCSqGSIb3
DQEBCwUAA4IBAQBbTb+LkcUWAo9j/GnG9d/DlEodWrgDsIt+LZoFkowSq7/x1TW5
ly9tslJPJZ2Qp20SMWhbEW3fBQ/f/OnwF99YwVfpO4lWjQqwivwOW72slUfgRgWs
3/UkbnN/kHT8SxuPmezXKgDwC9sJUKKYfTFsSSEFxDn1aT11cYWfkdl9GsZ2LjFP
qx7sowrel78h7h9rUPTFbUatUfJhVWaFda80VNPexC5pDEqNahsg2iomvSIaRZYz
highcsaJQSrTrxq1M4gXNBDciA8tPVAROeijIVTIXWE8k9Hdz7+4ljoCKInbYVmk
tm9+npx+BYGEsZPGD6B+2lwHhReCQgXaKvOm
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Configure ZooKeeper

Inspect default configuration.

$ cat conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/zookeeper/data
clientPort=2181
server.1=zookeeper1.example.org:2888:3888
server.2=zookeeper2.example.org:2888:3888
server.3=zookeeper3.example.org:2888:3888
4lw.commands.whitelist=*

Secure the ZooKeeper server communication with clients.

$ cat conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/zookeeper/data
secureClientPort=2181
server.1=zookeeper1.example.org:2888:3888
server.2=zookeeper2.example.org:2888:3888
server.3=zookeeper3.example.org:2888:3888
4lw.commands.whitelist=*

serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.trustStore.location=/opt/zookeeper/ca.pem
#ssl.trustStore.password=nopassword
ssl.keyStore.location=/opt/zookeeper/server.keystore.p12
ssl.keyStore.password=certpassword
client.secure=true

Restart service on each server.

$ sudo systemctl restart zookeeper

Test client certificate

Locally test client certificate.

$ CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
                   -Dzookeeper.ssl.trustStore.location=/opt/zookeeper/ca.pem
                   -Dzookeeper.ssl.keyStore.location=/opt/zookeeper/client.keystore.p12 -Dzookeeper.ssl.keyStore.password=clientcertpassword
                   -Dzookeeper.client.secure=true" /opt/zookeeper/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper/bin/../conf/zoo.cfg
Client port not found in static config file. Looking in dynamic config file.
grep: : No such file or directory
Client port not found in the server configs
Client port not found. Looking for secureClientPort in the static config.
Client port found: 2181. Client address: localhost. Client SSL: true.
Mode: leader

Create a client configuration file.

$ cat <<EOF | tee zkclient.cfg
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.trustStore.location=/opt/zookeeper/ca.pem
zookeeper.ssl.keyStore.location=/opt/zookeeper/client.keystore.p12
zookeeper.ssl.keyStore.password=clientcertpassword
zookeeper.client.secure=true
EOF

Configure NullAppender.

$ sudo -u zookeeper cat zookeeper/conf/log4j.properties 
[...]

# Null Appender
log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender

Use cli over secure connection.

$ sudo -u zookeeper ZOO_LOG4J_PROP="INFO,NullAppender" zookeeper/bin/zkCli.sh -client-configuration zookeeper/conf/zkclient.cfg -server zookeeper1.example.org config
/usr/bin/java
Connecting to zookeeper1.example.org

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
server.1=zookeeper1.example.org:2888:3888:participant
server.2=zookeeper2.example.org:2888:3888:participant
server.3=zookeeper3.example.org:2888:3888:participant
version=0