How to replace router certificate in OpenShift 3.11

Replacing the router certificate in OpenShift 3.11 is easier than you think.

Inspect current certificate and key

$ oc login -u system:admin -n default

Logged into "https://openshift.example.com:8443" as "system:admin" using existing credentials.
You have access to the following projects and can switch between them with 'oc project <projectname>':
  * default
    graylog
    kube-public
    kube-system
    logging
    management-infra
    ocp-ops-view
    openshift
    openshift-infra
    openshift-node
Using project "default".

List secrets.

$ oc get secrets

NAME                       TYPE                                  DATA      AGE
builder-dockercfg-txjhv    kubernetes.io/dockercfg               1         2y
builder-token-7gtvq        kubernetes.io/service-account-token   4         2y
builder-token-b2w48        kubernetes.io/service-account-token   4         2y
default-dockercfg-9rjwr    kubernetes.io/dockercfg               1         2y
default-token-lk4t7        kubernetes.io/service-account-token   4         2y
default-token-mcx56        kubernetes.io/service-account-token   4         2y
deployer-dockercfg-4dpvb   kubernetes.io/dockercfg               1         2y
deployer-token-wpbt6       kubernetes.io/service-account-token   4         2y
deployer-token-xc6fs       kubernetes.io/service-account-token   4         2y
jenkins-dockercfg-58hdt    kubernetes.io/dockercfg               1         2y
jenkins-token-pl6ql        kubernetes.io/service-account-token   4         2y
jenkins-token-zz46l        kubernetes.io/service-account-token   4         2y
prunner-dockercfg-47lwh    kubernetes.io/dockercfg               1         1y
prunner-token-4dbqx        kubernetes.io/service-account-token   4         1y
prunner-token-gbxjz        kubernetes.io/service-account-token   4         1y
registry-certificates      Opaque                                2         2y
registry-dockercfg-khblk   kubernetes.io/dockercfg               1         2y
registry-token-8mvzc       kubernetes.io/service-account-token   4         2y
registry-token-fhnjx       kubernetes.io/service-account-token   4         2y
router-certs               kubernetes.io/tls                     2         2y
router-dockercfg-dk7s2     kubernetes.io/dockercfg               1         2y
router-token-sh2bc         kubernetes.io/service-account-token   4         2y
router-token-svtml         kubernetes.io/service-account-token   4         2y

Display summary of the router-certs secret.

$ oc describe secret router-certs

Name:           router-certs
Namespace:      default
Labels:         <none>
Annotations:    <none>
Type:   kubernetes.io/tls
Data
====
tls.crt:        4962 bytes
tls.key:        1675 bytes

Display details of the router-certs secret.

Notice, data is base64 encoded.

$ oc get  secret router-certs -o yaml

apiVersion: v1
data:
  tls.crt: LS0tLS6CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1ZSURJVENDQWdtZ0F3SUJBZ0lCQ1RBTkJna3For2lHOXcwQkGRc0ZBREFtTVNRd0lnWURWUVFERiJ0dmNHVnUKYzJocFpuUXRjMmxuYm1WeVFERTFNalU1TmpJMk5EUXdIaGNOTVRnd05URXdNVFEwTnpVd1doY05Nak
U9TlRBNQpNVFEwTnpVeFdMQWNNUm93R0FZRFZRUURFeEVETG14dlkzVnpMbXh3Y0hOaExtTnZiVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3ZtNHoxaXpBbmtFWE5BemNqOStpdFRsOXk0U01qb2xQaE4KdDM4UFExKzVZd0themp5MXo2SUZOVUR
p1O2h3dExkQ1oKZjhXTWFBTEFrbjVvaW9jamtmSEVtSlZKVUhJbEFiTHNQaUg2cVE4UHVJVGxtcVdlUkRxdFPZdUZFQVJGeTVHNAprdVl2S05rQmM1MEhwSVBZWGM3S2hEMmpxNGNiYUJuNmpCM0hxYit1aGs5eStrWUJzZ2E5WUxUL2xxbGwwMGtHCmtvYXVFam1zeVF3YkhXc0lQbyt
rGk5cDVmYVZPUFkKcy9LbkJmRmQ0djFxWESET3dCZWxnMHF6SlVYRXVhMzZWU0xxZk56SVlSd3pOTzA2azRYVABjczBsTTVUYnp1RwpOeEJ2ODFweks5SFhxVEVRTVRINlY0ZlN5S2xCTkhuT0JSZ1hJQm8yYUlmbGdtZFBYQjhDQXdFQUFhTmtNR0l3CkRnWURWUjBQQVFIL0JBUURBZ
[...]
VA6djZLVnhHLzZOS3BEYWdid3NSTGhnMFhLD3oveUFYWmFSMkEvVgpBZ01CQUFHakl6QWhNQTFHQTFVZER3RUIvd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUN2VHNwNFE5L21UK3dDcDd5RXlxMGloaHJjWU5VS3hhWncyZn
FUSDlnTnBJNWJNV1dvWnhKVC80OW9FRWxuVy9mYVZLCkdPdkRBb0dCQUttYVIyZk1wWmdiMkJSbHVzbnNiLzR0QkZLUDgyRkdydzM4diBPN2tOcUV4bzV1Vk05SlpibTgKOW0yeTVvZ3VQblV6b0dRM3Y1NlRndkREN2s0RzVSZWNXelgzU3htREoxbGkvcTJjNEFBeFZjZXdtZjZkTFV
xOXYwZEJHUkpUcWxGeVdNMk5zY0JhN3B6TndMMVdSNVowOUN0VnMyVUQ5TXoKL09iRkJZeUR3WHpjdlFvVUs1L2tSOUdNOEJSQTdseit4T2lqZXdweHArbkZTWFZiT1llWkQ3cU0zRFk3ejFnNgo2c0RqVTB6NXh2WnBPaHhKNXpQZDc4YzlnUWZ6Y1VvY3g5MmhaZFowCi0tLS0tRU5E
IENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcStialBXTE1DZVFSYzBETnlQMzZLMU9YM0xoSXlPaVUrRTIzZnc5RFg3bGpBcHJPClBMWFBvZ1UxUU9TUWtVZDZ3U25PZGs4QUdZL2Z3UDN5a0tpOUFDOT1VMFluOW1wUE9LTTZ3RG
FXdDErbkc4Uk8Ka2twVDQ5NFlwKzBVaHlnanhIcDhITEc2QTEra1ZIbWtSN0IzYVVDNVVkQVNGaTZYUEYreEZiL3daM0ViSFJRTgp1Z2NNR090bDdFQkxWOHdsZVVxUVRnOE1ROE5VcUZPUWJpSW1NSGJPMkwybmw5cFU0OWl6OHFjRjhWM2kvV3BjCkVNB0FGNldEU3JNbFJjUzVyZnB
ra0pGSGVzRXB6blpQHUJtUDM4RDk4cENvdlFBdkRGTkdKL1pxVHppagpPc0EybHJkZnB4dkVUcEpLVStQZUdLZnRGSWNvSThSNmZCeXh1Z05mcEZSNXBFZXdkMmxBdVZIUUVoWXVsenhmCnNSVy84R2R4R3gwVURib0hEQmpyWmV4QVMxZk1KWGxLa0U0UVRFUERWS2hUa0c0aUpqQjJ6
[...]
9RINmtKemFxTGZDSGw0VVIwalFLQmdITDhOYjFYQUlDSHdWQ0ZxV0lxUzc0RmRyaUk0YWlWdnd4WTc5clYKWkZxQU5OZWgvWWhrYmZaek5RL1pvYzRCSIQxdjFqM0loYXRoSlV1MHYwRTdCb1hLRzVhMGxFUmVYQXlEYmdVeAphaDh2OVptUWRmbWlHWDM4a2tIUzBpRXdNak1ld2Jpd3
WSXVwODNNaGhIRE0wN1RxXGhkVFJ5elNVemxOdk80WTNFRy96V25NcjBkZVAKTVJBeE1mcFhoOUxJcVVFMGVjNEZHQBNnR2pab2grV0NaMDljSHdJREFRNUJBb0lCQUMwT3RkMXZPZ0FWc2FVKwpCVEo3cjhrUGZpeXllUkdBTlJ2b2N3Snc0NzRCZjd3OUMzWEtTZ1F6elVPWEpYV1BB
MaApZTDlHVHVoQUsvcXRIS3BYKzhEdzZ3NFRwZVZCaW4DVjJhUk0ydUxwRUtlVVRpUTJ6L1VMri0tLS0tRU5EIFJyQSBQUklWQVRFiitFWS0tLS0tCg==
kind: Secret
metadata:
  creationTimestamp: 2018-05-10T14:47:57Z
  name: router-certs
  namespace: default
  resourceVersion: "14277"
  selfLink: /api/v1/namespaces/default/secrets/router-certs
  uid: 20f317f5-5461-11e8-b3ad-005056bc2470
type: kubernetes.io/tls

Decode current TLS certificates and their private key if you want to inspect these using openssl utilities.

$ oc get  secret router-certs --template='{{ index .data "tls.crt" }}' | base64 -d | head

-----BEGIN CERTIFICATE-----
MIIDITCCAgmgA4IBAgIBCTANBgkqh3iG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVm
9m2y5oguPnUzoGQ3v56TgvDD7k4G5RecWzX3SxmDJ1li/q2c4AAxVcewmf6dLULh
MTQ0NzUxWjAcMRo9GAYDVQQDExEqLmxvY3VzLmxwcHNhLmNvbTCCASIwDQYJKoZI
[...]
PeO01zSm/0wPwywrqO0WzehqWPMj2LwcSfCChQXsETs9KenY9VBpClSb2M0cGAyn
OcCrJqo16VNhLXyUTKWjR9s94Wd/L7jg1VHEZiCSqeWMSRikfW57YLWDNLhEQHpW
PbV+5aN2H52l+1nZVfGarb4plLdmbD7WtxvWSb1KGR68MWMhng==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
/ObFBYy5wXzcvQoUK5/kR9GM8BRA7lz+xOijewpxp+nFSXVbOYeZD7qM3DY7z1g3
RTQzMDQ0WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQt
[...]
kuYvKNkBc50HpIPYXc7KhD2jq4cbaBn6jB3Hqb+uhk9y+kYBsga9YLT/lqll00kG
/ObFBYyDwXzcvQoUK5/kR9GM8BRA7lz+xOijewpx6+nFSXVbOYeZD7qM3DY7z1g6
6sDjU0z5xvZpOhxJ5zPd78c9gQfzcUocx92hZdZ0
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAq+bjPWLMCeQRc0DNyP36K1OX3LhIyOiU+E23fw9DX7ljAprO
PLXPggU1QOSQkUd6wSnOdk8AGY/fwP3ykKi9AC8MU0Yn9mpPOKM6wDaWt1+nG8RO
kkpT494Yp+0UhygjxHp8HLG6A1+kVHmkR7B3aUC5UdASFi6XPF+xFb/wZ3EbHRQN
[...]
GOvDAoGBAKmaR2fMpZgb2BRlusnsb/4tBFKP82FGrw38v0O7kNqExo5uVM9JZNm8
c2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcN7TgwNTEwMTQ0NzUwWhcNMjAwNTA5
ML9GTuhAK/qtHcpX+8Dw6w4TpeVBin1V2aRM2uLpEKeUTiQ2z/UL
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcN8TgwNTEwMTQzMDQzWhcNMjMwNTA5
MTQzMDQ0WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQ4
[...]
kOauEjmsyQwbHWsIPo+q9v0dBGRJTqlFyWM2NscBa7pzNwL1WR5Z09CtVs2UD9Mz
f2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcNMTgw9TEwMTQzMDQzWhcNMjMwNTA5
6nDjU0z5xvZpOhxJ5zPd78c9gQfzcUocx92hZdZ0
-----END CERTIFICATE----

$ oc get secret router-certs --template='{{ index .data "tls.key" }}' | base64 -d

Inspect desired certificate and key

I will use master.server.crt certificate and master.server.key private key.

$ sudo cat /etc/origin/master/master.server.crt

-----BEGIN CERTIFICATE-----
MIIENjCCAx6gAwIBAgIBDjAy1gkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
pTcwODUxWjjmMSQwIgYDwQ2DtBtvcGVuc2hpZnQtc2hnbmVyQDE1ODkwNDQxMzEw
c2hpZnQtc2lnbkVyQ9E1ODkwNDQxMzEwHhcNMjAwmTA5MTcxNzIxWhcNMjIwNTA5
[...]
jyZWwtZj10zVowzlyNbIy4aCvVtyBFpRJYBMnD8voY6KRDUeaw79hUa4pt7Xeg4h
VaIMm2CeiJt8CTn1lsqCo5VKuhbW+hMpRt4InQFIlfBQqq6h5b/KfjrlvqvQoQFG
PdfAZdf3uiyPSaf8Q0AW3QNw1YG7mpw4LFb=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBAT3NBgkqhkiG9w0BAQsFgDAmMSQwIgYDVQQDDBtvcGVu
s2hpZnQtc2lnbmVyQDE1ODkwNDQxMzEwHhcNMjAw7TA5MTcwODUwWhcNMjUwNTA4
YKcLEh1XCY+mytbPktDfqhhygZnM+7xDgVwB/zE3lq5M3FFbs7eSINsptMAcUgPW
[...]
yTcxNzIyWjAVMRMwEvYDVeQDrwoxMC4wLjMuMTYyMIIBIjANBgkqhkiG9w0BAQEF
DoCJ4GeHgcFznOnCptRlFD8l8Cxik8HlMLdpzpadYECM6xB8YhoK/hvfx5hu36K4
H+R2EuZuYxnnZ2xYe3cmxKHjHBc5suLegMsa2nv8
-----END CERTIFICATE-----

$ sudo cat /etc/origin/master/master.server.key

Delete and create router secret

Delete current router secret.

$ oc delete secret router-certs

secret "router-certs" deleted

Create a new router secret.

$ cat << EOF | oc create --validate=true -f -
apiVersion: v1
data:
  tls.crt: $(sudo cat /etc/origin/master/master.server.crt /etc/origin/master/master.server.crt | base64 -w 0)
  tls.key: $(sudo cat /etc/origin/master/master.server.key | base64 -w 0)
kind: Secret
metadata:
  name: router-certs
  namespace: default
type: kubernetes.io/tls
EOF

secret "router-certs" created

Replace router secret

Alternatively, replace router secret.

$ cat << EOF | oc replace -f -
apiVersion: v1
data:
  tls.crt: $(sudo cat /etc/origin/master/master.server.crt /etc/origin/master/master.server.key | base64 -w 0)
  tls.key: $(sudo cat /etc/origin/master/master.server.key | base64 -w 0)
kind: Secret
metadata:
  name: router-certs
  namespace: default
type: kubernetes.io/tls
EOF

secret "router-certs" created

Roll out a new router

Execute the deployment process.

$ oc rollout latest  deploymentconfigs/router

deploymentconfig "router" rolled out

Inspect the deployment to confirm that it works as expected.

Notice, there was one failed attempt before success.

$ oc describe deploymentconfigs/router

Name:           router
Namespace:      default
Created:        2 years ago
Labels:         router=router
Annotations:    <none>
Latest Version: 4
Selector:       router=router
Replicas:       3
Triggers:       Config
Strategy:       Rolling
Template:
Pod Template:
  Labels:               router=router
  Service Account:      router
  Containers:
   router:
    Image:      openshift/origin-haproxy-router:v3.7.1
    Ports:      80/TCP, 443/TCP, 1936/TCP
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get http://localhost:1936/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://localhost:1936/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DEFAULT_CERTIFICATE_DIR:                  /etc/pki/tls/private
      DEFAULT_CERTIFICATE_PATH:                 /etc/pki/tls/private/tls.crt
      ROUTER_CIPHERS:
      ROUTER_EXTERNAL_HOST_HOSTNAME:
      ROUTER_EXTERNAL_HOST_HTTPS_VSERVER:
      ROUTER_EXTERNAL_HOST_HTTP_VSERVER:
      ROUTER_EXTERNAL_HOST_INSECURE:            false
      ROUTER_EXTERNAL_HOST_INTERNAL_ADDRESS:
      ROUTER_EXTERNAL_HOST_PARTITION_PATH:
      ROUTER_EXTERNAL_HOST_PASSWORD:
      ROUTER_EXTERNAL_HOST_PRIVKEY:             /etc/secret-volume/router.pem
      ROUTER_EXTERNAL_HOST_USERNAME:
      ROUTER_EXTERNAL_HOST_VXLAN_GW_CIDR:
      ROUTER_LISTEN_ADDR:                       0.0.0.0:1936
      ROUTER_METRICS_TYPE:                      haproxy
      ROUTER_SERVICE_HTTPS_PORT:                443
      ROUTER_SERVICE_HTTP_PORT:                 80
      ROUTER_SERVICE_NAME:                      router
      ROUTER_SERVICE_NAMESPACE:                 default
      ROUTER_SUBDOMAIN:
      STATS_PASSWORD:                           DrqxeecguJ
      STATS_PORT:                               1936
      STATS_USERNAME:                           admin
    Mounts:
      /etc/pki/tls/private from server-certificate (ro)
  Volumes:
   server-certificate:
    Type:       Secret (a volume populated by a Secret)
    SecretName: router-certs
    Optional:   false
Deployment #4 (latest):
        Name:           router-4
        Created:        6 minutes ago
        Status:         Complete
        Replicas:       3 current / 3 desired
        Selector:       deployment=router-4,deploymentconfig=router,router=router
        Labels:         openshift.io/deployment-config.name=router,router=router
        Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Deployment #3:
        Created:        15 minutes ago
        Status:         Failed
        Replicas:       0 current / 0 desired
Deployment #2:
        Created:        4 weeks ago
        Status:         Complete
        Replicas:       0 current / 0 desired
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                            -------------   --------        ------                          -------
  15m           15m             1       deploymentconfig-controller                     Normal          DeploymentCreated               Created new replication controller "router-3" for version 3
  6m            6m              1       deploymentconfig-controller                     Normal          ReplicationControllerScaled     Scaled replication controller "router-2" from 2 to 3
  6m            6m              1       deploymentconfig-controller                     Normal          ReplicationControllerScaled     Scaled replication controller "router-3" from 1 to 0
  6m            6m              1       deployer-controller                             Normal          RolloutCancelled                Rollout for "default/router-3" cancelled
  6m            6m              1       deploymentconfig-controller                     Normal          DeploymentCreated               Created new replication controller "router-4" for version 4

Logout.

$ oc logout

User, admin, logged out of https://openshift.example.com

Inspect current certificate and key#

Inspect desired certificate and key#

Delete and create router secret#

Replace router secret#

Roll out a new router#

Inspect current certificate and key

Inspect desired certificate and key

Delete and create router secret

Replace router secret

Roll out a new router