Renew Certificates automatically in OpenShift 4

7 minute read

How can we generate certificates for our apps and sign them with a free and trusted Certificate Authority? And most importantly, how can we automate the process of generating and renewing these certificates?

Let’s start!

First of all, we will use Let’s Encrypt certificates for this purpose. Let’s Encrypt is an automated and open Certificate Authority that can be used to sign our certificates without spending a dime (or euro in my case :D).

But to sign the certificate with the Let’s Encrypt CA, some steps must be executed. That’s where the magic of the cert-manager operator can be very helpful.

Installation of the cert-manager

Create a namespace to run cert-manager in:

# oc create namespace cert-manager
namespace/cert-manager created

Disable resource validation on the cert-manager namespace

# oc label namespace cert-manager certmanager.k8s.io/disable-validation=true
namespace/cert-manager labeled

NOTE: The –validate=false flag is added to the oc apply command above else you will receive a validation error relating to the caBundle field of the ValidatingWebhookConfiguration resource.

Install the cert-manager components (CRDs, cert-manager and webhook component):

# oc apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.8.1/cert-manager-openshift.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created

Check the resources generated by the installation:

# oc get pod -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-68cfd787b6-cz27g              1/1     Running   0          68s
cert-manager-cainjector-5975fd64c5-4wpxs   1/1     Running   0          69s
cert-manager-webhook-5c7f95fd44-tdrpt      1/1     Running   0          68s

# oc get clusterrole | grep cert-manager
cert-manager                                                           3m50s
cert-manager-cainjector                                                3m50s
cert-manager-edit                                                      3m50s
cert-manager-view                                                      3m50s
cert-manager-webhook:webhook-requester                                 3m50s

# oc get deployment -n cert-manager
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           4m10s
cert-manager-cainjector   1/1     1            1           4m11s
cert-manager-webhook      1/1     1            1           4m11s

Patch the certmanager deployment to allow resolving an external DNS (8.8.8.8)

Because the cert-manager deployment has the spec dnsPolicy: ClusterFirst, it cannot reach an external DNS to perform the DNS Challenge and communicate with the Let’s Encrypt API (one of the reasons for deploying cert-manager).

For this reason, a patch for the cert-manager Deployment must be applied:

Original

# oc get deploy cert-manager -n cert-manager -o yaml | grep dnsPolicy
dnsPolicy: ClusterFirst

Patched

# oc get deploy cert-manager -n cert-manager -o yaml
...
        terminationMessagePolicy: File
      dnsConfig:
        nameservers:
        - 8.8.8.8
      dnsPolicy: None
      restartPolicy: Always
...

NOTE: dnsPolicy to None allows Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the dnsConfig field in the Pod Spec.

For more information, check the DNS Pod Service Kubernetes Guide

# oc get pod -n cert-manager
NAME                                       READY   STATUS              RESTARTS   AGE
cert-manager-679fd5459-868cs               0/1     ContainerCreating   0          11s
cert-manager-68cfd787b6-cz27g              1/1     Running             0          61m
cert-manager-cainjector-5975fd64c5-4wpxs   1/1     Running             0          61m
cert-manager-webhook-5c7f95fd44-tdrpt      1/1     Running             0          61m

# oc exec -ti cert-manager-679fd5459-868cs -n cert-manager /bin/sh
~ $ nc www.marca.com -zv 443
www.marca.com (151.101.37.50:443) open

Setting up Issuers

Before you can begin issuing certificates, you must configure at least one Issuer or ClusterIssuer resource in your cluster.

These represent a certificate authority from which signed x509 certificates can be obtained, such as Let’s Encrypt, or your own signing key pair stored in a Kubernetes Secret resource. They are referenced by Certificate resources in order to request certificates from them.

Issuer: scoped to a single namespace, and can only fulfill Certificate resources within its own namespace. Useful in a multi-tenant environment where multiple teams or independent parties operate within a single cluster.
ClusterIssuer: cluster wide version of an Issuer. It is able to be referenced by Certificate resources in any namespace.

NOTE: cert-manager supports a number of different issuer backends, each with their own different types of configuration. In our case, ACME issuer backend will be used due to Let’s Encrypt certificates will be generated.

ACME issuers in Cert-Manager

Cert-manager can be used to obtain certificates from a CA using the ACME protocol. The ACME protocol supports various challenge mechanisms which are used to prove ownership of a domain so that a valid certificate can be issued for that domain.

One such challenge mechanism is DNS-01. With a DNS-01 challenge, you prove ownership of a domain by proving you control its DNS records. This is done by creating a TXT record with specific content that proves you have control of the domains DNS records.

NOTE: Let’s Encrypt does not support issuing wildcard certificates with HTTP-01 challenges. To issue wildcard certificates, you must use the DNS-01 challenge.

Generate required IAM Policy and User for ACME Cert-Manager Issuer

The ACME Issuer type represents a single Account registered with the ACME server.

When you create a new ACME Issuer, cert-manager will generate a private key which is used to identify you with the ACME server.

To set up a basic ACME issuer, you should create a new Issuer or ClusterIssuer resource.

But first of all, cert-manager requires an IAM Policy that must be defined.

First, generate an IAM policy with the permissions listed below:

# aws iam list-policies | grep acme
            "PolicyName": "acme-route53",
            "Arn": "arn:aws:iam::920348280276:policy/acme-route53"

    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ListHostedZonesByName",
            "Resource": "*"
        }
    ]
}

Generate an IAM user and attach the new IAM Policy:

# aws iam  list-users | grep cert
            "UserName": "cert-manager-iam",
            "Arn": "arn:aws:iam::920348280276:user/cert-manager-iam"

# aws iam list-attached-user-policies --user-name cert-manager-iam
{
    "AttachedPolicies": [
        {
            "PolicyName": "acme-route53",
            "PolicyArn": "arn:aws:iam::920348280276:policy/acme-route53"
        }
    ]
}

Generate a secret with the secret-access-key of the new user generated:

# oc create secret generic --from-literal=secret-access-key=xxxxxxxxxx-n cert-manager acme-route53
secret/acme-route53 created

# oc get secret acme-route53 -n cert-manager -o yaml
apiVersion: v1
data:
  secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: "2019-08-09T10:17:56Z"
  name: acme-route53
  namespace: cert-manager
  resourceVersion: "1493413"
  selfLink: /api/v1/namespaces/cert-manager/secrets/acme-route53
  uid: f4dc42b9-ba8e-11e9-9ee8-06221c570820
type: Opaque

Generating ACME Issuers for AWS Route 53

Generate the ClusterIssuer resource:

apiVersion: v1
items:
- apiVersion: certmanager.k8s.io/v1alpha1
  kind: ClusterIssuer
  metadata:
    name: apps-ocp4-dev-opentlc-com
  spec:
    acme:
      dns01:
        providers:
        - name: dns
          route53:
            accessKeyID: yyyy
            hostedZoneID: zzzz
            region: eu-central-1
            secretAccessKeySecretRef:
              key: secret-access-key
              name: acme-route53
      email: rcarrata@redhat.com
      privateKeySecretRef:
        name: cluster-issuer
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Check that the clusterissuer is generated properly:

# oc get clusterissuer
NAME                        AGE
apps-ocp4-dev-opentlc-com   2m12s

Issuing Certificates

The Certificate resource type is used to request certificates from different Issuers.

A Certificate resource specifies fields that are used to generate certificate signing requests, which are then fulfilled by the issuer type you have referenced.

# oc get certificates -n openshift-ingress -o yaml
apiVersion: v1
items:
- apiVersion: certmanager.k8s.io/v1alpha1
  kind: Certificate
  metadata:
    name: apps-ocp4-dev-xbyorange-com
    namespace: openshift-ingress
  spec:
    acme:
      config:
      - dns01:
          provider: dns
        domains:
        - '*.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com'
    commonName: '*.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com'
    dnsNames:
    - '*.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com'
    issuerRef:
      kind: ClusterIssuer
      name: apps-ocp4-dev-opentlc-com
    secretName: apps-ocp4
    duration: 2160h #90d
    renewBefore: 360h #15d
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Certificates specify which issuer will be used to obtain the certificate by specifying the spec.issuerRef field (in our case, the cluster issuer created in the step above).

Furthermore, the Certificate resource will generate the certificate for the wildcard domain (the route *.apps in our cluster) with a specific duration (90d) and will be renewed automatically 15d before its expiration.

The signed certificate (Let’s Encrypt) for the default ingress controller exposing *.apps routes will be generated automatically in the openshift-ingress namespace and stored in a Secret resource named apps-ocp4, once the issuer has successfully issued the requested certificate.

The Certificate will be issued using the issuer named ca-issuer in the openshift-ingress namespace.

# oc logs -f cert-manager-679fd5459-868cs
I0809 12:08:41.656962       1 dns.go:101] Presenting DNS01 challenge for domain
"apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com"
I0809 12:09:14.982797       1 dns.go:112] Checking DNS propagation for
"apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com" using name servers: [8.8.8.8:53]
I0809 12:09:15.959388       1 dns.go:124] Waiting DNS record TTL (60s) to allow propagation of DNS
record for domain "_acme-challenge.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com."
I0809 12:10:15.959599       1 dns.go:126] ACME DNS01 validation record propagated for
"_acme-challenge.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com."

After waiting some minutes for the DNS Challenge between the cert-manager operator and the ACME API, a Certificate is generated and stored in a Secret (check the SecretName value in the Certificate CRD above):

# oc get secrets -n openshift-ingress
NAME                           TYPE                                  DATA   AGE
apps-ocp4                      kubernetes.io/tls                     3      5m51s

# oc get certificates -n openshift-ingress
NAME                        READY   SECRET      AGE
apps-ocp4-dev-opentlc-com   True    apps-ocp4   8m33s

Patching the OpenShift Routers with the new certificates

Update the Ingress Controller configuration with the newly created secret:

# oc patch ingresscontroller.operator default --type=merg
e -p '{"spec":{"defaultCertificate": {"name": "apps-ocp4"}}}' -n openshift-ingress-operator

ingresscontroller.operator.openshift.io/default patched

# oc get pod -n openshift-ingress
NAME                              READY   STATUS              RESTARTS   AGE
router-default-79998d9946-lllvj   0/1     ContainerCreating   0          12s
router-default-84ff5bdcb8-ktmk9   1/1     Running             0          3d19h

After couple of minutes, the default Certificate exposed is updated and now serves the certificate generated by the Cert-Manager:

# oc get ingresscontroller -n openshift-ingress-operator -o yaml | grep -A1 defaultCertificate
    defaultCertificate:
      name: apps-ocp4

Check the Certificate exposed by the OCP routers

If we check the certificate exposed by the OpenShift Routers with openssl, we can see that the certificate exposed is the one generated by Let’s Encrypt:

# openssl s_client -showcerts -servername console-openshift-console.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com  -connect  console-openshift-console.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com:443 </dev/null | grep Issuer
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.apps.rcarrata-ipi-aws.8237.sandbox258.opentlc.com
verify return:1
DONE

NOTE: Opinions expressed in this blog are my own and do not necessarily reflect that of the company I work for.

Happy OpenShifting!!!

Share on

Twitter Facebook LinkedIn

Roberto Carratalá

Renew Certificates automatically in OpenShift 4

Installation of the cert-manager

Setting up Issuers

ACME issuers in Cert-Manager

Generate required IAM Policy and User for ACME Cert-Manager Issuer

Generating ACME Issuers for AWS Route 53

Issuing Certificates

Patching the OpenShift Routers with the new certificates

Check the Certificate exposed by the OCP routers

Share on

You may also enjoy

How to Use Generative AI for Securing Cloud Infrastructures (Part II)

How to Use Generative AI for Securing Cloud Infrastructures

Deploying an AI ChatBot in Azure Red Hat OpenShift fully integrated with Azure OpenAI

Exposing apps using Application Gateway LB in Private ARO clusters