personal-site/src/static/data/blogs/nws-how-to-do-ssl.md

Background

NWS has a service that is tentatively called "Container Deployment Service (CDS)" (It was previously called "Cruise"). This service allows people to deploy a docker container to our cluster of kubernetes clusters. In it's current (read: early) form, NWS will add a GitHub Actions pipeline yaml file to the repo which a user wants to deploy. This yaml file will create a dockerfile, and build it into a docker container with 2 tags: an autogenerated semver, and the name of the branch. From here, the NWS API will store metadata about the application in a database and then add the kubernetes deployment manifests to a gitrepo (this is a newish thing that I learned about called gitops). Rancher Fleet periodically scans this repo, and when it sees the new yaml file, it deploys it to all kubernetes clusters on the NWS network. Then, we use DNS loadbalancing to balance the load between all of the clusters. Formerly, we used HAProxy to balance requests, we no longer do but HAProxy is still running on the servers (this detail will be important later).

The Issue

In order to provide a secure connection for end-users of applications hosted on NWS (and so that browsers won't spit out warning messages that the site is insecure), it is important to have SSL setup. This is rather complicated to do since we want:

1. Each application to have it's own certificate
2. Certificates to be able to auto-renew
3. Certificates to be able to auto-generate

We have 3 (main) options for enabling SSL on NWS:

1. We have people who host on NWS upload their own SSL certificates. This can be a pain since SSL certificates aren't super easy to generate, and the user would have to remember to upload new certificates.
2. (This is what applications on NWS currently do) Proxy traffic through Cloudflare, which uses it's "Flexible" SSL option to allow for the connection between the client and CF to be encrypted while the connection between CF and NWS is not. This is insecure as traffic between CF and NWS is not encrypted. This is not ideal but currently since it's just personal sites, it's not a huge deal. Additionally, this would require everyone who wants to host on NWS to use Cloudflare
3. We use a tool to autogenerate certificates. This would be very hard to do with NWS' load balancing, but ultimately may be the best option.

Getting auto generated SSL certificates is easy enough to do on a traditional webserver by using a tool such as certbot. However, since we have many instances of a single application running and we have multiple clusters running them, it would be hard to properly route requests to the pod that is running certbot. Additionally, trying to have all of the pods run certbot at once would not work since they would all generate unique certificate requests.

The Solution

The way NWS will work around this will be by having HAProxy to route all requests to the path .well-known/acme-challenge/* to an SSL generation server running on NWS. This server will run software that at some arbitrary time during the day will check the expiry of all SSL certificates, and any expiring within the next week will be renewed by certbot. Once renewed, they will be committed to the aforementioned git repo with the k8s manifest files. In the case of new applications, they will have their certificate request made automatically after creation.