Upgrading from Traefik 1.7 to Traefik 2.x

About 3 weeks ago, I took the time to upgrade my reverse proxy system to go from traefik 1.7 to traefik 2.4 and I want to talk through what I did in order to upgrade. I have a server at home (that this blog runs on as well as many other services) that has a lot of fun projects and things that I use to learn new skills as well as solidify my knowledge and understanding of new and exciting technologies and things. One of the main reasons why I have my server is to host a plex server (with many Terabytes of storage space so I don't have to delete media and can re-watch it) but I also have other media management tools included on my server, which provide the ability to better maintain the state of media that I have available to me. It also provides me with a server (self-hosted) that I can throw garbage projects on to simply mess around.

Before I get to the upgrade, I want to provide some historical context about my server (which will probably end up being its own blog post at some point). Before I went full-versed with a beefy 4U storage server in a rack sitting in my basement, I simply had a mid-tower case holding a CPU and as many hard drives as I could (which at the time was 8) and it was running on a strange linux mint distro with mdadm for software-based raid on those drives.  IN addition to all of that, I wrote custom nginx configs in order to appropriately route traffic from custom ports to a port 80 non-secure domain name (ddellspe.net) with subdomains for different services.  When I transitioned to my new server, there were a number of things I wanted to do. First, I wanted to leverage docker as much as possible.  At the time, at work, there was a lot of conversation happening about leveraging docker to deploy our services, so I figured that if I had experience running services via docker, that could help.  Second, I wanted to be able to better automate the exposure of services to the "public internet" in a safe and secure manner. This ended up with having to determine which services it would make sense to use in order to provide let's encrypt ssl certificates to my server to provde proper SSL.  Let's Encrypt has made securing the internet so easy that it totally makes 100% sense to leverage it wherever you can. Third, I wanted all of these processes to automate as much as possible, I wanted to be able to add a new docker container-based service and get the domain name/certificate generated with as minimal work as possible.  This lead me down the road of traefik.

When I first put together my reverse proxy configuration, I followed this guide which leveraged the latest (at the time) version of traefik in order to reverse proxy simple things (at the time, the major reverse proxy desire was plex, but now there's so much more including this blog). Configuration via docker labels made sense to me, it made the configuration a part of the docker image deployment (in docker-compose) and provided a consistent pattern for doing it going forward.  If the docker container had a default port to expose, you would only have to 1) enable proxy automation and 2) give it a domain name to leverage. I had to learn toml in order to configure the system, and I wasn't familiar with toml at the time, but started to better understand it when I compared to things like yaml later on.

Traefik had a migration guide from 1.x to 2.x that I had looked at, but I didn't particularly want to configure everything for traefik (the traefik container) via labels, because that seemed like a real easy way for something to be broken by an update.  For reference, all of my ghost configuration exists at a docker label level, and I find it to be somewhat annoying to have to go in and update, because most of the ghost documentation talks about config via config file, but I'm running ghost in a docker container, so I'm only really able to do config via docker labels, so that translation is somewhat annoying to have to continually make.  So, rather than docker labels, I decided to config via yaml file in the backend.  This was a change since my traefik 1.7 config was in toml and it differed quite a bit.  So, what's different? What did I have to do?  Here is my previous traefik.toml file that used to power my traefik 1.7 configuration (some items redacted for security/privacy)

defaultEntryPoints = ["https","http"]
insecureSkipTerify = true

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
    minVersion = "VersionTLS12"
    cipherSuites = [
      "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
      "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
      "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
      "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
    ]
  [entryPoints.traefik]
  address = ":8080"

[api]

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "ddellspe.net"
watch = true
exposedbydefault = false
insecureSkipVerify = true

[acme]
email = "<REDACTED>"
storage = "acme.json"
OnHostRule = true
entryPoint = "https"
  [acme.httpChallenge]
  entryPoint = "http"

[[acme.domains]]
main = "ddellspe.net"
sans = ["<REDACTED>"]

[file]

This configuration is pretty simple and straightforward, but I give a section-by-section breakdown here.  First, we have the entrypoints, this indicates that my reverse proxy is exposing both port 80  (http) and 443 (https) to the public and both are acceptable ways to enter into the reverse proxy system. insecureSkipVerify indicates that if what I'm proxying has its own ssl certificate, don't worry that it's not a verified ssl certificate.  Many of the services that I use have self-signed ssl certificates, and those don't verify with a root Certificate Authority, so they would give your browser a warning saying that it's likely that the site you are using isn't secure accordingly, since I'm putting my own SSL certificate in front of that via let's encrypt, I don't care that the internal ssl certificate is invalid. Next is the entrypoints configuration, which provides the ports for the http and https entry point and directs the http entry point to automatically redirect to the https (so if you type in ddellspe.net into your browser, in the back-end it will go from http://ddellspe.net directly to https://ddellspe.net (this is still active with the new config, but feel free to try to hit http://ddellspe.net and see that it does redirect to the https site.  The other item defined here is the configuration around expected tls certificate information, this includes providing at a minimum TLS 1.2 (TLS 1.1 was deprecated earlier in 2020, so a minimum of 1.2 would be web-standard at this point). TLS 1.3 is now supported by many modern browsers (Safari, Firefox, Chrome, Edge) so part of my upgrade is to move to TLS 1.3 being the minimum. The third entrypoint specified is the traefik entrypoint (this provides the dashboard view for all traefik configuration being exposed via the web gui). That goes along with the next section which enables that traefik configuration view [api]. After the api section is the docker configuration, which provides the docker socket used to read the labels from the docker images, and default configurations (watch provides the ability to watch for new docker images and changes in docker images, exposedByDefault means when true that all docker images will auto-expose via traefik, false means we'll have to set a label to expose them, insecureSkipVerify is the same as the overall file, but provides it specifically for the docker images. The last meaningful section is the acme section which sets up the configuration for let's encrypt within traefik (this is the reason why I really like traefik, simple acme configuration). The configuration specifies the storage location of the certifiates themselves, the type of challenge being used (http for my setup) and the default entrypoint to look for alignment with that default challenge. I then have a default certificate for my domain (ddellspe.net) and some SANs (Subject Alternative Names) for that domain itself. I also added a file section for reverse proxying items that aren't deployed as docker images.

This configuration worked very well for the time that I had it, it did all of the reverse proxying that I needed it to, but when I suggested traefik to friends, I didn't know how traefik 2.0 worked and most of the default examples now being searchable were for traefik 2.x. So, here is the same configuration, but for traefik 2.x for my server.

api:
  insecure: true
  dashboard: true

tls:
  options:
    default:
      minVersion: "VersionTLS13"
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: leresolver


certificatesResolvers:
  leresolver:
    acme:
      email: <REDACTED>
      storage: /acme/acme.json
      httpChallenge:
        entryPoint: web

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

  file:
    directory: /configs
    watch: true

First off, the difference in traefik 2.x to traefik 1.x is that you can have global certificate resolvers (leresolver for me stands for let's encrypt resolver) and other configuration has been set up to be multi-capable and available by configured name, like the tls options, etc. A new providers section now has all of the sources of endpoints to be proxying (docker and file in this example). The major differences are entrypoint http is now entrypoint web and entrypoint https is now entrypoint websecure, and I changed the locatino of the stored letsencrypt file (because it's changed structure). The major change other than this toml-to-yaml conversion is in the required change in labels for the individual docker images. Because I was strongly worried about all of how this worked, I decided to do a trial run on my windows desktop PC using WSL (Windows Subsystem for Linux) on a custom domain (ddellspe.hax) where everything except the actual letsencrypt certificate generating was happening. This provided the trial that I needed to make sure I COULD smoothly transition all of the provider setup from traefik 1.7 to traefik 2.4 (which was released as I was doing this work).

I've successfully migrated from traefik 1.7 to 2.4, it was a learning curve, and for SOME of my stuff, I've had to add some additional containers and pull out the let's encrypt containers in a slightly better way, but the reality is that I know I'm not the only person who's had to deal with all of this stuff, so I can always find another guide to help me out.