Set up an automated HTTP/2 reverse proxy for your Docker containers using H2O, Docker-gen, Let's Encrypt and Ansible on a single host

#docker #h2o #ansible #ssl #tls #letsencrypt #http2 #docker-gen

TL;DR: Check out the Ansible role on GitHub.

What?

It took me some time to understand why I should care about web application containerization for other reasons than “this is the future!“.

But I actually found that Docker APIs approach is quite interesting when it comes to shipping and scaling a web application and I particularly like the analogy between containers and cartridges that Shopify drew on their engineering blog.

There are multiple obvious reasons why you may want to achieve zero-downtime deployment using Docker. One way to achieve this is to use a reverse proxy in front of containers (if your application is stateless).
Jason Wilder came up with Docker-gen and a ready-to-use solution using nginx.

For a personal small-scale project, I decided to build a similar solution that makes Docker-gen, H2O and Let’s encrypt work all together using Ansible.

Why H2O?

In one acronym : HTTP/2.
HTTP/2 brings numerous benefits such as multiplexing and server push.

H2O is less full-featured and battle-tested than Nginx and Apache but unlike Nginx or Apache, H2O was built from ground up to do HTTP/2 and it takes full advantage of HTTP/2 features. It also provides a reverse proxy implementation module (using a HTTP/1 client) and I think it it worth giving it a try.
Thanks to projects such as Let’s encrypt, it’s now possible to obtain TLS/SSL certificates for free and take advantage of these features.

For more information about H2O, I recommend these slides.

Okay, let’s think about it!

When we start a new web app container on the docker host, we want:

TLS/SSL certificates to be created automatically using a Let’s encrypt client
H2O configuration to be updated automatically.

If the certificates do not exist yet, it would give us a process like this:

Update H2O configuration with a new virtual host and serve .well-known/acme-challenge through HTTP on port 80
Reload H2O
Run Let’s encrypt client to create certificates using a configuration file generated by docker-gen and the webroot method
Update H2O configuration again to use the new certificates
Reload H2O

If the certificates already exists, we can skip the 3 first steps.

It would be quite convenient if we could run H2O and Let’s encrypt client in containers instead of installing all dependencies on the docker host.
We could run H2O in container by using this image.
For running the Let’s encrypt client, I built a docker-gen friendly certbot (official Let’s encrypt client) image that runs with a non-root user.

We could actually bundle all services (Docker-gen, H2O and Certbot) in a single container and use forego to start them but I would prefer a separate containers approach for different reasons:

I want to prevent having the docker socket (required by docker-gen) bound to a publicly exposed container service like H2O
It allows using separate existing and well written docker images (Unix philosophy: Do One Thing and Do It Well)

Since www data, certificates and H2O configuration are resources that would need to be shared between our services, we will have to bind some volumes from the host and set up different levels of access (H2O need read-only on www data and certs, etc.).

We would basically end up with something like this:

Okay, let’s write this!

First, let’s create a task to install docker-gen on the host:

tasks/docker-gen-install.yml

---
- get_url:
    url: https://github.com/jwilder/docker-gen/releases/download/0.7.3/docker-gen-linux-amd64-0.7.3.tar.gz
    dest: ~/docker-gen.tar.gz
    checksum: "sha1:9bd460f66e2e26323ea439c11956a145bb305c53"
- shell: tar -xf ~/docker-gen.tar.gz -C /usr/local/bin
- file: path=~/docker-gen.tar.gz state=absent

Then let’s build a docker-gen template (using the Go text/template language) for H2O configuration:

files/h2o.tmpl

# Base conf
user: nobody
http2-reprioritize-blocking-assets: ON
send-server-name: OFF
file.dirlisting: OFF
access-log: /dev/stdout
error-log: /dev/stderr
compress: ON

# Listen to both HTTP and HTTPS on start
hosts:
  default:
    listen:
      host: 0.0.0.0
      port: 80
    listen:
      host: 0.0.0.0
      port: 443
      ssl:
        key-file: /etc/h2o/ssl/self-signed.key
        certificate-file: /etc/h2o/ssl/self-signed.crt
    paths:
      /:
        file.dir: /srv/www/default

{{ define "proxy" }}
      /:
        proxy.preserve-host: ON
        proxy.timeout.keepalive: 0
        {{ if .Address }}
        {{ if (and .Container.Node.ID .Address.HostPort) }}
        proxy.reverse.url: "http://{{ .Container.Node.Address.IP }}:{{ .Address.HostPort }}"
        {{ else }}
        proxy.reverse.url: "http://{{ .Address.IP }}:{{ .Address.Port }}"
        {{ end }}
        {{ else }}
        proxy.reverse.url: "http://{{ .Container.IP }} down"
        {{ end }}
        setenv:
          HTTP_PROXY: ""
{{ end }}

{{ define "host" }}
hosts:
  "{{ .Host }}:80":
    listen:
      host: 0.0.0.0
      port: 80
    paths:
      /.well-known/acme-challenge:
        file.dir: /srv/www/letsencrypt/{{ .Host }}/.well-known/acme-challenge/
  {{ if (exists (printf "/etc/letsencrypt/live/%s/privkey.pem" .Host)) }}
      "/":
        redirect: https://{{ .Host }}/
  "{{ .Host }}:443":
    listen:
      host: 0.0.0.0
      port: 443
      ssl:
        certificate-file: "/etc/letsencrypt/live/{{ .Host }}/fullchain.pem"
        key-file: "/etc/letsencrypt/live/{{ .Host }}/privkey.pem"
    paths:
      {{ template "proxy" .Proxy }}
  {{ end }}
{{ end }}

{{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
{{ if (ne $host "") }}
{{ range $container := $containers }}
  {{ $addrLen := len $container.Addresses }}
  {{ if eq $addrLen 1 }}
    {{ $address := index $container.Addresses 0 }}
    {{ $proxy := (dict "Container" $container "Address" $address) }}
    {{ template "host" (dict "Host" $host "Proxy" $proxy) }}
  {{ else }}
    {{ $port := coalesce $container.Env.VIRTUAL_PORT "80" }}
    {{ $address := where $container.Addresses "Port" $port | first }}
    {{ $proxy := (dict "Container" $container "Address" $address) }}
    {{ template "host" (dict "Host" $host "Proxy" $proxy) }}
  {{ end }}
{{ end }}
{{ end }}
{{ end }}

Here is what we do here:

We listen to both HTTP and HTTPS on start using a default virtual host
We create one virtual host per container using the VIRTUAL_HOST and VIRTUAL_PORT environment variables
For each virtual host, we listen to port 80 for serving the .well-known/acme-challenge for Let’s encrypt and to redirect other requests to HTTPS
For HTTPS, we use certificates generated by Let’s encrypt only if they already exists

Now, let’s write a template to generate a very simple configuration file with a list of domains to validate/update with Certbot:

files/domains.tmpl

{{ range $host, $container := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
{{ if (ne $host "") }}
{{ $host }}
{{ end }}
{{ end }}

Let’s write a bash script that will execute the Certbot container letsencrypt-docker-gen-updater (we will create it later), update H2O configuration and gracefully reload H2O.
This script could also be used for periodically update the certificates.

files/letsencrypt-docker-gen-update.sh

readarray DOMAINS < /etc/letsencrypt/domains.conf
DOMAINS_COUNT=${#DOMAINS[@]}
if [ $DOMAINS_COUNT -gt 0 ]
then
    docker start -a letsencrypt-docker-gen-updater
    docker-gen -onlyexposed /etc/docker-gen/templates/h2o.tmpl /etc/h2o/h2o.conf
    docker kill -s HUP h2o
fi

We can now write the docker-gen main config file:

files/docker-gen.cfg

[[config]]
template = "/etc/docker-gen/templates/h2o.tmpl"
dest = "/etc/h2o/h2o.conf"
onlyexposed = true
watch = true
notifycmd = "docker kill -s HUP h2o"

[[config]]
template = "/etc/docker-gen/templates/domains.tmpl"
dest = "/etc/letsencrypt/domains.conf"
onlyexposed = true
watch = true
notifycmd = "/usr/local/bin/letsencrypt-docker-gen-update.sh >> /var/log/letsencrypt/docker-gen-update.log"

Using this configuration, when we run a new container with a VIRTUAL_HOST for which we don’t have certificates yet, the H2O configuration will be updated first to expose the .well-known/acme-challenge directory and a second time when executing letsencrypt-docker-gen-update.sh (the bash script we just wrote).

Copy all these files to the host using Ansible:

tasks/docker-gen-conf.yml

- file: path=/etc/docker-gen/templates state=directory mode=0755
- copy: src=h2o.tmpl dest=/etc/docker-gen/templates/h2o.tmpl
- copy: src=domains.tmpl dest=/etc/docker-gen/templates/domains.tmpl
- copy: src=docker-gen.cfg dest=/etc/docker-gen/docker-gen.cfg

We will now set up the Let’s encrypt (Certbot) container:

files/cli.ini

authenticator = webroot
email = hello@example.net
text = True
agree-tos = True
keep-until-expiring = True

tasks/letsencrypt.yml

---
- name: Create letsencrypt user
  user: name=letsencrypt state=present system=yes uid=1250 createhome=no

- name: Create /etc/letsencrypt directory
  file: path=/etc/letsencrypt state=directory mode=0700 owner=letsencrypt

- name: Copy cli.ini
  copy: src=cli.ini dest=/etc/letsencrypt/cli.ini owner=letsencrypt

- stat: path=/etc/letsencrypt/domains.conf
  register: letsencrypt_domains_conf_file

- name: Touch domains.conf if absent
  file: path=/etc/letsencrypt/domains.conf state=touch mode=600 owner=letsencrypt
  when: letsencrypt_domains_conf_file.stat.exists == False

- name: Create letsencrypt webroot directory
  file: path=/srv/www/letsencrypt state=directory mode=0755 owner=letsencrypt

- name: Create letsencrypt var/lib directory
  file: path=/var/lib/letsencrypt state=directory mode=700 owner=letsencrypt

- name: Copy update script
  copy: src=letsencrypt-docker-gen-update.sh dest=/usr/local/bin/letsencrypt-docker-gen-update.sh mode=700

- name: Ensure letsencrypt log directory exists
  file: path=/var/log/letsencrypt state=directory mode=700

- stat: path=/var/log/letsencrypt/docker-gen-update.log
  register: letsencrypt_log_file

- name: Ensure log file exists
  file: path=/var/log/letsencrypt/docker-gen-update.log state=touch mode=600
  when: letsencrypt_log_file.stat.exists == False

- name: Ensure we have a Let's encrypt container ready
  docker_container:
    name: letsencrypt-docker-gen-updater
    state: present
    image: "cedricbl/letsencrypt-webroot"
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - /srv/www/letsencrypt:/srv/www/letsencrypt
      - /var/lib/letsencrypt:/var/lib/letsencrypt

Here is what we do here:

We copy a cli.ini file for certbot
We create a normal user letsencrypt with a UID that match the container user UID
We create a file to log our bash script operations
We just set up the container, we don’t start it (state: present)

Run a H2O container:

files/index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Hello world!</title>
</head>
<body>
    <h1>It works!</h1>
</body>
</html>

tasks/h2o.yml

---
- name: Ensure /etc/h2o directory exists
  file: path=/etc/h2o state=directory mode=0700

- name: Check if h2o.conf exists
  stat: path=/etc/h2o/h2o.conf
  register: h2o_conf_file

- name: Create h2o.conf if not present yet
  shell: docker-gen /etc/docker-gen/templates/h2o.tmpl /etc/h2o/h2o.conf
  when: h2o_conf_file.stat.exists == False

- name: Ensure /etc/h2o/ssl directory exists
  file: path=/etc/h2o/ssl state=directory mode=0700

- stat: path=/etc/h2o/ssl/self-signed.crt
  register: h2o_self_signed_cert_file

- name: Create a self-signed SSL cert
  command: openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
          -keyout /etc/h2o/ssl/self-signed.key -out /etc/h2o/ssl/self-signed.crt \
          -subj "/C=US/ST=NY/L=NYC/O=IT/CN=mydefaulthost.tld"
  when: h2o_self_signed_cert_file.stat.exists == False

- name: Create www directory
  file: path=/srv/www state=directory mode=0755 owner=root

- name: Create default www directory
  file: path=/srv/www/default state=directory mode=0755

- name: Copy index.html file
  copy: src=index.html dest=/srv/www/default/index.html

- name: Ensure we have a h2o container running
  docker_container:
    name: h2o
    state: started
    image: "lkwg82/h2o-http2-server"
    ports:
     - "80:80"
     - "443:443"
    volumes:
      - /etc/h2o:/etc/h2o:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
      - /srv/www:/srv/www:ro
    command:
      - h2o
      - -m
      - master

Here is what we do here:

We make sure H2O config file exists
We generate a self-signed certificate for the default host
We copy a sample HTML file for the default host
We start the web server in master mode, so we can just send a SIGHUP signal to the container to reconfigure or upgrade the server

Configure docker-gen as a systemd service:

files/docker-gen.service

[Unit]
Description=A file generator that renders templates using Docker Container meta-data.
Documentation=https://github.com/jwilder/docker-gen
After=network.target docker.socket
Requires=docker.socket

[Service]
ExecStart=/usr/local/bin/docker-gen -config /etc/docker-gen/docker-gen.cfg

[Install]
WantedBy=multi-user.target

tasks/docker-gen-service.yml

---
- name: Copy service file
  copy: src=docker-gen.service dest=/lib/systemd/system/docker-gen.service

- name: Ensure service is started
  service: name=docker-gen state=started

And we’re done!

Check the result

At this point, we should have a docker-gen service running and listening for docker events.

We can run a sample web app with a VIRTUAL_HOST environment variable, let’s use the training/webapp for the example:

1 2	docker pull training/webapp docker run -d --name training_webapp -e "VIRTUAL_HOST=webapp.dev" training/webapp

If we check docker-gen service logs by executing systemctl status docker-gen, we should see something like this:

Generated '/etc/h2o/h2o.conf' from 2 containers
Running 'docker kill -s HUP h2o'
Generated '/etc/letsencrypt/domains.conf' from 2 containers
Running '/usr/local/bin/letsencrypt-docker-gen-update.sh >> /var/log/letsencrypt/docker-gen'

/etc/h2o/h2o.conf should include this for a few seconds:

hosts:
  "webapp.dev:80":
    listen:
      host: 0.0.0.0
      port: 80
    paths:
      /.well-known/acme-challenge:
        file.dir: /srv/www/letsencrypt/webapp.dev/.well-known/acme-challenge/

/etc/letsencrypt/domains.conf should look like this:

1	webapp.dev

docker ps output should mention that letsencrypt-docker-gen-updater container has been started a few seconds or minutes ago:

1	Exited (1) 5 minutes ago letsencrypt-docker-gen-updater

If Let’s encrypt certificate authorization procedure succeeded, it should appear in letsencrypt-docker-gen-updater container logs:

1
2
3

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
 [...]

And /etc/h2o/h2o.conf should now include something like this:

hosts:
  "webapp.dev:80":
    listen:
      host: 0.0.0.0
      port: 80
    paths:
      /.well-known/acme-challenge:
        file.dir: /srv/www/letsencrypt/webapp.dev/.well-known/acme-challenge/
      /:
        redirect: https://webapp.dev/
  "webapp.dev:443":
    listen:
      host: 0.0.0.0
      port: 443
      ssl:
        certificate-file: "/etc/letsencrypt/live/webapp.dev/fullchain.pem"
        key-file: "/etc/letsencrypt/live/webapp.dev/privkey.pem"
    paths:
      /:
        proxy.preserve-host: ON
        proxy.timeout.keepalive: 0
        proxy.reverse.url: "http://172.17.0.3:80"
        setenv:
          HTTP_PROXY: ""

We’re now serving the containerized app with SSL/TLS.

Final code

Check out the Ansible role on GitHub.