Set up an automated HTTP/2 reverse proxy for your Docker containers using H2O, Docker-gen, Let's Encrypt and Ansible on a single host

2016-08-21

TL;DR: Check out the Ansible role on GitHub.

What?

It took me some time to understand why I should care about web application containerization for other reasons than “this is the future!“.

But I actually found that Docker APIs approach is quite interesting when it comes to shipping and scaling a web application and I particularly like the analogy between containers and cartridges that Shopify drew on their engineering blog.

There are multiple obvious reasons why you may want to achieve zero-downtime deployment using Docker. One way to achieve this is to use a reverse proxy in front of containers (if your application is stateless).
Jason Wilder came up with Docker-gen and a ready-to-use solution using nginx.

For a personal small-scale project, I decided to build a similar solution that makes Docker-gen, H2O and Let’s encrypt work all together using Ansible.

Why H2O?

In one acronym : HTTP/2.
HTTP/2 brings numerous benefits such as multiplexing and server push.

H2O is less full-featured and battle-tested than Nginx and Apache but unlike Nginx or Apache, H2O was built from ground up to do HTTP/2 and it takes full advantage of HTTP/2 features. It also provides a reverse proxy implementation module (using a HTTP/1 client) and I think it it worth giving it a try.
Thanks to projects such as Let’s encrypt, it’s now possible to obtain TLS/SSL certificates for free and take advantage of these features.

For more information about H2O, I recommend these slides.

Okay, let’s think about it!

When we start a new web app container on the docker host, we want:

  • TLS/SSL certificates to be created automatically using a Let’s encrypt client
  • H2O configuration to be updated automatically.

If the certificates do not exist yet, it would give us a process like this:

  1. Update H2O configuration with a new virtual host and serve .well-known/acme-challenge through HTTP on port 80
  2. Reload H2O
  3. Run Let’s encrypt client to create certificates using a configuration file generated by docker-gen and the webroot method
  4. Update H2O configuration again to use the new certificates
  5. Reload H2O

If the certificates already exists, we can skip the 3 first steps.

It would be quite convenient if we could run H2O and Let’s encrypt client in containers instead of installing all dependencies on the docker host.
We could run H2O in container by using this image.
For running the Let’s encrypt client, I built a docker-gen friendly certbot (official Let’s encrypt client) image that runs with a non-root user.

We could actually bundle all services (Docker-gen, H2O and Certbot) in a single container and use forego to start them but I would prefer a separate containers approach for different reasons:

  • I want to prevent having the docker socket (required by docker-gen) bound to a publicly exposed container service like H2O
  • It allows using separate existing and well written docker images (Unix philosophy: Do One Thing and Do It Well)

Since www data, certificates and H2O configuration are resources that would need to be shared between our services, we will have to bind some volumes from the host and set up different levels of access (H2O need read-only on www data and certs, etc.).

We would basically end up with something like this:

Okay, let’s write this!

First, let’s create a task to install docker-gen on the host:

tasks/docker-gen-install.yml
1
2
3
4
5
6
7
---
- get_url:
url: https://github.com/jwilder/docker-gen/releases/download/0.7.3/docker-gen-linux-amd64-0.7.3.tar.gz
dest: ~/docker-gen.tar.gz
checksum: "sha1:9bd460f66e2e26323ea439c11956a145bb305c53"
- shell: tar -xf ~/docker-gen.tar.gz -C /usr/local/bin
- file: path=~/docker-gen.tar.gz state=absent

Then let’s build a docker-gen template (using the Go text/template language) for H2O configuration:

files/h2o.tmpl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
# Base conf
user: nobody
http2-reprioritize-blocking-assets: ON
send-server-name: OFF
file.dirlisting: OFF
access-log: /dev/stdout
error-log: /dev/stderr
compress: ON

# Listen to both HTTP and HTTPS on start
hosts:
default:
listen:
host: 0.0.0.0
port: 80
listen:
host: 0.0.0.0
port: 443
ssl:
key-file: /etc/h2o/ssl/self-signed.key
certificate-file: /etc/h2o/ssl/self-signed.crt
paths:
/:
file.dir: /srv/www/default

{{ define "proxy" }}
/:
proxy.preserve-host: ON
proxy.timeout.keepalive: 0
{{ if .Address }}
{{ if (and .Container.Node.ID .Address.HostPort) }}
proxy.reverse.url: "http://{{ .Container.Node.Address.IP }}:{{ .Address.HostPort }}"
{{ else }}
proxy.reverse.url: "http://{{ .Address.IP }}:{{ .Address.Port }}"
{{ end }}
{{ else }}
proxy.reverse.url: "http://{{ .Container.IP }} down"
{{ end }}
setenv:
HTTP_PROXY: ""
{{ end }}

{{ define "host" }}
hosts:
"{{ .Host }}:80":
listen:
host: 0.0.0.0
port: 80
paths:
/.well-known/acme-challenge:
file.dir: /srv/www/letsencrypt/{{ .Host }}/.well-known/acme-challenge/
{{ if (exists (printf "/etc/letsencrypt/live/%s/privkey.pem" .Host)) }}
"/":
redirect: https://{{ .Host }}/
"{{ .Host }}:443":
listen:
host: 0.0.0.0
port: 443
ssl:
certificate-file: "/etc/letsencrypt/live/{{ .Host }}/fullchain.pem"
key-file: "/etc/letsencrypt/live/{{ .Host }}/privkey.pem"
paths:
{{ template "proxy" .Proxy }}
{{ end }}
{{ end }}

{{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
{{ if (ne $host "") }}
{{ range $container := $containers }}
{{ $addrLen := len $container.Addresses }}
{{ if eq $addrLen 1 }}
{{ $address := index $container.Addresses 0 }}
{{ $proxy := (dict "Container" $container "Address" $address) }}
{{ template "host" (dict "Host" $host "Proxy" $proxy) }}
{{ else }}
{{ $port := coalesce $container.Env.VIRTUAL_PORT "80" }}
{{ $address := where $container.Addresses "Port" $port | first }}
{{ $proxy := (dict "Container" $container "Address" $address) }}
{{ template "host" (dict "Host" $host "Proxy" $proxy) }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}

Here is what we do here:

  • We listen to both HTTP and HTTPS on start using a default virtual host
  • We create one virtual host per container using the VIRTUAL_HOST and VIRTUAL_PORT environment variables
  • For each virtual host, we listen to port 80 for serving the .well-known/acme-challenge for Let’s encrypt and to redirect other requests to HTTPS
  • For HTTPS, we use certificates generated by Let’s encrypt only if they already exists

Now, let’s write a template to generate a very simple configuration file with a list of domains to validate/update with Certbot:

files/domains.tmpl
1
2
3
4
5
{{ range $host, $container := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
{{ if (ne $host "") }}
{{ $host }}
{{ end }}
{{ end }}

Let’s write a bash script that will execute the Certbot container letsencrypt-docker-gen-updater (we will create it later), update H2O configuration and gracefully reload H2O.
This script could also be used for periodically update the certificates.

files/letsencrypt-docker-gen-update.sh
1
2
3
4
5
6
7
8
readarray DOMAINS < /etc/letsencrypt/domains.conf
DOMAINS_COUNT=${#DOMAINS[@]}
if [ $DOMAINS_COUNT -gt 0 ]
then
docker start -a letsencrypt-docker-gen-updater
docker-gen -onlyexposed /etc/docker-gen/templates/h2o.tmpl /etc/h2o/h2o.conf
docker kill -s HUP h2o
fi

We can now write the docker-gen main config file:

files/docker-gen.cfg
1
2
3
4
5
6
7
8
9
10
11
12
13
[[config]]
template = "/etc/docker-gen/templates/h2o.tmpl"
dest = "/etc/h2o/h2o.conf"
onlyexposed = true
watch = true
notifycmd = "docker kill -s HUP h2o"

[[config]]
template = "/etc/docker-gen/templates/domains.tmpl"
dest = "/etc/letsencrypt/domains.conf"
onlyexposed = true
watch = true
notifycmd = "/usr/local/bin/letsencrypt-docker-gen-update.sh >> /var/log/letsencrypt/docker-gen-update.log"

Using this configuration, when we run a new container with a VIRTUAL_HOST for which we don’t have certificates yet, the H2O configuration will be updated first to expose the .well-known/acme-challenge directory and a second time when executing letsencrypt-docker-gen-update.sh (the bash script we just wrote).

Copy all these files to the host using Ansible:

tasks/docker-gen-conf.yml
1
2
3
4
- file: path=/etc/docker-gen/templates state=directory mode=0755
- copy: src=h2o.tmpl dest=/etc/docker-gen/templates/h2o.tmpl
- copy: src=domains.tmpl dest=/etc/docker-gen/templates/domains.tmpl
- copy: src=docker-gen.cfg dest=/etc/docker-gen/docker-gen.cfg

We will now set up the Let’s encrypt (Certbot) container:

files/cli.ini
1
2
3
4
5
authenticator = webroot
email = hello@example.net
text = True
agree-tos = True
keep-until-expiring = True
tasks/letsencrypt.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
---
- name: Create letsencrypt user
user: name=letsencrypt state=present system=yes uid=1250 createhome=no

- name: Create /etc/letsencrypt directory
file: path=/etc/letsencrypt state=directory mode=0700 owner=letsencrypt

- name: Copy cli.ini
copy: src=cli.ini dest=/etc/letsencrypt/cli.ini owner=letsencrypt

- stat: path=/etc/letsencrypt/domains.conf
register: letsencrypt_domains_conf_file

- name: Touch domains.conf if absent
file: path=/etc/letsencrypt/domains.conf state=touch mode=600 owner=letsencrypt
when: letsencrypt_domains_conf_file.stat.exists == False

- name: Create letsencrypt webroot directory
file: path=/srv/www/letsencrypt state=directory mode=0755 owner=letsencrypt

- name: Create letsencrypt var/lib directory
file: path=/var/lib/letsencrypt state=directory mode=700 owner=letsencrypt

- name: Copy update script
copy: src=letsencrypt-docker-gen-update.sh dest=/usr/local/bin/letsencrypt-docker-gen-update.sh mode=700

- name: Ensure letsencrypt log directory exists
file: path=/var/log/letsencrypt state=directory mode=700

- stat: path=/var/log/letsencrypt/docker-gen-update.log
register: letsencrypt_log_file

- name: Ensure log file exists
file: path=/var/log/letsencrypt/docker-gen-update.log state=touch mode=600
when: letsencrypt_log_file.stat.exists == False

- name: Ensure we have a Let's encrypt container ready
docker_container:
name: letsencrypt-docker-gen-updater
state: present
image: "cedricbl/letsencrypt-webroot"
volumes:
- /etc/letsencrypt:/etc/letsencrypt
- /srv/www/letsencrypt:/srv/www/letsencrypt
- /var/lib/letsencrypt:/var/lib/letsencrypt

Here is what we do here:

  • We copy a cli.ini file for certbot
  • We create a normal user letsencrypt with a UID that match the container user UID
  • We create a file to log our bash script operations
  • We just set up the container, we don’t start it (state: present)

Run a H2O container:

files/index.html
1
2
3
4
5
6
7
8
9
10
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Hello world!</title>
</head>
<body>
<h1>It works!</h1>
</body>
</html>
tasks/h2o.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
---
- name: Ensure /etc/h2o directory exists
file: path=/etc/h2o state=directory mode=0700

- name: Check if h2o.conf exists
stat: path=/etc/h2o/h2o.conf
register: h2o_conf_file

- name: Create h2o.conf if not present yet
shell: docker-gen /etc/docker-gen/templates/h2o.tmpl /etc/h2o/h2o.conf
when: h2o_conf_file.stat.exists == False

- name: Ensure /etc/h2o/ssl directory exists
file: path=/etc/h2o/ssl state=directory mode=0700

- stat: path=/etc/h2o/ssl/self-signed.crt
register: h2o_self_signed_cert_file

- name: Create a self-signed SSL cert
command: openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /etc/h2o/ssl/self-signed.key -out /etc/h2o/ssl/self-signed.crt \
-subj "/C=US/ST=NY/L=NYC/O=IT/CN=mydefaulthost.tld"
when: h2o_self_signed_cert_file.stat.exists == False

- name: Create www directory
file: path=/srv/www state=directory mode=0755 owner=root

- name: Create default www directory
file: path=/srv/www/default state=directory mode=0755

- name: Copy index.html file
copy: src=index.html dest=/srv/www/default/index.html

- name: Ensure we have a h2o container running
docker_container:
name: h2o
state: started
image: "lkwg82/h2o-http2-server"
ports:
- "80:80"
- "443:443"
volumes:
- /etc/h2o:/etc/h2o:ro
- /etc/letsencrypt:/etc/letsencrypt:ro
- /srv/www:/srv/www:ro
command:
- h2o
- -m
- master

Here is what we do here:

  • We make sure H2O config file exists
  • We generate a self-signed certificate for the default host
  • We copy a sample HTML file for the default host
  • We start the web server in master mode, so we can just send a SIGHUP signal to the container to reconfigure or upgrade the server

Configure docker-gen as a systemd service:

files/docker-gen.service
1
2
3
4
5
6
7
8
9
10
11
[Unit]
Description=A file generator that renders templates using Docker Container meta-data.
Documentation=https://github.com/jwilder/docker-gen
After=network.target docker.socket
Requires=docker.socket

[Service]
ExecStart=/usr/local/bin/docker-gen -config /etc/docker-gen/docker-gen.cfg

[Install]
WantedBy=multi-user.target
tasks/docker-gen-service.yml
1
2
3
4
5
6
---
- name: Copy service file
copy: src=docker-gen.service dest=/lib/systemd/system/docker-gen.service

- name: Ensure service is started
service: name=docker-gen state=started

And we’re done!

Check the result

At this point, we should have a docker-gen service running and listening for docker events.

We can run a sample web app with a VIRTUAL_HOST environment variable, let’s use the training/webapp for the example:

1
2
docker pull training/webapp
docker run -d --name training_webapp -e "VIRTUAL_HOST=webapp.dev" training/webapp

If we check docker-gen service logs by executing systemctl status docker-gen, we should see something like this:

1
2
3
4
Generated '/etc/h2o/h2o.conf' from 2 containers
Running 'docker kill -s HUP h2o'
Generated '/etc/letsencrypt/domains.conf' from 2 containers
Running '/usr/local/bin/letsencrypt-docker-gen-update.sh >> /var/log/letsencrypt/docker-gen'

/etc/h2o/h2o.conf should include this for a few seconds:

1
2
3
4
5
6
7
8
hosts:
"webapp.dev:80":
listen:
host: 0.0.0.0
port: 80
paths:
/.well-known/acme-challenge:
file.dir: /srv/www/letsencrypt/webapp.dev/.well-known/acme-challenge/

/etc/letsencrypt/domains.conf should look like this:

1
webapp.dev

docker ps output should mention that letsencrypt-docker-gen-updater container has been started a few seconds or minutes ago:

1
Exited (1) 5 minutes ago  letsencrypt-docker-gen-updater

If Let’s encrypt certificate authorization procedure succeeded, it should appear in letsencrypt-docker-gen-updater container logs:

1
2
3
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at
[...]

And /etc/h2o/h2o.conf should now include something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
hosts:
"webapp.dev:80":
listen:
host: 0.0.0.0
port: 80
paths:
/.well-known/acme-challenge:
file.dir: /srv/www/letsencrypt/webapp.dev/.well-known/acme-challenge/
/:
redirect: https://webapp.dev/
"webapp.dev:443":
listen:
host: 0.0.0.0
port: 443
ssl:
certificate-file: "/etc/letsencrypt/live/webapp.dev/fullchain.pem"
key-file: "/etc/letsencrypt/live/webapp.dev/privkey.pem"
paths:
/:
proxy.preserve-host: ON
proxy.timeout.keepalive: 0
proxy.reverse.url: "http://172.17.0.3:80"
setenv:
HTTP_PROXY: ""

We’re now serving the containerized app with SSL/TLS.

Final code

Check out the Ansible role on GitHub.