My way to auto update Let's Encrypt certs without downtime - Sun, Dec 27, 2015
It’s been a while since my last post here. This is my first post with the new platform - Hexo. It’s faster and simpler than Octopress and it’s not Ruby but nevermind…
The whole concept with the Certification Authorities is completely broken but we don’t have something better which is working. A world with fully encrypted web is a really a good idea since the whole internet traffic is monitored by governments and other private organizations. Let’s encrypt is an attempt for that. It’s a colaborative project between Linux foundation, EFF and some other organizations.
They are providing free (completely free!) certificates with 3 months of validity. After that time, the certificates can be updated again.
Signing and delivery of the certificates
Let’s encrypt is using the ACME (Automated Certificate Management Environment) protocol which defines automatically obtaining of certificates. More information about the protocol can be found at the Let’s Encrypt - How it works page.
So, let’s start the technical part
In this setup I’ll use Ubuntu 14.04 with HAProxy for load balancing and managing the traffic for all of the domains.
Install HAProxy:
HAProxy will directly deliver bind the HTTPS content but we need SNI checks for the Acme client. So, it’s a bit bizzare. We will have a loop inside HAPproxy. A TCP proxy frontend which is proxying a backend from localhost to the HTTPS proxy frontend.
TCP frontend -> HTTPS backend/ACME client backend -> HTTPS frontend -> Application Servers HTTP backends
sudo apt-get install -y software-properties-common
sudo apt-add-repository ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get install -y haproxy
Configure HAProxy:
Let’s start with the default global config (/etc/haproxy/haproxy.cfg):
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
tune.bufsize 131072
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /srv/certs
crt-base /srv/certs
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:!RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
# tunning
maxconn 16384
Then, append the default configuration for the http/s frontends (/etc/haproxy/haproxy.cfg):
defaults http
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
timeout connect 5s
timeout client 310s
timeout server 310s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
Setup the HTTP frontend. It will only refer the http requests to https:
frontend www-http
bind 0.0.0.0:80
reqadd X-Forwarded-Proto:\ http
option forwardfor
#ACLs
acl example_sites hdr(host) -i example.com www.example.com
#Redirects
redirect prefix https://cloud.grandcity-property.com if example_sites
Now, create the HTTPS frontend. The port must be different than 443.
frontend www-https
bind 0.0.0.0:4443 ssl crt example.com.pem crt www.example.com.pem crt ./ no-sslv3
reqadd X-Forwarded-Proto:\ https
option forwardfor
#ACLs
acl example_sites hdr(host) -i example.com www.example.com
use_backend examplecom if example_sites
Create the backend for example.com:
backend examplecom
timeout server 30m
balance leastconn
option httpclose
option forwardfor
cookie JSESSIONID prefix
server node1 192.168.0.10:80 cookie A check
Now, the TCP Proxy part. TCP defaults:
#### TCP Section
defaults tcp
log global
mode tcp
option tcplog
timeout connect 10s
timeout client 600s
timeout server 600s
The TCP frontend (listening on port 443):
frontend www-https-tcp
log global
mode tcp
option tcplog
bind 0.0.0.0:443
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
# Matching all SNI names with *.acme.invalid
acl app_letsencrypt req.ssl_sni -m end .acme.invalid
use_backend letsencrypt if app_letsencrypt
# sending everything that doesn't match *.acme.invalid to the HTTPS backend
default_backend bk_frontend_https_loop
And the backends for www-https-tcp:
backend bk_frontend_https_loop
log global
mode tcp
option tcplog
server localserver 0.0.0.0:4443
backend letsencrypt
log global
mode tcp
option tcplog
server letsencrypt 0.0.0.0:63443 #By default, Let's encrypt works on 443.
The whole haproxy.cfg is in this gist.
Now, install the official Let’s encrypt client:
Install git:
sudo apt-get install -y git
Download the client:
git clone https://github.com/letsencrypt/letsencrypt /root/letsencrypt
And create the certificate for example.com and wwww.example.com:
/root/letsencrypt/letsencrypt-auto --email admin@example.com -d example.com -d www.example.com --authenticator standalone --tls-sni-01-port 63443 --text auth --http-01-port 8099
Now, download ssl-cert-check from Prefetch.net. This is very useful script because it calculates the time difference between the current time and the certification expiration date:
sudo wget -O http://prefetch.net/code/ssl-cert-check /usr/local/bin/ssl-cert-check
sudo chmod +x /usr/local/bin/ssl-cert-check
Put this script to /etc/cron.daily/updatessl and make it executable :
#!/bin/bash
########################################################################
## Script for updating Let's encrypt certs
## http://petrovs.info/2015/12/27/My-way-to-auto-update-Lets-Encrypt/
## (C) 2015 Blagovest Petrov (blagovest <at> petrovs dot info)
## Released under the GPL
########################################################################
cd /etc/letsencrypt/live
for DIRS in *; do
if ssl-cert-check -c $DIRS/fullchain.pem | grep -q "Valid"; then
echo On `date`, $DIRS is Valid. >> /var/log/letsencrypt-update.log
#cat $DIRS/fullchain.pem > /srv/certs/$DIRS.pem
#cat $DIRS/privkey.pem >> /srv/certs/$DIRS.pem
else
echo On `date`, $DIRS is NOT Valid. Generating a new certificate from Let\'s encrypt... >> /var/log/letsencrypt-update.log
/root/letsencrypt/letsencrypt-auto --email it@veriosoft.com -d $DIRS --authenticator standalone --tls-sni-01-port 63443 --renew-by-default --text auth --http-01-port 8099 >> /var/log/letsencrypt-update.log
echo Generating fullchain token for $DIRS to be used for HAProxy... >> /var/log/letsencrypt-update.log
cat $DIRS/fullchain.pem > /srv/certs/$DIRS.pem
cat $DIRS/privkey.pem >> /srv/certs/$DIRS.pem
echo Reloading the HAProxy configuration... >> /var/log/letsencrypt-update.log
service haproxy reload
echo Done :\) >> /var/log/letsencrypt-update.log
fi
done
sudo wget -O https://gist.githubusercontent.com/eniac111/c7146b3e59c7eff27fbe/raw/506643d797811ec99ce6e32d8f9e23ea3a9200d4/updatessl.sh /etc/cron.daily/updatessl
sudo chmod +x /etc/cron.daily/updatessl
The last thing is to create logrotate config for the update log:
Install logrotate it it’s not installed:
sudo apt-get update && sudo apt-get install -y logrotate
Put this to /etc/logrotate/letsencrypt-update:
/var/log/letsencrypt-update.log {
monthly
rotate 12
compress
delaycompress
missingok
notifempty
create 644 root root
}
Correct me if there is a better way. The double loop inside HAProxy is really bizzare.
Comments