*******    ***    ******   *******  ******   **  **    ***    *******
  ***     ** **   **   **  **       **   **  ** **    ** **     ***  
  ***    **   **  ******   *******  **   **  ****    **   **    ***  
  ***    *******  **  **   **       **   **  ** **   *******    ***  
***      **   **  **   **  *******  *****    **  **  **   **    *** 

The Problem

When using ProxyPass configuration on a apache webserver, as is neccessitated when running multiple webservers behind a NAT router on a single IP, as is such hosting from home, the two solutions are either to use virtualhosts or proxypass. I use virtualhosts where I can, but, some things are a lot easier if they’re running on their own servers. In my case, these being NextCloud and Gitlab.

Proxypass basically behaves as a proxy, listening on one server, and using those HTTP requests to ask another server for the content, then presenting that content as it’s own. It’s the perfect solution when configured withing a virtualhost to allow different hostnames to request different contnet from different servers on the LAN.

The problem comes, though, with getting SSL for these sites. Proxypass, in it’s own way, breaks SSL in it’s own way. Essentially, in order to act as a proxy, it needs to request the content from the origin server, but needs to decrypt the SSL in order to do so, with no way of resigning the certificate for transmission to the requesting client. So, because of this, the client never sees the originating server’s SSL. It can only see the Proxy’s own SSL as it re-signs the content before sending it over the wire.

Requesting a “Let’s Encrypt” certificate, though, requires validating the origin server via it’s hostname or domain. If we were to request a certificate on the origin server, it aught to work, as the validation would go through, but this SSL would be installed to the wrong spot, as we’d need the proxy server to have the SSL in order to re-sign the traffic to traverse the internet instead of just the LAN. This is because Let’s Encrypt validates the webserver by checking for a specific file, .well-known/acme-challenge/<token> from the hostname you’re requesting a cert for. With proxypass set up, the proxy server happily forwards this to the origin server in the LAN, and can’t validate itself because of that.

There’s a couple ways this could be solved. One option, and maybe the most legitimate, is to set up a script to SCP the let’s encrypt certificate over to the proxy server once it’s recieved by the certbot on the origin server. But given my configuration, this does not work totally well, since I am using Virtualmin, figuring out the API to tell Virtualmin to restart Apache to load the cert is… beyond the scope of my motivation.

I used to handle this by signing certs on my own certificate authority, and putting them in place myself, and that works wonders for me, myself, and I, but for friends and family who might not want or might not be able to install a custom certificate authority to make things work nicely, It would be better if the proxy server could sign with a cert from a globally trusted certificate authority.

As mentioned above, using Let’s Encrypt poses problems in this proxy configuration. But, we’ve got a solution.

The solution

The solution, simply, is to run the Let’s Encrypt certbot from the proxy machine itself, and let the origin server on the LAN sign any other way it likes.

As mentioned, with a simple proxypass config, the proxy will send the request to the origin server, but put the validation file on the proxy server. Because the request get’s sent off, Let’s Encrypt never gets to see the validation file and the process fails.

An example of that sort of config would be like such:

<VirtualHost *.443>
    servername: gitlab.jaredkat.net
    DocumentRoot /home/jaredkat/domains/gitlab.jaredkat.net/public_html
    [...]
    ProxyPass / http://10.13.37.73/
    ProxyPassReverse / http://10.13.37.73/
    ProxyPreserveHost On
</VirtualHost>

Well, one option to solve this, is to use a directive to disable proxypass for one particular directory.

<VirtualHost *.443>
    servername: gitlab.jaredkat.net
    DocumentRoot /home/jaredkat/domains/gitlab.jaredkat.net/public_html
    [...]
    ProxyPass /.well-known/acme-challenge !
    ProxyPass / http://10.13.37.73/
    ProxyPassReverse / http://10.13.37.73/
    ProxyPreserveHost On
</VirtualHost>

To explain the change, we simply tell ProxyPass, via the exclamation mark to indicate a NOT instruction, to route requests to the directory /.well-known/acme-challenge/ to the actual file on the server and ignore the global proxypass rule that’s otherwise in place.

This also allows most of any other .well-known file to be hosted from the origin server on the LAN, such as Nextcloud’s use of them for webdav configurations.

With this configuration change, I was able to request a Let’s Encrypt certificate through the virtualmin interface with ease, everything worked out fine, and now I can direct friends and family to my self-hosted services without having to guide them through “Accepting the untrusted certificate” each time or going through the hassle of installing my custom CA as a trusted root authority.

Easy enough, right?