Fixing the 'Unknown blob' error with Apache and Private docker registries

I recently took on the task of setting up a private docker container registry. It's useful if you would like to host private images and are comfortable running your own infrastructure.

A private container registry is relatively simple to setup using the instructions on the official docker setup guide. The steps at the time of writing generally are generally,

1. Pull the registry:2 image
2. Setup basic authentication
3. Start a container with the image

You can optionally configure ssl with the image, however if you are already sitting behind a reverse proxy, it should work just fine with your exiting setup.

While configuring my instance running behind an Apache2 web server, I found that attempting to push images to the registry I kept hitting what seems to be a common recurring error.

$ docker push <redacted>/image

The push refers to repository [<redacted>/image]
8846eeb38ced: Pushing [==================================================>]  160.8kB
a7be517662ec: Pushing [===>                                               ]  1.022MB/15.45MB
18cec866437b: Pushing [==================================================>]  2.048kB
9d1f139ac886: Pushing [==================>                                ]  2.217MB/5.923MB
029d8a704a27: Pushing [=>                                                 ]  1.498MB/49.17MB
00023a62e045: Waiting
73046094a9b8: Waiting
unknown blob

The push begins to run and promptly times out with an unknown blob message.

After looking through a couple of issues, specifically this and this, it seems to be an issue with reverse proxies incorrectly resolving the scheme. Marking the forwaded request headers as https using the mod_headers feature seems to have fixed the issue for me.

You will need to enable this feature in Apache,

$ a2enmod headers

and finally add the configuration to the appropriate virtual host.

<IfModule mod_ssl.c>
<VirtualHost *:443>
  # Mark the header as https
  Header add X-Forwarded-Proto "https"
  RequestHeader add X-Forwarded-Proto "https"
  ...
</VirtualHost>
</IfModule>