Fixing the 'Unknown blob' error with Apache and Private docker registries
I recently took on the task of setting up a private docker container registry. It's useful if you would like to host private images and are comfortable running your own infrastructure.
A private container registry is relatively simple to setup using the instructions on the official docker setup guide. The steps at the time of writing generally are generally,
- Pull the
- Setup basic authentication
- Start a container with the image
You can optionally configure
ssl with the image, however if you are already sitting behind a reverse proxy, it should work just fine with your exiting setup.
While configuring my instance running behind an Apache2 web server, I found that attempting to push images to the registry I kept hitting what seems to be a common recurring error.
$ docker push <redacted>/image The push refers to repository [<redacted>/image] 8846eeb38ced: Pushing [==================================================>] 160.8kB a7be517662ec: Pushing [===> ] 1.022MB/15.45MB 18cec866437b: Pushing [==================================================>] 2.048kB 9d1f139ac886: Pushing [==================> ] 2.217MB/5.923MB 029d8a704a27: Pushing [=> ] 1.498MB/49.17MB 00023a62e045: Waiting 73046094a9b8: Waiting unknown blob
The push begins to run and promptly times out with an
unknown blob message.
After looking through a couple of issues, specifically this and this, it seems to be an issue with reverse proxies incorrectly resolving the scheme. Marking the forwaded request headers as
https using the
mod_headers feature seems to have fixed the issue for me.
You will need to enable this feature in Apache,
$ a2enmod headers
and finally add the configuration to the appropriate virtual host.
<IfModule mod_ssl.c> <VirtualHost *:443> # Mark the header as https Header add X-Forwarded-Proto "https" RequestHeader add X-Forwarded-Proto "https" ... </VirtualHost> </IfModule>