How to install a Certificate Authority (CA) root certificate for Docker registry on local Boot2Docker VM

Annoyingly often internal Docker registries are secured with certificates signed by company’s own PKI or enterprise IT does a MitM to replace all HTTPS certs.

Commonly, company’s root CA certificate is installed by IT on developers machines and servers, but not on VMs run by developers on their own machines. When using Docker with local VMs like boot2docker, do we need to install the company root CA certificate on the VM to avoid x509: certificate signed by unknown authority errors.

There are two ways to do it  – both are documented here.

Adding trusted CA root certificates to VM OS cert store

Let’s start with this option. Docker daemon respects OS cert store. To make the certificate survive machine restart it has  to be placed in /var/lib/boot2docker/certs directory on persistent partition . In Boot2Docker certificates (in .pem or .crt format) from that directory are automatically load at boot. See boot2docker/rootfs/etc/rc.d/install-ca-certs for details.

There’s also open issue in docker-machine to support  installing root CA certificates on machine creation and instruction how to build boot2docker ISO with custom CA certificate for private docker registry.

Addding trusted CA root certificate for specific registries

Docker allows to specify custom CA root for a
specific registry hostname. It can configured per registry by creating a directory under /etc/docker/certs.d using the same name as the registry’s hostname (including port number if any). All *.crt files  from this directory are added as CA roots, details are in moby/registry.go#newTLSConfig.

Another option to deal with insecure registries is enabling insecure communication with specified registries (no certificate verification and HTTP fallback). See insecure-registry for details.

Reklama

Publishing a port from a running Docker container

It’s not obvious, but it’s possible to publish an additional port from a running Docker container.

Note: Assuming container is attached to user-defined bridge network or default bridge network. Publishing ports doesn’t make sense with MacVLAN or IPVLAN since then every container has a uniquely routable IP address and can offer service on any port.

What can we do to make container port externally accessible?
One possibile solution is to exploit the fact that communication between containers inside the same bridge network is switched at layer 2 and they can communiate with each other without any restrictions (unless ICC is disabled, that is). That means when can run another container with socat inside target network with published port and setup forwarding

docker run --rm -it \
	--net=[TARGET_NETWORK]
	-p [PORT]:[PORT] \
	bobrik/socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]

What else can we do do? It’s not immediately obvious, but each bridge network is reachable from the host. We can verify with `ip route show` that in host’s routing table exists routing table entry for each bridge network

$ ip route show
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-b4247d00e80c proto kernel scope link src 172.18.0.1
172.19.0.0/16 dev br-acd7d3307ea3 proto kernel scope link src 172.19.0.1

That means we can as well setup socat relay in host network namespace

docker run --rm -it \
	--net=host \
	-p [PORT]:[PORT] \
	bobrik/socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]

or just run socat directly on host

socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]

But we can still do better! We can replace socat relay with iptables DNAT. For inbound traffic, we’ll need to create a custom rule that uses Network Address Translation (NAT) to map a port on the host to the service port in the container. We can do that with a rule like this:

# must be added into DOCKER chain
 iptables -t nat -A DOCKER -p tcp -m tcp \
    --dport [PORT] -j DNAT \
    --to-destination [CONTAINER_IP]:[CONTAINER_PORT]

This is pretty much how Docker publishes ports from containers using the bridged network when option -publish|-p is used with docker run. The only difference is that DNAT rule created by Docker is restricted not to affect traffic originating from bridge

 iptables -t nat -A DOCKER ! -i [BRIDGE_NAME] -p tcp -m tcp \
    --dport [PORT] -j DNAT \
    --to-destination [CONTAINER_IP]:[CONTAINER_PORT]

The DOCKER chain is a custom chain defined at the FORWARD chain. When a packet hits any interface and is bound to the one of bridge interfaces, it is sent to the custom DOCKER chain. Now the DOCKER chain will take all incoming packets, except ones coming from bridge (say docker0 for default network), and send them to a container IP (usually 172.x.x.x) and port.