Published on dev.to: https://dev.to/jarekprzygodzki/a-curious-case-of-slow-docker-image-builds-2o7k
Generating JVM memory dumps from JRE
Published on dev.to: Generating JVM memory dumps from JRE
How to install a Certificate Authority (CA) root certificate for Docker registry on local Boot2Docker VM
Annoyingly often internal Docker registries are secured with certificates signed by company’s own PKI or enterprise IT does a MitM to replace all HTTPS certs.
Commonly, company’s root CA certificate is installed by IT on developers machines and servers, but not on VMs run by developers on their own machines. When using Docker with local VMs like boot2docker, do we need to install the company root CA certificate on the VM to avoid x509: certificate signed by unknown authority errors.
There are two ways to do it – both are documented here.
Adding trusted CA root certificates to VM OS cert store
Let’s start with this option. Docker daemon respects OS cert store. To make the certificate survive machine restart it has to be placed in /var/lib/boot2docker/certs directory on persistent partition . In Boot2Docker certificates (in .pem or .crt format) from that directory are automatically load at boot. See boot2docker/rootfs/etc/rc.d/install-ca-certs for details.
There’s also open issue in docker-machine to support installing root CA certificates on machine creation and instruction how to build boot2docker ISO with custom CA certificate for private docker registry.
Addding trusted CA root certificate for specific registries
Docker allows to specify custom CA root for a
specific registry hostname. It can configured per registry by creating a directory under /etc/docker/certs.d using the same name as the registry’s hostname (including port number if any). All *.crt files from this directory are added as CA roots, details are in moby/registry.go#newTLSConfig.
Another option to deal with insecure registries is enabling insecure communication with specified registries (no certificate verification and HTTP fallback). See insecure-registry for details.
Publishing a port from a running Docker container
It’s not obvious, but it’s possible to publish an additional port from a running Docker container.
Note: Assuming container is attached to user-defined bridge network or default bridge network. Publishing ports doesn’t make sense with MacVLAN or IPVLAN since then every container has a uniquely routable IP address and can offer service on any port.
What can we do to make container port externally accessible?
One possibile solution is to exploit the fact that communication between containers inside the same bridge network is switched at layer 2 and they can communiate with each other without any restrictions (unless ICC is disabled, that is). That means when can run another container with socat inside target network with published port and setup forwarding
docker run --rm -it \ --net=[TARGET_NETWORK] -p [PORT]:[PORT] \ bobrik/socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]
What else can we do do? It’s not immediately obvious, but each bridge network is reachable from the host. We can verify with `ip route show` that in host’s routing table exists routing table entry for each bridge network
$ ip route show 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 dev br-b4247d00e80c proto kernel scope link src 172.18.0.1 172.19.0.0/16 dev br-acd7d3307ea3 proto kernel scope link src 172.19.0.1
That means we can as well setup socat relay in host network namespace
docker run --rm -it \ --net=host \ -p [PORT]:[PORT] \ bobrik/socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]
or just run socat directly on host
socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]
But we can still do better! We can replace socat relay with iptables DNAT. For inbound traffic, we’ll need to create a custom rule that uses Network Address Translation (NAT) to map a port on the host to the service port in the container. We can do that with a rule like this:
# must be added into DOCKER chain
iptables -t nat -A DOCKER -p tcp -m tcp \
--dport [PORT] -j DNAT \
--to-destination [CONTAINER_IP]:[CONTAINER_PORT]
This is pretty much how Docker publishes ports from containers using the bridged network when option -publish|-p is used with docker run. The only difference is that DNAT rule created by Docker is restricted not to affect traffic originating from bridge
iptables -t nat -A DOCKER ! -i [BRIDGE_NAME] -p tcp -m tcp \
--dport [PORT] -j DNAT \
--to-destination [CONTAINER_IP]:[CONTAINER_PORT]
The DOCKER chain is a custom chain defined at the FORWARD chain. When a packet hits any interface and is bound to the one of bridge interfaces, it is sent to the custom DOCKER chain. Now the DOCKER chain will take all incoming packets, except ones coming from bridge (say docker0 for default network), and send them to a container IP (usually 172.x.x.x) and port.
Monitorowanie i przechwytywanie ruchu sieciowego kontenerów Dockera
Ten wpis to krótki tutorial jak przechwycić ruch sieciowych z wybranych kontenerów Dockera, np. usług uruchomionych w ramach klastra Docker Swarm.
Na początek minimum teorii. Docker używa przestrzeni nazw w celu zapewnienia izolacji procesów. Przestrzenie nazw (których od jądra 4.10 jest 7 rodzajów) są cechą jądra systemu Linux, która pozwala na izolację i wirtualizację zasobów systemowych. Funkcja przestrzeni nazw jest taka sama dla każdego typu: każdy proces jest powiązany z przestrzenią nazw danego rodzaju i może wykorzystywać zasoby powiązane z tą przestrzenią, oraz, w stosownych przypadkach, przestrzeniami nazw potomków. W kontekście ruchu sieciowego interesuje nas przestrzeń nazw dla sieci (network namespace).
Sieciowe przestrzenie nazw wirtualizują stos sieciowy – umożliwiają odizolowanie całego podsystemu sieciowego dla grupy procesów. Każda przestrzeń nazw posiada prywatny zestaw adresów IP, własną tabelę routingu, listę gniazd, tabelę śledzenia połączeń, zaporę i inne zasoby związane z siecią.
W celu przechwycenia ruchu sieciowego kontenera musimy uruchomić nasz program przechwytujący (tdpdump, tshark, etc.) w kontekście właściwej przestrzeni sieciowej docelowego kontenera. Jak to zrobić? Istnieją dwa typowe sposoby, w zależności od tego, czy chcemy skorzystać z oprogramowania dostępnego w systemie hosta czy też wolimy uruchomić stosowne narzędzia w kontenerze.
nsenter
nsenter umożliwia uruchomienie procesu (domyślnie shella) we wskazanych przestrzeniach nazw docelowego procesu. W przypadku kontenerów Dockera typowy sposób postępowania polega na tym, że najpierw znajdujemy identyfikator kontenera, a następnie jego PID w przestrzeni procesów hosta.
mkdir ~/data
CID=$(docker ps --format {{.Names}} | grep CONTAINER_NAME)
PID=$(docker inspect --format {{.State.Pid}} $CID)
nsenter --target $PID --net tcpdump -i any \
-w ~/data/$(hostname)-$(date +%Y-%m-%d_%H-%M).pcap
Przełącznik -i any powoduje przechwytywanie ruchu z wszystkich interfejsów sieciowych (za pomocą tcpdump -D lub ip addr show można sprawdzić jakich dokładnie). Jeśli wersja nsenter dostępna w repozytorium dystrybucji nie jest odpowiednia można zainstalować aktualną w sposób opisany tutaj.
Uruchomienie kontenera z wybranymi narzędziami w przestrzeni sieciowej docelowego kontenera przez –net=container:<name|id>
Uruchamiając kontener można użyć istniejącej przestrzeni sieciowej związanej z kontenerem który chcemy monitorować; stosując opcję –net=container: można uruchomić procesy używając wirtualnego stosu sieciowego, który został już utworzony w innym kontenerze.
Ta opcja jest wygodna, gdy nie możemy/nie chcemy instalować niczego w systemie hosta. Często używanym do tego celu kontenerem z mnóstwem dostępnych narzędzie jest nicolaka/netshoot
mkdir ~/data
CID=$(docker ps --format {{.Names}} | grep CONTAINER_NAME)
docker run --rm -it -v ~/data:/data \
--net=container:$CID nicolaka/netshoot \
tcpdump -i any -w /data/$(hostname)-$(date +%Y-%m-%d_%H-%M).pcap
Debugging any JVM in a Docker container
Thanks to JAVA_TOOL_OPTIONS variable it’s easy to run any JVM-based Docker image in debug mode. All we have to do is add environment variable definition „JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005” in docker run or docker-compose.yml and expose port to connect debugger
# docker run
-e "JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" \
-p 5005:5005
# docker-compose
environment:
- "JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
ports:
- 5005:5005
If it's possible, it's handy to extend application entrypoint
# Set debug options if required
if [ x"${JAVA_ENABLE_DEBUG}" != x ] && [ "${JAVA_ENABLE_DEBUG}" != "false" ]; then
java_debug_args="-agentlib:jdwp=transport=dt_socket,server=y,suspend=${JAVA_DEBUG_SUSPEND:-n},address=${JAVA_DEBUG_PORT:-5005}"
fi
JVM in Docker and PTRACE_ATTACH
Docker nowadays (since 1.10, the original pull request is here docker/docker/#17989) adds some security to running containers by wrapping them in both AppArmor (or presumably SELinux on RedHat systems) and seccomp eBPF based syscall filters (here’s a nice article about it). And ptrace is disabled in the default seccomp profile.
$ docker run alpine sh -c 'apk add -U strace && strace echo' fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz (1/1) Installing strace (4.11-r2) Executing busybox-1.24.2-r11.trigger OK: 6 MiB in 12 packages strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted +++ exited with 1 +++
Why am I writing about this? Because some JDK tools depend on PTRACE_ATTACH on Linux. One of them is very useful jmap.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ jmap -heap <pid> | |
| Attaching to process ID 142, please wait... | |
| Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 142: Operation not permitted | |
| sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 142: Operation not permitted | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278) | |
| at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671) | |
| at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611) | |
| at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337) | |
| at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304) | |
| at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140) | |
| at sun.jvm.hotspot.tools.Tool.start(Tool.java:185) | |
| at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) | |
| at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at sun.tools.jmap.JMap.runTool(JMap.java:201) | |
| at sun.tools.jmap.JMap.main(JMap.java:130) | |
| Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 142: Operation not permitted | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138) |
Turning seccomp off (–security-opt seccomp=unconfined) is not recommended, but we can add just this one explicit capability with –cap-add=SYS_PTRACE.
$ docker run --cap-add=SYS_PTRACE alpine sh -c 'apk add -U strace && strace echo'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/1) Installing strace (4.11-r2)
Executing busybox-1.24.2-r11.trigger
OK: 6 MiB in 12 packages
execve("/bin/echo", ["echo"], [/* 5 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7feaca3a8b28) = 0
set_tid_address(0x7feaca3a8b60) = 10
mprotect(0x7feaca3a5000, 4096, PROT_READ) = 0
mprotect(0x558c47ec6000, 16384, PROT_READ) = 0
getuid() = 0
write(1, "\n", 1) = 1
exit_group(0) = ?
+++ exited with 0 +++
Docker Compose supports cap_add since version 1.1.0 (2015-02-25).
If you run into an issue with the jmap and jstack from OpenJDK failing with exception java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap make sure you install openjdk-debuginfo package (or openjdk-8-dbg or something similiar depending on distro).
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ jmap -heap 66 | |
| Attaching to process ID 66, please wait... | |
| Debugger attached successfully. | |
| Server compiler detected. | |
| JVM version is 25.111-b14 | |
| using parallel threads in the new generation. | |
| using thread-local object allocation. | |
| Concurrent Mark-Sweep GC | |
| Heap Configuration: | |
| MinHeapFreeRatio = 40 | |
| MaxHeapFreeRatio = 70 | |
| MaxHeapSize = 268435456 (256.0MB) | |
| NewSize = 87228416 (83.1875MB) | |
| MaxNewSize = 87228416 (83.1875MB) | |
| OldSize = 181207040 (172.8125MB) | |
| NewRatio = 2 | |
| SurvivorRatio = 8 | |
| MetaspaceSize = 21807104 (20.796875MB) | |
| CompressedClassSpaceSize = 1073741824 (1024.0MB) | |
| MaxMetaspaceSize = 17592186044415 MB | |
| G1HeapRegionSize = 0 (0.0MB) | |
| Heap Usage: | |
| Exception in thread "main" java.lang.reflect.InvocationTargetException | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at sun.tools.jmap.JMap.runTool(JMap.java:201) | |
| at sun.tools.jmap.JMap.main(JMap.java:130) | |
| Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap | |
| at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:144) | |
| at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) | |
| at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) | |
| at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) | |
| at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49) | |
| ... 6 more |
Running Linux graphical applications in Docker on Windows with Cygwin/X
Install Babun
Cygwin is a great tool, but not the easiest to install. Babun consists of a pre-configured Cygwin that does not interfere with existing Cygwin installation.
Download the dist file from http://babun.github.io, unzip it and run the install.bat script. After a few minutes the application will be installed to the %USERPROFILE%\.babun directory. You can use the /target (or /t) option to install babun to a custom directory.
Install Cygwin/X
Run pact from babun shell (pact is a babun package manager )
pact install xorg-server xinit xhost
Start the X server
Once the installation has completed, open a Cygwin terminal and run XWin :0 -listen tcp -multiwindow. This will start an X server on Windows machine
with the ability to listen to connections from the network (-listen tcp) and display
each application in its own window (-multiwindow), rather than a single window acting
as a virtual screen to display applications on. Once it’s started, you should see an
„X” icon in Windows tray area.
Run graphical application
fr3nd/xeyes is a good test to run
// don't forget to change WINDOWS_MACHINE_IP_ADDR! // 'localhost' obviously won't work from within Docker container docker run -e DISPLAY=$WINDOWS_MACHINE_IP_ADDR:0 --rm fr3nd/xeyes
Or we can build ourselves image with Firefox using the following Dockerfile as a starting point
FROM centos RUN yum -y update && yum install -y firefox CMD /usr/bin/firefox
docker build -t firefox . it and run the container with
export DISPLAY=$WINDOWS_MACHINE_IP_ADD:0 docker run -ti --rm -e DISPLAY=$DISPLAY firefox
If all goes well you should see Firefox running from within a Docker container.
Troubleshooting
If you have issues with authorization you may want to try running the insecure xhost + command to permit access from all machines. See xhost(1) Linux man page.
Alternatives
There are a few different options to run GUI applications inside a Docker container like using SSH with X11 forwarding or VNC.
Docker Clean Up!
Over time docker eat up the disk space of the host. Here’s a script to keep it under control.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # Author: Jarek Przygódzki <jarek.przygodzki@gmail.com> | |
| # | |
| # Docker 1.9+ | |
| # | |
| # In Docker 1.9, there is the ability to manage volumes better | |
| # with docker volume (see https://github.com/docker/docker/pull/14242) | |
| # | |
| # Delete all stopped containers | |
| echo "Delete all stopped containers" | |
| docker ps -q -f status=exited | xargs --no-run-if-empty docker rm | |
| # Delete all dead containers | |
| echo "Delete all dead containers" | |
| docker ps -q -f status=dead | xargs --no-run-if-empty docker rm | |
| # Delete all dangling (unused) images | |
| echo "Delete all dangling (unused) images" | |
| docker images -q -f dangling=true | xargs --no-run-if-empty docker rmi | |
| # Delete all dangling (orphaned) volumes | |
| echo "Delete all dangling (orphaned) volumes" | |
| docker volume ls -qf dangling=true | xargs --no-run-if-empty docker volume rm |
Use with care.
curl -fsSL https://git.io/voKIi | sh
Jarek Przygódzki. Blog programisty 