Published on dev.to: https://dev.to/jarekprzygodzki/a-curious-case-of-slow-docker-image-builds-2o7k
Generating JVM memory dumps from JRE
Published on dev.to: Generating JVM memory dumps from JRE
How to install a Certificate Authority (CA) root certificate for Docker registry on local Boot2Docker VM
Annoyingly often internal Docker registries are secured with certificates signed by company’s own PKI or enterprise IT does a MitM to replace all HTTPS certs.
Commonly, company’s root CA certificate is installed by IT on developers machines and servers, but not on VMs run by developers on their own machines. When using Docker with local VMs like boot2docker, do we need to install the company root CA certificate on the VM to avoid x509: certificate signed by unknown authority errors.
There are two ways to do it – both are documented here.
Adding trusted CA root certificates to VM OS cert store
Let’s start with this option. Docker daemon respects OS cert store. To make the certificate survive machine restart it has to be placed in /var/lib/boot2docker/certs directory on persistent partition . In Boot2Docker certificates (in .pem or .crt format) from that directory are automatically load at boot. See boot2docker/rootfs/etc/rc.d/install-ca-certs for details.
There’s also open issue in docker-machine to support installing root CA certificates on machine creation and instruction how to build boot2docker ISO with custom CA certificate for private docker registry.
Addding trusted CA root certificate for specific registries
Docker allows to specify custom CA root for a
specific registry hostname. It can configured per registry by creating a directory under /etc/docker/certs.d using the same name as the registry’s hostname (including port number if any). All *.crt files from this directory are added as CA roots, details are in moby/registry.go#newTLSConfig.
Another option to deal with insecure registries is enabling insecure communication with specified registries (no certificate verification and HTTP fallback). See insecure-registry for details.
Publishing a port from a running Docker container
It’s not obvious, but it’s possible to publish an additional port from a running Docker container.
Note: Assuming container is attached to user-defined bridge network or default bridge network. Publishing ports doesn’t make sense with MacVLAN or IPVLAN since then every container has a uniquely routable IP address and can offer service on any port.
What can we do to make container port externally accessible?
One possibile solution is to exploit the fact that communication between containers inside the same bridge network is switched at layer 2 and they can communiate with each other without any restrictions (unless ICC is disabled, that is). That means when can run another container with socat inside target network with published port and setup forwarding
docker run --rm -it \ --net=[TARGET_NETWORK] -p [PORT]:[PORT] \ bobrik/socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]
What else can we do do? It’s not immediately obvious, but each bridge network is reachable from the host. We can verify with `ip route show` that in host’s routing table exists routing table entry for each bridge network
$ ip route show 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 dev br-b4247d00e80c proto kernel scope link src 172.18.0.1 172.19.0.0/16 dev br-acd7d3307ea3 proto kernel scope link src 172.19.0.1
That means we can as well setup socat relay in host network namespace
docker run --rm -it \ --net=host \ -p [PORT]:[PORT] \ bobrik/socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]
or just run socat directly on host
socat TCP-LISTEN:[PORT],fork TCP:[CONTAINER_IP]:[CONTAINER_PORT]
But we can still do better! We can replace socat relay with iptables DNAT. For inbound traffic, we’ll need to create a custom rule that uses Network Address Translation (NAT) to map a port on the host to the service port in the container. We can do that with a rule like this:
# must be added into DOCKER chain
iptables -t nat -A DOCKER -p tcp -m tcp \
--dport [PORT] -j DNAT \
--to-destination [CONTAINER_IP]:[CONTAINER_PORT]
This is pretty much how Docker publishes ports from containers using the bridged network when option -publish|-p is used with docker run. The only difference is that DNAT rule created by Docker is restricted not to affect traffic originating from bridge
iptables -t nat -A DOCKER ! -i [BRIDGE_NAME] -p tcp -m tcp \
--dport [PORT] -j DNAT \
--to-destination [CONTAINER_IP]:[CONTAINER_PORT]
The DOCKER chain is a custom chain defined at the FORWARD chain. When a packet hits any interface and is bound to the one of bridge interfaces, it is sent to the custom DOCKER chain. Now the DOCKER chain will take all incoming packets, except ones coming from bridge (say docker0 for default network), and send them to a container IP (usually 172.x.x.x) and port.
Monitorowanie i przechwytywanie ruchu sieciowego kontenerów Dockera
Ten wpis to krótki tutorial jak przechwycić ruch sieciowych z wybranych kontenerów Dockera, np. usług uruchomionych w ramach klastra Docker Swarm.
Na początek minimum teorii. Docker używa przestrzeni nazw w celu zapewnienia izolacji procesów. Przestrzenie nazw (których od jądra 4.10 jest 7 rodzajów) są cechą jądra systemu Linux, która pozwala na izolację i wirtualizację zasobów systemowych. Funkcja przestrzeni nazw jest taka sama dla każdego typu: każdy proces jest powiązany z przestrzenią nazw danego rodzaju i może wykorzystywać zasoby powiązane z tą przestrzenią, oraz, w stosownych przypadkach, przestrzeniami nazw potomków. W kontekście ruchu sieciowego interesuje nas przestrzeń nazw dla sieci (network namespace).
Sieciowe przestrzenie nazw wirtualizują stos sieciowy – umożliwiają odizolowanie całego podsystemu sieciowego dla grupy procesów. Każda przestrzeń nazw posiada prywatny zestaw adresów IP, własną tabelę routingu, listę gniazd, tabelę śledzenia połączeń, zaporę i inne zasoby związane z siecią.
W celu przechwycenia ruchu sieciowego kontenera musimy uruchomić nasz program przechwytujący (tdpdump, tshark, etc.) w kontekście właściwej przestrzeni sieciowej docelowego kontenera. Jak to zrobić? Istnieją dwa typowe sposoby, w zależności od tego, czy chcemy skorzystać z oprogramowania dostępnego w systemie hosta czy też wolimy uruchomić stosowne narzędzia w kontenerze.
nsenter
nsenter umożliwia uruchomienie procesu (domyślnie shella) we wskazanych przestrzeniach nazw docelowego procesu. W przypadku kontenerów Dockera typowy sposób postępowania polega na tym, że najpierw znajdujemy identyfikator kontenera, a następnie jego PID w przestrzeni procesów hosta.
mkdir ~/data
CID=$(docker ps --format {{.Names}} | grep CONTAINER_NAME)
PID=$(docker inspect --format {{.State.Pid}} $CID)
nsenter --target $PID --net tcpdump -i any \
-w ~/data/$(hostname)-$(date +%Y-%m-%d_%H-%M).pcap
Przełącznik -i any powoduje przechwytywanie ruchu z wszystkich interfejsów sieciowych (za pomocą tcpdump -D lub ip addr show można sprawdzić jakich dokładnie). Jeśli wersja nsenter dostępna w repozytorium dystrybucji nie jest odpowiednia można zainstalować aktualną w sposób opisany tutaj.
Uruchomienie kontenera z wybranymi narzędziami w przestrzeni sieciowej docelowego kontenera przez –net=container:<name|id>
Uruchamiając kontener można użyć istniejącej przestrzeni sieciowej związanej z kontenerem który chcemy monitorować; stosując opcję –net=container: można uruchomić procesy używając wirtualnego stosu sieciowego, który został już utworzony w innym kontenerze.
Ta opcja jest wygodna, gdy nie możemy/nie chcemy instalować niczego w systemie hosta. Często używanym do tego celu kontenerem z mnóstwem dostępnych narzędzie jest nicolaka/netshoot
mkdir ~/data
CID=$(docker ps --format {{.Names}} | grep CONTAINER_NAME)
docker run --rm -it -v ~/data:/data \
--net=container:$CID nicolaka/netshoot \
tcpdump -i any -w /data/$(hostname)-$(date +%Y-%m-%d_%H-%M).pcap
Debugging any JVM in a Docker container
Thanks to JAVA_TOOL_OPTIONS variable it’s easy to run any JVM-based Docker image in debug mode. All we have to do is add environment variable definition „JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005” in docker run or docker-compose.yml and expose port to connect debugger
# docker run
-e "JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" \
-p 5005:5005
# docker-compose
environment:
- "JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
ports:
- 5005:5005
If it's possible, it's handy to extend application entrypoint
# Set debug options if required
if [ x"${JAVA_ENABLE_DEBUG}" != x ] && [ "${JAVA_ENABLE_DEBUG}" != "false" ]; then
java_debug_args="-agentlib:jdwp=transport=dt_socket,server=y,suspend=${JAVA_DEBUG_SUSPEND:-n},address=${JAVA_DEBUG_PORT:-5005}"
fi
On why and how to exit JVM on OnOutOfMemoryError
For a long time all my JVM-based Docker images were configured to exit on OOM error with -XX:OnOutOfMemoryError=”kill -9 %p” (%p is the current JVM process PID placeholder). It works well with XX:+HeapDumpOnOutOfMemoryError, because JVM will dump heap first, and then execute OnOutOfMemoryError command (see the relevant code in vm/utilities/debug.cpp ). But with version 8u92 there’s now a JVM option in the JDK to make the JVM exit when an OutOfMemoryError occurs:
From the release notes:
ExitOnOutOfMemoryError
When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.CrashOnOutOfMemoryError
If this option is enabled, when an out-of-memory error occurs, the JVM crashes and produces text and binary crash files.
Enhancement Request: JDK-8138745 (parameter naming is wrong though JDK-8154713, ExitOnOutOfMemoryError instead of ExitOnOutOfMemory)
Why exit on OOM? OutOfMemoryError may seem like any other exception, but if it escapes from Thread.run() it will cause thread to die. When thread dies it is no longer a GC root, and thus all references kept only by this thread are eligible for garbage collection. While it means that JVM has a chance recover from OOME, its not recommended that you try. It may work, but it is generally a bad idea. See this answer on SO.
JVM in Docker and PTRACE_ATTACH
Docker nowadays (since 1.10, the original pull request is here docker/docker/#17989) adds some security to running containers by wrapping them in both AppArmor (or presumably SELinux on RedHat systems) and seccomp eBPF based syscall filters (here’s a nice article about it). And ptrace is disabled in the default seccomp profile.
$ docker run alpine sh -c 'apk add -U strace && strace echo' fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz (1/1) Installing strace (4.11-r2) Executing busybox-1.24.2-r11.trigger OK: 6 MiB in 12 packages strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted +++ exited with 1 +++
Why am I writing about this? Because some JDK tools depend on PTRACE_ATTACH on Linux. One of them is very useful jmap.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ jmap -heap <pid> | |
| Attaching to process ID 142, please wait... | |
| Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 142: Operation not permitted | |
| sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 142: Operation not permitted | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278) | |
| at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671) | |
| at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611) | |
| at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337) | |
| at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304) | |
| at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140) | |
| at sun.jvm.hotspot.tools.Tool.start(Tool.java:185) | |
| at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) | |
| at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at sun.tools.jmap.JMap.runTool(JMap.java:201) | |
| at sun.tools.jmap.JMap.main(JMap.java:130) | |
| Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 142: Operation not permitted | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269) | |
| at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138) |
Turning seccomp off (–security-opt seccomp=unconfined) is not recommended, but we can add just this one explicit capability with –cap-add=SYS_PTRACE.
$ docker run --cap-add=SYS_PTRACE alpine sh -c 'apk add -U strace && strace echo'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/1) Installing strace (4.11-r2)
Executing busybox-1.24.2-r11.trigger
OK: 6 MiB in 12 packages
execve("/bin/echo", ["echo"], [/* 5 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7feaca3a8b28) = 0
set_tid_address(0x7feaca3a8b60) = 10
mprotect(0x7feaca3a5000, 4096, PROT_READ) = 0
mprotect(0x558c47ec6000, 16384, PROT_READ) = 0
getuid() = 0
write(1, "\n", 1) = 1
exit_group(0) = ?
+++ exited with 0 +++
Docker Compose supports cap_add since version 1.1.0 (2015-02-25).
If you run into an issue with the jmap and jstack from OpenJDK failing with exception java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap make sure you install openjdk-debuginfo package (or openjdk-8-dbg or something similiar depending on distro).
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ jmap -heap 66 | |
| Attaching to process ID 66, please wait... | |
| Debugger attached successfully. | |
| Server compiler detected. | |
| JVM version is 25.111-b14 | |
| using parallel threads in the new generation. | |
| using thread-local object allocation. | |
| Concurrent Mark-Sweep GC | |
| Heap Configuration: | |
| MinHeapFreeRatio = 40 | |
| MaxHeapFreeRatio = 70 | |
| MaxHeapSize = 268435456 (256.0MB) | |
| NewSize = 87228416 (83.1875MB) | |
| MaxNewSize = 87228416 (83.1875MB) | |
| OldSize = 181207040 (172.8125MB) | |
| NewRatio = 2 | |
| SurvivorRatio = 8 | |
| MetaspaceSize = 21807104 (20.796875MB) | |
| CompressedClassSpaceSize = 1073741824 (1024.0MB) | |
| MaxMetaspaceSize = 17592186044415 MB | |
| G1HeapRegionSize = 0 (0.0MB) | |
| Heap Usage: | |
| Exception in thread "main" java.lang.reflect.InvocationTargetException | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at sun.tools.jmap.JMap.runTool(JMap.java:201) | |
| at sun.tools.jmap.JMap.main(JMap.java:130) | |
| Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap | |
| at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:144) | |
| at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) | |
| at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) | |
| at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) | |
| at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49) | |
| ... 6 more |
Single-file Play Framework application with Ammonite
Single-file Play Framework application with Ammonite courtesy of Li Haoyi @li_haoyi
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /** | |
| * Single-file play framework application! Make sure everything | |
| * works, as this is the test case that un-earthed #371 | |
| */ | |
| load.ivy("com.typesafe.play" %% "play" % "2.5.0") | |
| load.ivy("com.typesafe.play" %% "play-netty-server" % "2.5.0") | |
| load.ivy("org.scalaj" %% "scalaj-http" % "2.2.1") | |
| @ | |
| import play.core.server._, play.api.routing.sird._, play.api.mvc._ | |
| import scalaj.http._ | |
| val server = NettyServer.fromRouter(new ServerConfig( | |
| rootDir = new java.io.File("."), | |
| port = Some(19000), sslPort = None, | |
| address = "0.0.0.0", mode = play.api.Mode.Dev, | |
| properties = System.getProperties, | |
| configuration = play.api.Configuration( | |
| "play.server.netty" -> Map( | |
| "maxInitialLineLength" -> 4096, | |
| "maxHeaderSize" -> 8192, | |
| "maxChunkSize" -> 8192, | |
| "log.wire" -> false, | |
| "eventLoopThreads" -> 0, | |
| "transport" -> "jdk", | |
| "option.child" -> Map() | |
| ) | |
| ) | |
| )) { | |
| case GET(p"/hello/$to") => Action { Results.Ok(s"Hello $to") } | |
| } | |
| Console.readLine() | |
| server.stop() | |
I hope that this awesome code snippet will encourage you to check out Ammonite-REPL (A Modernized Scala REPL).
Play! Framework – część I
Z związku z moim zainteresowaniem Scalą i Akką naturalnym jest zajęcie się ostatnim elementem stosu reaktywnych technologii dla JVM od Typesafe – frameworkiem Play. Niniejszy wpis stanowi, mam nadzieję, pierwszy z serii wpisów opisujących moją przygodę z tym frameworkiem.
Czym jest Play?
Play jest moim zdaniem najlepszym frameworkiem ogólnego przeznaczenia dla JVM
Ogólnie rzeczy ujmując Play to implementacja wzorca MVC w oparciu o model aktów Akka połączona ze środowiskiem do zarządzania projektem i uruchamiania go
Co w nim tak bardzo mi się podoba?
Jednym z głównych filarów architektonicznych Play jest podejście bezstanowe i asynchroniczne. W Play nie ma tradycyjnej „sesji”. Takie podejście nastręcza czasami pewne trudności, ale są one z nawiązką rekompensowane przez zalety takie jak łatwość horyzontalnego skalowania i wsparcie dla technologi reaktywnych. I Scala. Uwielbiam Scalę.
Zacznijmy jednak od podstaw.
Szkielet aplikacji Play można łatwo wygenerować za pomocą narzędzia Typesafe Activator – activator new [name] [template-id] – ale nie jest dobry sposób na rozpoczęcie przygody z Play.
activator new my-play-app play-scala
Zacznijmy więc od minimalnej aplikacji Play do której będziemy dodawać kolejne elementy w miarę potrzeby. Minimalna aplikacja jest naprawdę niewielka i składa się z czterech plików z których jeden (application.conf) jest pusty a jeden (build.properties) tak naprawdę opcjonalny.
$ ls -R .: build.properties build.sbt conf/ project/ ./conf: application.conf ./project: plugins.sbt
build.sbt
name := """my-play-app"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.11.7"
resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"
project/plugins.sbt
// The Play plugin
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.5.4")
project/build.properties
sbt.version=0.13.11
Nie jest to co prawda zbyt przydatna aplikacja – ale kompiluje się i uruchamia bez przeszkód o czym możemy się przekonać wydając polecenie activator ~run w katalogu aplikacji.
Skrypt wykonujący poszczególne krotki jest dostępny jako Gist
bash <(curl -fsSL https://git.io/vKW88) my-play-app
Jarek Przygódzki. Blog programisty 