A server compiler crash during loop optimization

I encountered a server compiler crash which happens in a Google server. Here's a link to more details:

http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2008-September/000320.html

It appears to happen in the split-if graph transformation during the loop optimization. It only happens in a big Java server and I couldn't create a small test for the crash. I was able to create a small patch as a workaround that detects the crash condition early and aborts the JIT compilation of the particular method.

It is, unfortunately, often the case of JVM crashes only happen in an entire application run (production code) and they are not reproducible in tests. One possible reason is that only some particular method inlining patterns trigger crash during JIT compilation.

JNI crashes in FontManager code

I hit a JVM crash happening in the Java 2D font manager code. Here's the stack trace at the crash point:

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x298366]
C  [libfontmanager.so+0x59e4]
C  [libfreetype.so.6+0x73c9]  FT_Stream_Close+0x19
C  [libfreetype.so.6+0xa065]  FT_Stream_Free+0x25
C  [libfreetype.so.6+0xa6e2]
C  [libfreetype.so.6+0xaf78]  FT_Done_Face+0x78
C  [libfontmanager.so+0x6964]
Java_sun_font_FreetypeFontScaler_disposeNativeScaler+0x34

This appeared to be due to two JNI pitfalls:

• The JNIEnv is unique to the thread. It cannot be saved by one thread and reused by another. Use GetEnv instead.
• The font2D jobject in freetypeScaler.c needs to be converted into a global reference because its lifetime exceeds the lifetime of a native method call.

BTW, here's a link to one of the JNI references:

http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/jniTOC.html

Martin Buchholz and I suggested a patch. But here's a fix that was actually submitted by Igor Nekrestyanov:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/11d333de082f

Thanks, guys.

Java 2D incompatibility between Sun JDK and OpenJDK

The Java 2D font renderer component was replaced when Sun JDK was open-sourced as OpenJDK with an open source version. But it appeared to have some bug. It's was about the vertical position gap in rendered fonts. I reported this bug on the OpenJDK mailing list and got a bug at:

http://bugs.sun.com/view_bug.do?bug_id=6761856

The results from the test (included in the bug page):

Sun JDK:   x=5.78125,y=-47.796875,w=633.71484,h=57.515625
OpenJDK:   x=5.78125,y=0.0,w=637.21875,h=96.34375

As you can see, the vertical positions are significantly off.

A fix was provided in OpenJDK7 by Sun folks:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/9cdababf6179

Thanks to Phil Race and Igor Nekrestyanov.

Stabilizing AsyncGetCallTrace

AsyncGetCallTrace() is the unofficial interface for low overhead CPU profiling in the JVM. It allows CPU profilers to use a signal (SIGPROF) to collect samples of stack traces. Unlike many Java profiling tools out there that instruments bytecode, it has much lower runtime overhead. The problem was that it's a bit unstable and sometimes caused JVM crashes in OpenJDK6. So, I backported the change

http://hg.openjdk.java.net/jdk7/jdk7/hotspot/rev/93b6525e3b82

into OpenJDK6 b11 (the patch attached) to avoid JVM crashes in the
AsyncGetCallTrace(). The change appears to have been first introduced
in OpenJDK7 b27 (Hotspot v13-b01).

I sent the patch to the OpenJDK mailing list, but unfortunately it's not been accepted yet.

[Note: the patch is in Hotspot V14. So, upgrading to the JDK based on it will fix the crashes.]

Inaccurate permgen usage stats (jmap -permstat)

The jmap is a JVM memory inspection tool. "jmap -permstat pid" is the option to show the stats on the permgen (permanent generation) heap inside the JVM, where class metadata and interned strings are allocated.

I happened to find a bug in jmap. In a small test, the "jmap -permstat" command reports only about 50-60% of the permgen memory usage (compared to the actual usage of permgen based on what the "jmap" command reports). To fix it, I contributed the following patch to OpenJDK, which increases the number up to 80-90%:

http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/7f601f7c9b48
http://bugs.sun.com/view_bug.do?bug_id=6731726

Thanks to Martin Buchholz who submitted the patch on my behalf and Swamy Venkataramanappa and Daniel Daugherty for reviewing it.

Building OpenJDK

[This might be already out of date.]

Here's how I build a 32 bit OpenJDK (build 31) on my Ubuntu 8.04 (Hardy Heron) system.

1. Download
Download the following two files

• the source bundle,
  
• the binary plug which contains a small amount of the binary-only components
  

from the download page, into the directory ~/openjdk7-ws/ and you get two files: the source bundle

~/openjdk7-ws/openjdk-7-ea-src-b31-17_jul_2008.zip

and the binary plugs

~/openjdk7-ws/jdk-7-ea-plug-b31-linux-i586-17_jul_2008.jar

2. Install required packages
Make sure that you install the following Ubuntu packages (I use the Synaptic Package Manger)

• ant
• ant-optional
• gawk
• gcc
• g++
• libasound2-dev
• libcupsys2-dev
• libfreetype6-dev
• libmotif-dev
• libx11proto-print-dev
• libxi-dev
• libxrender-dev
• libxt-dev
• libxext-dev
• libxtst-dev
• sun-java6-jdk

3. Unpack the source
Unzip the source bundle by typing

$ cd ~/openjdk7-ws
$ unzip openjdk-7-ea-src-b31-17_jul_2008.zip

and you will get the source directory

~/openjdk7-ws/openjdk.

4. Unpack the binary plugs
Expand the binary plugs by typing

$ cd ~/openjdk7-ws
$ java -jar jdk-7-ea-plug-b31-linux-i586-17_jul_2008.jar

which invokes a Java GUI extractor. Accept the license and select ~/openjdk7-ws as the destination and you get the binary plugs directory

~/openjdk7-ws/openjdk-binary-plugs.

5. Suppress compiler warnings and errors
Edit the file

~/openjdk7-ws/openjdk/hotspot/make/linux/makefiles

and add the following compiler option

CFLAGS += -Wno-write-strings

at the end of the list of the compiler options (line 60.)

And replace

#!/bin/sh

with

#!/bin/bash

in the first line of the six .sh files under

~/openjdk7-ws/openjdk/jdk/make/java/nio/

and change the value of variable SH from sh to bash in line 121 of

~/openjdk7-ws/openjdk/jdk/make/common/shared/Defs-utils.gmk

6. Build
Start the build by typing the following

$ cd ~/openjdk7-ws
$ unset JAVA_HOME
$ export ALT_BOOTDIR=/usr/lib/jvm/java-6-sun
$ export ALT_BINARY_PLUGS_PATH=~/openjdk7-ws/openjdk-binary-plugs
$ export LANG=C
$ make debug_build

and wait for a while and you will get the debug build in

$ ~/openjdk7-ws/openjdk/build/linux-i586-debug/j2sdk-image

To build a product build, type

$ make product_build

and you will get the product build in

$ ~/openjdk7-ws/openjdk/build/linux-i586/j2sdk-image

To clean the build directory, use

$ make clobber