Wednesday, December 31, 2008

A server compiler crash during loop optimization

I encountered a server compiler crash which happens in a Google server. Here's a link to more details:

http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2008-September/000320.html

It appears to happen in the split-if graph transformation during the loop optimization. It only happens in a big Java server and I couldn't create a small test for the crash. I was able to create a small patch as a workaround that detects the crash condition early and aborts the JIT compilation of the particular method.

It is, unfortunately, often the case of JVM crashes only happen in an entire application run (production code) and they are not reproducible in tests. One possible reason is that only some particular method inlining patterns trigger crash during JIT compilation.

JNI crashes in FontManager code

I hit a JVM crash happening in the Java 2D font manager code. Here's the stack trace at the crash point:

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x298366]
C [libfontmanager.so+0x59e4]
C [libfreetype.so.6+0x73c9] FT_Stream_Close+0x19
C [libfreetype.so.6+0xa065] FT_Stream_Free+0x25
C [libfreetype.so.6+0xa6e2]
C [libfreetype.so.6+0xaf78] FT_Done_Face+0x78
C [libfontmanager.so+0x6964]
Java_sun_font_FreetypeFontScaler_disposeNativeScaler+0x34

This appeared to be due to two JNI pitfalls:
  • The JNIEnv is unique to the thread. It cannot be saved by one thread and reused by another. Use GetEnv instead.
  • The font2D jobject in freetypeScaler.c needs to be converted into a global reference because its lifetime exceeds the lifetime of a native method call.
BTW, here's a link to one of the JNI references:

http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/jniTOC.html

Martin Buchholz and I suggested a patch. But here's a fix that was actually submitted by Igor Nekrestyanov:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/11d333de082f

Thanks, guys.

Java 2D incompatibility between Sun JDK and OpenJDK

The Java 2D font renderer component was replaced when Sun JDK was open-sourced as OpenJDK with an open source version. But it appeared to have some bug. It's was about the vertical position gap in rendered fonts. I reported this bug on the OpenJDK mailing list and got a bug at:

http://bugs.sun.com/view_bug.do?bug_id=6761856

The results from the test (included in the bug page):

Sun JDK: x=5.78125,y=-47.796875,w=633.71484,h=57.515625
OpenJDK: x=5.78125,y=0.0,w=637.21875,h=96.34375

As you can see, the vertical positions are significantly off.

A fix was provided in OpenJDK7 by Sun folks:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/9cdababf6179


Thanks to Phil Race and Igor Nekrestyanov.

Stabilizing AsyncGetCallTrace

AsyncGetCallTrace() is the unofficial interface for low overhead CPU profiling in the JVM. It allows CPU profilers to use a signal (SIGPROF) to collect samples of stack traces. Unlike many Java profiling tools out there that instruments bytecode, it has much lower runtime overhead. The problem was that it's a bit unstable and sometimes caused JVM crashes in OpenJDK6. So, I backported the change

http://hg.openjdk.java.net/jdk7/jdk7/hotspot/rev/93b6525e3b82

into OpenJDK6 b11 (the patch attached) to avoid JVM crashes in the
AsyncGetCallTrace(). The change appears to have been first introduced
in OpenJDK7 b27 (Hotspot v13-b01).

I sent the patch to the OpenJDK mailing list, but unfortunately it's not been accepted yet.

[Note: the patch is in Hotspot V14. So, upgrading to the JDK based on it will fix the crashes.]

Inaccurate permgen usage stats (jmap -permstat)

The jmap is a JVM memory inspection tool. "jmap -permstat pid" is the option to show the stats on the permgen (permanent generation) heap inside the JVM, where class metadata and interned strings are allocated.

I happened to find a bug in jmap. In a small test, the "jmap -permstat" command reports only about 50-60% of the permgen memory usage (compared to the actual usage of permgen based on what the "jmap" command reports). To fix it, I contributed the following patch to OpenJDK, which increases the number up to 80-90%:

http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/7f601f7c9b48
http://bugs.sun.com/view_bug.do?bug_id=6731726

Thanks to Martin Buchholz who submitted the patch on my behalf and Swamy Venkataramanappa and Daniel Daugherty for reviewing it.

Friday, July 25, 2008

Building OpenJDK

[This might be already out of date.]

Here's how I build a 32 bit OpenJDK (build 31) on my Ubuntu 8.04 (Hardy Heron) system.

1. Download
Download the following two files
from the download page, into the directory ~/openjdk7-ws/ and you get two files: the source bundle
~/openjdk7-ws/openjdk-7-ea-src-b31-17_jul_2008.zip
and the binary plugs
~/openjdk7-ws/jdk-7-ea-plug-b31-linux-i586-17_jul_2008.jar


2. Install required packages
Make sure that you install the following Ubuntu packages (I use the Synaptic Package Manger)
  • ant
  • ant-optional
  • gawk
  • gcc
  • g++
  • libasound2-dev
  • libcupsys2-dev
  • libfreetype6-dev
  • libmotif-dev
  • libx11proto-print-dev
  • libxi-dev
  • libxrender-dev
  • libxt-dev
  • libxext-dev
  • libxtst-dev
  • sun-java6-jdk

3. Unpack the source

Unzip the source bundle by typing
$ cd ~/openjdk7-ws
$ unzip openjdk-7-ea-src-b31-17_jul_2008.zip
and you will get the source directory
~/openjdk7-ws/openjdk.

4. Unpack the binary plugs

Expand the binary plugs by typing
$ cd ~/openjdk7-ws
$ java -jar jdk-7-ea-plug-b31-linux-i586-17_jul_2008.jar
which invokes a Java GUI extractor. Accept the license and select ~/openjdk7-ws as the destination and you get the binary plugs directory
~/openjdk7-ws/openjdk-binary-plugs.


5. Suppress compiler warnings and errors

Edit the file
~/openjdk7-ws/openjdk/hotspot/make/linux/makefiles
and add the following compiler option
CFLAGS += -Wno-write-strings
at the end of the list of the compiler options (line 60.)

And replace
#!/bin/sh
with
#!/bin/bash
in the first line of the six .sh files under
~/openjdk7-ws/openjdk/jdk/make/java/nio/
and change the value of variable SH from sh to bash in line 121 of
~/openjdk7-ws/openjdk/jdk/make/common/shared/Defs-utils.gmk


6. Build

Start the build by typing the following
$ cd ~/openjdk7-ws
$ unset JAVA_HOME
$ export ALT_BOOTDIR=/usr/lib/jvm/java-6-sun
$ export ALT_BINARY_PLUGS_PATH=~/openjdk7-ws/openjdk-binary-plugs
$ export LANG=C
$ make debug_build
and wait for a while and you will get the debug build in
$ ~/openjdk7-ws/openjdk/build/linux-i586-debug/j2sdk-image
To build a product build, type
$ make product_build
and you will get the product build in
$ ~/openjdk7-ws/openjdk/build/linux-i586/j2sdk-image
To clean the build directory, use
$ make clobber