OpenJDK at Google

At the JVM Language Summit 2013, Jeremy Manson gave a talk on some of the work we did around OpenJDK at Google:

  Slides: http://www.oracle.com/technetwork/java/jvmls2013manson-2013920.pdf
  Video: http://medianetwork.oracle.com/video/player/2630310903001

It covers some of my profiling and garbage collection work (on slides 29, 36 and 37):

- Making the stack trace decoding code more robust and precise and adding support for the native stack frames for the low-overhead AsyncGetCallTrace-based profiler.

- Parallel full GC for the CMS garbage collector. 2-4x pause time improvements in full GC.

- Parallel initial-mark and remark phases in the CMS garbage collector. 2-4x pause time improvements.

- Partial heap defragmentation/compaction for the CMS garbage collector. Decrease heap fragmentation-caused full GC by up to 90% (or eliminate completely.)

- Give back unused RAM to the system. 20-30% RAM savings.

Parallel initial mark (CMSParallelInitialMarkEnabled) and more-parallel remark (CMSEdenChunksRecordAlways) phases in the CMS garbage collector

The Concurrent Mark Sweep (CMS) garbage collector in the Hotspot JVM has two stop-the-world pauses in its algorithm: the initial mark and the remark phases. The initial mark phase starts the concurrent mark process by marking the objects in the old generation that are directly (as opposed to transitively) reachable from the GC roots (such as local variables, static fields, and the objects in the young generation.) The remark phase finishes the concurrent mark process by rescanning the objects in the GC roots and the objects in the dirtied cards.

One issue that I had with the CMS garbage collector was that the pause times of these two phases often spike up to 500 milliseconds or more, which is rather long especially when it's desirable to have server request latency that's much shorter than that.

A reason for the long initial mark pauses is that the initial mark code is not parallelized. Since the general trend of the increasing Java heap sizes and the initial mark phase needs to scan the entire young generation (whose size is typically a proportion of the heap size,) the existing single-threaded initial mark phase tends to result in long pauses even on modern processors.

Similarly, one reason for the long remark phase is that while the remark code is already parallelized (handled by multiple threads and CPU cores), its existing algorithm has a glitch that the workload distribution among the GC worker threads during its young generation scan sometimes gets unbalanced. When the parallel workload is unbalanced, the worker threads with less work will just wait idle for the ones with more work, and the overall time to finish the entire workload gets longer.

I recently made OpenJDK contributions that fix those issues. I implemented a parallel version of the initial mark phase (the CMSParallelInitialMarkEnabled flag/option) and a more evenly workload distributed version of the remark phase (the CMSEdenChunksRecordAlways flag/option). With these contributions, the initial/remark pause times get shorter by a factor of 5 or more. For example, in a test with a 1 GB young generation (within a 3 GB heap), the pause times stayed below 100 milliseconds compared to 500 milliseconds or more without these contributions.

A more detailed description is here:

  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-May/007374.html

After several long email threads that are archived at:

  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-May/thread.html

  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-June/thread.html

  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-August/thread.html

The patches have been accepted into the OpenJDK Hotspot code base as in:

  http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/ca9dedeebdec (parallel initial mark)
  http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/7b06ae405d7b (better parallelized remark)
  http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/cb9da55b1990 (a bug fix, authored by Jon)

Hopefully, with these changes, your initial/remark pause times will be shorter than before, especially in your big-heap Java applications.

Thanks to Jon Masamitsu, Thomas Schatzl, and Chuck Rasbold for sponsoring and/or reviewing the patches.

Making the JVM release memory

It's well known that a Java application (the JVM) won't typically release much memory once it's warmed up, even when the application is lightly loaded or even idle at a later time.

If you plot the memory usage (or the resident set size) of a Java application, it typically looks like a mostly flat line after an upslope at the beginning. At a low level, this corresponds to the memory pages of the Java heap gradually getting allocated, and once all the pages are allocated, the memory usages stays mostly flat even when a large portion of the heap is not used * **.

This can be a problem if a Java application is run on a non-dedicated system (a server or desktop) where it co-exists with other (non-Java) applications. In a non-dedicated system, one application that's not playing nice with others by dominating the memory can slow down the other applications, or prevent them from running.

This is where an experimental JVM feature, DeallocateHeapPages, that I worked on comes in. It causes the underlying memory pages that correspond to the unused (free) parts of the heap to be deallocated (released) and helps reduce the memory usage of a Java application. Internally, it calls the system call madvise(MADV_DONTNEED) for the bodies of free chunks in the old generation without unmapping the heap address space.

Another way to look at this is that this feature makes the memory usage of a Java application behave more like that of a C/C++ application where the process memory usage is more in line with the memory actually used by the application.

This has been very useful for servers and desktop tools that we have at Google and helped save a lot of memory (RAM) usage.

The implementation currently supports the concurrent mark sweep (CMS) collector and the Linux platform.

Here's the email thread on the OpenJDK mailing list and a link to the JVM patch:

  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-January/005664.html

  http://cr.openjdk.java.net/~hiroshi/webrevs/dhp/webrev.00/

The patch hasn't been accepted (yet) as the support for all the other OS platforms is deemed necessary for that to happen, which it lacks. I might be able to address that at some point, if I have the time and resources to make it happen.

* For simplicity, I am ignoring the memory use other than the heap such as the native C heap and the thread stacks here as the heap uses usually by far the largest amount of memory.

** Though the serial garbage collector (-XX:+UseSerialGC) of the JVM can occasionally shrink the heap and return memory, it's almost never used in production for an obvious performance reason. The parallel collector and the concurrent mark sweep (CMS) collector, which are often used in production, almost never shrink the heap and return memory, in my experience.

The JVM continuation contribution

As described in the previous post, I gave a talk on an experiment that used JVM continuations for writing servers. I worked on a JVM continuation implementation based off the original continuation patch and used it for the experiment. I contributed the work back to the OpenJDK MLVM project as the following checked-in patches:

    http://hg.openjdk.java.net/mlvm/mlvm/hotspot/rev/b63b73832341
    http://hg.openjdk.java.net/mlvm/mlvm/jdk/rev/568ebe7237f1

The overview of this implementation is described in a MLVM wiki page.

I thank Lukas Stadler, John Rose, and Chuck Rasbold for their help.

Talk: Continuations in servers

OK. It happened two years ago. But, for the record, I gave a talk Continuations in Servers at the JVM Language Summit 2010.

It's about experimenting with a JVM-level continuation and allowing servers to be written like a synchronous server for better productivity, but with the low overhead of an asynchronous server for better scalability. These two types of servers differ in terms of how threads are mapped to server requests:

1) A synchronous server dedicates a thread for a request for the lifetime of the request. Since a thread is dedicated, the request handling code can block, for example, for network I/O calls, database accesses, etc. It leads to more natural easier-to-understand coding style. For example, the following request handler calls the database twice in a blocking way.

Response handleRequest(Request req) {

  SomeData a = calldb1(req.param1);

  OtherData b = calldb2(a, req.param2);

  return new Response(a, b);

}

But, since a thread is relatively an expensive resource, the maximum number of in-flight requests that a server can simultaneously handle is bounded by the maximum number of threads and the associated overhead that a system can manage. This type of servers is sometimes called a thread-per-request,  thread-per-connection, or thread-based server.

2) An asynchronous server, on the other hand, does not dedicate a thread for a request for the lifetime of the request. It instead releases the thread whenever the request handling code blocks for I/O. The rest of the request handling logic is typically executed as a callback (after the I/O finishes) by (most likely) a different thread from a thread pool. This leads to a less intuitive harder-to-understand coding style. For example, the above request handler code would look like the following in an asynchronous style:

void handleRequest(final Response res, final Request req) {

  calldb1(req.param1,

    new Callback() {

      public void callback(final SomeData a) {

        calldb2(a, req.param2,

          new Callback() {

            public void callback(OtherData b) {

              res.send(a, b);

            }});

      }});

}

Basically, the flow of the code is interrupted at each I/O point, and you lose your call context (or call stack) because each part may be executed by a different thread.

But because the thread can be released and reused to handle other requests while a request is blocking in an I/O (imagine a network I/O which can take a while to come back,) it is more scalable. Here, a server can handle more requests than the number of available threads. This type of servers is sometimes called a select server or event-based server.

So, it's about a tradeoff between ease-of-coding and scalability.

Here's where a continuation comes in. A continuation can save a call context, detach it of the thread, and resume it later in a different thread. In theory, if it's applied at each I/O point (inside the I/O library, hidden from the application code), it can allow for the illusion of the synchronous style of coding for application code, while it allows the thread to be released at each I/O point as in an asynchronous style. So, it could have the best of both styles, that is, productivity and scalability.

I have done an experiment with this idea. I developed a JVM-level continuation implementation (based on a patch from the MLVM project) and applied it to a Java-based remote procedural call (RPC) library and an application server. The results are encouraging. The server code is written in a synchronous style just like in a synchronous server while the memory footprint overhead (of the thread stacks) and the server throughput are almost as good as an asynchronous server.

Of course, this is just an experiment, and the open question is whether continuations are suitable for Java and the JVM.

Joined the OpenJDK Hotspot group

Back in April, I was voted in and joined the OpenJDK HotSpot group. Here's a link to the call-for-votes email thread:

http://mail.openjdk.java.net/pipermail/hotspot-dev/2010-April/002866.html

Here's the list of people in various groups of the OpenJDK project:

http://db.openjdk.java.net/people

I thank Chuck Rasbold, who coordinated and hosted the voting process, and those who voted for my membership.

Short/Character.reverseBytes compiler intrinsics

I worked on a small improvement on the Hotspot server compiler.

Here's the mailing list thread:

http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2010-April/003054.html

Here's the committed change:

http://hg.openjdk.java.net/jdk7/hotspot-comp/hotspot/rev/d7f654633cfe

The Short/Character.reverseBytes() methods swap the bytes within the 2 byte short or char values. This is commonly done when, for a certain reason, one needs to convert a value from a big endian to little endian, or vice versa. The Short.reverseBytes() method looks like:

public static short reverseBytes(short i) {
  return (short) (((i & 0xFF00) >> 8) | (i << 8));
}

Ordinarily, this method is most likely going to be compiled into machine instruction sequences including an logical AND and two shift instructions. Since there is a machine instruction on x86 that does the swapping (BSWAP), why not let the compiler generate such code as a special code generation rule (called compiler intrinsics). In fact, the Integer/Long.reverseBytes() methods already took advantage of the instruction. This improvement is similar. The intrinsified Short.reverseBytes() looks like on x86 (32 bit):

BSWAP  reg 
SAR    reg, 16

Suppose the register contained four bytes B1 B2 B3 B4 (from the highest to the lowest on 32 bit). Since it was a short value (signed 2-byte integer), it must be either 0x00 0x00 B3 B4 (if the highest bit of B3 is 0) or 0xFF 0xFF B3 B4 (if the highest bit of B3 is 1). What we want eventually is either 0x00 0x00 B4 B3 (if the highest bit of B4 is 0) or 0xFF 0xFF B4 B3 (if B4's highest bit is 1).

The BSWAP instruction reverses the byte order of the word (four bytes) in register reg and produces B4 B3 B2 B1 in the register. Then, the SAR instruction shifts the word to the right by 2 bytes so that the lower two bytes are B4 B3. Since SAR fills the higher bits with the same bit as B4's highest bit, we get what we want.

Character.reverseBytes() are likewise, except that it needs to be treated unsigned.

In a microbenchmark bundled with the JDK, jdk/test/java/nio/Buffer/SwapMicroBenchmark, the char and short byte swap methods did speed up by 32% and 42%, respectively, with this improvement.

Thanks to colleagues, Chuck Rasbold (putting up the webrev on my behalf), Martin Buchholz (who worked on the JDK change to make it effective), Tom Rodriguez (who reviewed the change and finished the SPARC implementation), Christian Thalinger (who also reviewed the change).

Shorter long multiplication in Server Compiler on x86 32 bit

The other day, I contributed a patch that lets Hotspot Server compiler emit shorter multiplication sequence for long (64bit) multiplications. Here are some links:

The patch + the commit message: http://hg.openjdk.java.net/jdk7/hotspot-comp/hotspot/rev/e8443c7be117

Sun bug:
http://bugs.sun.com/view_bug.do?bug_id=6921969

Mailing list posts:http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2010-January/002639.html

http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2010-February/002660.html

It applies well to the BigInteger.mulAdd loop:

  private final static long LONG_MASK = 0xffffffffL;
  static int mulAdd(int[] out, int[] in, int offset, int len, int k) {
    long kLong = k & LONG_MASK;
    long carry = 0;
    offset = out.length-offset - 1;
    for (int j=len-1; j >= 0; j--) {
      long product = (in[j] & LONG_MASK) * kLong +
          (out[offset] & LONG_MASK) + carry;
      out[offset--] = (int)product;
      carry = product >>> 32;
    }
    return (int)carry;
  }

It improved an internal microbenchmark that uses BigInteger.mulAdd by 12%, and crypto.rsa and crypto.signverify in SPECjvm2008 by 7% and 2.3%, respectively in my measurements.

Here's low level details. On x86 (32 bit), a 64 bit long multiplication is implemented as a sequence of multiple 32 bit multiplications as in:

  MOV    EBP,ECX.lo
  IMUL   EBP,EDX
  MOV    EDX,ECX.hi
  IMUL   EDX,EAX
  ADD    EBP,EDX
  MUL    EDX:EAX,ECX.lo
  ADD    EDX,EBP

This is based on the following idea:

Result = x * y

is implemented as

lo(result) = lo(x_lo * y_lo)

hi(result) = hi(x_lo * y_lo) + lo(x_hi * y_lo) + lo(x_lo * y_hi)

where lo(x) indicates the lower 32 bits of x and hi(x) the higher 32 bits of x.

In the above mulAdd code, the higher 32 bits of both of operands are known to be zero because of the '&' operations. In such cases, some 32 bit multiplications are redundant. In the above formula, if x_hi=0 and y_hi=0,

lo(result) = lo(x_lo * y_lo)
hi(result) = hi(x_lo * y_lo)

because lo(x_hi * y_lo) = 0 and lo(x_lo * y_hi) = 0.

In this case, we need to emit only

MUL    EDX:EAX,ECX.lo

which is one instruction instead of 7 and is faster to execute. There are variations of this where only one of x_hi and y_hi is zero.

I thank Chuck Rasbold, Tom Rodriguez, Christian Thalinger, and Vladimir Kozlov who reviewed this change.

GC Threads

There are two JVM flags/options that control how many GC threads are used in the JVM: ParallelGCThreads and ParallelCMSThreads.

ParallelGCThreads

This flag controls the number of threads used in the parallel garbage collector. This includes the young generation collector used by default. If the parallel GC is used (-XX:+UseParallelGC) or turned on by default on a 'server-class' machine, this is what you care with regard to the number of GC threads. Here's the formula that decides how many GC threads are used in the JVM on Linux/x86:

ParallelGCThreads = (ncpus <= 8) ? ncpus : 3 + ((ncpus * 5) / 8)

Some examples are:

When ncpus=4, ParallelGCThreads=4

When ncpus=8, ParallelGCThreads=8

When ncpus=16, ParallelGCThreads=13

A rationale for the number of GC threads lower than the core count in higher core count machines, that I can think of, is that parallel GC does not scale perfectly and the extra core count didn't help or even degraded the performance.

ParallelCMSThreads

This flag controls the number of threads used for the CMS (concurrent mark and sweep) garbage collector (-XX:+UseConcMarkSweepGC). CMS is often used to minimize the server latency by running the old generation GC with the application threads mostly concurrently. Even when CMS is used (for the old gen heap), a parallel GC is used for the young gen heap. So, the value of ParallelGCThreads still matters. Here's how the default value of ParallelCMSThreads is computed on Linux/x86:

ParallelCMSThreads = (ParallelGCThreads + 3) / 4

Some examples are:

When ncpus=4, ParallelCMSThreads =1

When ncpus=8, ParallelCMSThreads =2

When ncpus=16, ParallelCMSThreads =4

Typically, when the CMS GC is active, the CMS threads occupy the cores. The rest of the cores are available for application threads. For example, on a 8 core machine, since ParallelCMSThreads is 2, the remaining 6 cores are available for application threads. (As a side note, because all the threads have the same scheduling priority at the POSIX thread level in the JVM under Linux/x86, the CMS threads may not necessarily be on cores all of the time.)

Takeaways

Here are the takeaways for GC tuners out there:

• Since ParallelCMSThreads is computed based on the value of ParallelGCThreads, overriding ParallelGCThreads when using CMS affects ParallelCMSThreads and the CMS performance.
• Knowing how the default values of the flags helps better tune both the parallel GC and the CMS GC. Since the Sun JVM engineers probably empirically determined the default values in certain environment, it may not necessarily be the best for your environment.
• If you have worked around some multithreaded CMS crash bug in older Sun JDKs by running it single-threaded (for example this one), the workaround would have caused a tremendous performance degradation on many-core machines. So, if you run newer JDK and still uses the workaround, it's time to get rid of the workaround and allow CMS to take advantage of multicores.

Building OpenJDK faster

A basic build instruction was described here. But the full build takes long. Here are the variables that I use to build OpenJDK faster for everyday builds:

• NO_DOCS=true. This causes the build not to generate the javadoc docs for the JDK source code, which isn't very useful for daily engineering.
• NO_IMAGES=true and DEV_ONLY=true. By default, the JDK makefile builds the JDK and the JRE images (as in the state ready for deployment). For everyday debugging purposes, it's not necessary. By setting this variable to true, the image creation is omitted, which saves the build time.
• HOTSPOT_BUILD_JOBS=[ncpu] (for the JVM) and PARALLEL_COMPILE_JOBS=[ncpu](for the JDK). These are for parallel builds. By setting them to the number of the CPU cores available on the build machine, the build process runs in parallel.

There are ways to build either the JVM or the JDK parts of the build, which is a real build time saver.

When I only need to build the JVM part, here's how I build the JVM only:

• Go to the hotspot/make directory.
• Build make targets all_fastdebug copy_fastdebug_jdk export_fastdebug_jdk. Replace 'fastdebug' with 'product' for a product build. The JDK part of the build is copied from the import JDK.
• Look for a java launcher in hotspot/build/.

When I only need to build the JDK part, here's how I build the JDK only:

• Go to the jdk/make direcotry.
• Build make target fastdebug. The JVM part of the build is copied from the import JDK.
• Look for a java launcher in jdk/build/.

JVM process memory

Have you wondered what consumes memory in the JVM process? Here are the most of the list:

• The Java heap. The maximum size is controlled by flag -Xmx. This is where Java objects are allocated.
• The permanent generation (perm gen) heap. The maximum size is controlled by -XX:MaxPermSize. The default is 64MB on Linux/x86. This is where the JVM-level class metadata objects, interned strings (String.intern), and JVM-level symbol data are allocated. This often fills up unexpectedly when you use dynamic code/class generation in your application.
• The code cache. The JIT compiled native code is allocated here.
• The memory mapped .jar and .so files. The JDK's standard class library jar files and application's jar files are often memory mapped (typically only part of the files.) Various JDK shared library files (.so files) and application shared library files (JNI) are also memory mapped.
• The thread stacks. The maximum size of a thread's stack is controlled by flag -Xss or -X:ThreadStackSize. On Linux/x86, 320KB is the default (per thread.)
• The C/malloc heap. Both the JVM itself and any native code (either JDK's or application's) typically uses malloc to allocate memory from this heap. NIO direct buffers are allocated via malloc on Linux/x86.
• Any other mmap calls. Any native code could call to allocate pages in the address space using mmap.

A side note is that most of the above are allocated lazily. That is, they are allocated in terms of virtual memory early but committed only on demand. Your application's physical memory use (RSS) may look small under light load, but may get substantially high under heavy load. A takeaway is it makes sense to consider the above factors when diagnosing memory footprint problems in the JVM.

More 2D bugfixes

I worked on a patch to fix a bug in the Pisces renderer:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/fb03586d68b6

http://bugs.openjdk.java.net/show_bug.cgi?id=100030

The bug was that a circle was rendered like a 'C' shape, that is, the right most edge of the circle wasn't rendered properly. It turned out that there was a off-by-one error.

And there are a few more bugfixes in progress:

https://bugs.openjdk.java.net/show_bug.cgi?id=100035

https://bugs.openjdk.java.net/show_bug.cgi?id=100031

https://bugs.openjdk.java.net/show_bug.cgi?id=100088

https://bugs.openjdk.java.net/show_bug.cgi?id=100089

Thanks to Jennifer Godinez.

[Note: at least the 100031 patch has been checked into OpenJDK.]

A server compiler crash fix

Here's my latest contribution to OpenJDK:

http://hg.openjdk.java.net/jdk7/hotspot-comp/hotspot/rev/fd50a67f97d1

The patch fixes a server compiler crash that happens in a very rare condition, described here. A key attribute of the server compiler IR is that an instruction must dominate all its inputs. However, in the function remix_address_expressions(), this property was violated and caused a crash in a later split if transformation.

Thanks to Chuck Rasbold and Tom Rodriguez.

An event dispatch bug in JVMTI

I encountered an event dispatch bug in JVMTI. See here for the communication on the openjdk mailing list. Here's a summary of what happened.

According to the JVMTI spec, no JVMTI events should be sent during the JVMTI dead phase (after the VMDeath event was sent). However, I observed that the CompiledMethodLoad and CompiledMethodUnload events were sent during the dead phase after the Agent_OnUnload callback happened. These compile events were actually for the last Java method JIT-compiled. This can cause a nasty memory corruption bug because is Agent_OnUnload is usually where the data structures of a JVMTI agent are deallocated and the callback handlers for the above compile events touch the already-deallocated data structure.

After looking into the Hotspot code, I noticed that events dispatch and the JVMTI phase changes are not synchronized at all (i.e, race conditions). And in theory, this bug can happen not just for the two compile events, but for any events. In practice, this bug would probably happen for the compile events because those are triggered by the compile threads rather than application threads. I was able to suppress this bug in two ways.

1. By not deallocating memory (perhaps, only the one related to the compile events) in Agent_OnUnload. That way, late event callbacks handlers only touch still-valid memory.
2. By adding extra synchronization in the Hotspot JVMTI code (the details are in the mailing list log).

1 is a more practical approach where you cannot change the VM code, or you want to be portable. 2 is harder since it's not obvious what the performance implications would be and because once we start fixing the race conditions, we need to fix more race conditions.

As far as I can tell, the same race conditions exist in updating event callback handlers (SetEventCallbacks) and enabling/disabling individual event callback (SetEventNotificationMode). So, what does it mean? It means that, if you are a JVMTI agent writer, your request to change the event callback handlers in the middle of an application run or to disable an event dispatch temporarily and enable back later on may not be honored due to the race conditions. Scary? Yes, especially on modern multicore machines.

Java 2D Miter Line Join Decoration Bugfix

I'm not really a Java 2D person, but I contributed a small bugfix to OpenJDK for the miter line join rendering in the Java 2D Rendering Engine:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/9318628e8eee

What's miter line join? For a quick tutorial on the basics of Java 2D rendering, see this page from the Java tutorial:

http://java.sun.com/docs/books/tutorial/2d/geometry/strokeandfill.html

Multithreaded CMS crash

I ran across a CMS crash bug in Hotspot 10 b19 (OpenJDK6 b11), which is described here:

http://bugs.sun.com/view_bug.do?bug_id=6722116

The symptom is that VM crashes in the concurrent GC thread at the following stack trace (with mangled symbols):

_ZN24YieldingFlexibleWorkGang10start_taskEP24YieldingFlexibleGangTask
_ZN12CMSCollector13do_marking_mtEb
_ZN12CMSCollector13markFromRootsEb
_ZN12CMSCollector21collect_in_backgroundEb
_ZN25ConcurrentMarkSweepThread3runEv
_Z10java_startP6Thread

A workaround is to use the JVM option -XX:-CMSConcurrentMTEnabled to disable the multithreaded CMS collection. Upgrading to the JDK with newer Hotspot is a better idea, though.

Mysterious int arrays in a heap dump

This week I have been investigating the issue of mysterious (primitive) int array objects showing up in a heap dump (generated by the jmap utility or the Hotspot MXBean.) They are mysterious because they do not have any references to them (appears to be dead objects) according to the jhat output.

Today I learned that the garbage collectors may fabricate fake int array objects in certain cases. A heap compaction may not fully compact the heap and leave some 'holes' in the heap when the amount of the de-fragmented memory isn't worth the cost of compaction. Those holes are turned into int arrays perhaps because there is an assertion that the heap needs to look fully compacted for a certain reason, I guess. The int arrays are harmless because they look just like dead objects (unreachable from the roots) and will be garbage collected in a near-future garbage collection.

A server compiler crash during loop optimization

I encountered a server compiler crash which happens in a Google server. Here's a link to more details:

http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2008-September/000320.html

It appears to happen in the split-if graph transformation during the loop optimization. It only happens in a big Java server and I couldn't create a small test for the crash. I was able to create a small patch as a workaround that detects the crash condition early and aborts the JIT compilation of the particular method.

It is, unfortunately, often the case of JVM crashes only happen in an entire application run (production code) and they are not reproducible in tests. One possible reason is that only some particular method inlining patterns trigger crash during JIT compilation.

JNI crashes in FontManager code

I hit a JVM crash happening in the Java 2D font manager code. Here's the stack trace at the crash point:

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x298366]
C  [libfontmanager.so+0x59e4]
C  [libfreetype.so.6+0x73c9]  FT_Stream_Close+0x19
C  [libfreetype.so.6+0xa065]  FT_Stream_Free+0x25
C  [libfreetype.so.6+0xa6e2]
C  [libfreetype.so.6+0xaf78]  FT_Done_Face+0x78
C  [libfontmanager.so+0x6964]
Java_sun_font_FreetypeFontScaler_disposeNativeScaler+0x34

This appeared to be due to two JNI pitfalls:

• The JNIEnv is unique to the thread. It cannot be saved by one thread and reused by another. Use GetEnv instead.
• The font2D jobject in freetypeScaler.c needs to be converted into a global reference because its lifetime exceeds the lifetime of a native method call.

BTW, here's a link to one of the JNI references:

http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/jniTOC.html

Martin Buchholz and I suggested a patch. But here's a fix that was actually submitted by Igor Nekrestyanov:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/11d333de082f

Thanks, guys.

Java 2D incompatibility between Sun JDK and OpenJDK

The Java 2D font renderer component was replaced when Sun JDK was open-sourced as OpenJDK with an open source version. But it appeared to have some bug. It's was about the vertical position gap in rendered fonts. I reported this bug on the OpenJDK mailing list and got a bug at:

http://bugs.sun.com/view_bug.do?bug_id=6761856

The results from the test (included in the bug page):

Sun JDK:   x=5.78125,y=-47.796875,w=633.71484,h=57.515625
OpenJDK:   x=5.78125,y=0.0,w=637.21875,h=96.34375

As you can see, the vertical positions are significantly off.

A fix was provided in OpenJDK7 by Sun folks:

http://hg.openjdk.java.net/jdk7/2d/jdk/rev/9cdababf6179

Thanks to Phil Race and Igor Nekrestyanov.