/wp

waving android

I am currently a software engineer at Google, where as a member of the Android platform team I build frameworks and user interfaces.

The blog here at is mostly historical; you can find more recent posts on .

Leakage.

January 3rd, 2006

Wow.

After chasing a FeedTree memory leak for a solid week (actually longer if you count the time during the semester that I half-heartedly feinted at the problem), I’ve finally narrowed it down. It’s a bug in the way the finite entry caches interact with the replay defense mechanism, which tracks UIDs for very old entries; that tracking system keeps them alive long after they ought to have expired from the cache, eventually eating up all your memory with ancient news.

Fig. 1: Memory leak. Like the Price is Right cliffhanger, this graph of the app’s memory usage goes up and up and up, until you hear an awful crashing noise.

Now allow me to briefly hate on the Java runtime memory profiler (-Xrunhprof:heap=all) and HAT (heap analysis tool): Once I figured out that I needed to use them, figuring out how to get them to work correctly was almost as unpleasant as not having them at all. (Answer: update your entire system to J2SE 1.5, and then try the tools on your program 50 times; it will crash inexplicably for the first 49.)

This leads to Dan’s garbage collection lemma: It’s impossible in a GC system to leak objects by losing them. But if, rather than losing them, you merely misplace them, you are well and truly screwed without good heap analysis tools.

Some of the problems I had with heap profiling:

  1. Sometimes hitting ctrl-\ in the JVM to cause a memory dump (binary format) causes a random crash. Example:


    Dumping Java heap ...HPROF ERROR: Cannot check is method native (JVMTI Error JVMTI_ERROR_INVALID_METHODID(23)) [../../../src/share/demo/jvmti/hprof/hprof_util.c:1157]

    HPROF TERMINATED PROCESS

  2. Sometimes the JVM generates incomplete dumps; HAT pukes a java.io.EOFException in response.
  3. Once you get a good dump, subsequent trips to ctrl-\ (or a System.exit or SIGINT) don’t seem to update the dumpfile. Or they update it in a way that HAT ignores. Yes, I tried using hat dump.hprof#X for various values of X; they all failed in spectacular ways.
  4. Another hateful exception from HAT:

    java.io.IOException: Thread 200000 not found for JNI local ref

Here’s the stack trace (in the HAT web interface) that finally clued me in to the location of a strong reference to an object that was supposed to have slipped out of a memory-sensitive cache:


Static reference from net.feedtree.proxyapp.WebProxyClient.global (from class net.feedtree.proxyapp.WebProxyClient) :
--> net.feedtree.proxyapp.WebProxyClient@0x50004bb3 (53 bytes) (field m_archives:)
--> java.util.Collections$SynchronizedMap@0x500051dc (20 bytes) (field m:)
--> java.util.HashMap@0x500051da (32 bytes) (field table:)
--> Instance of java.util.HashMap$Entry[]@0x500051db (16 bytes) (Element 3 of Instance of java.util.HashMap$Entry[]:)
--> java.util.HashMap$Entry@0x500043c0 (16 bytes) (field value:)
--> net.feedtree.proxyapp.FeedInfo@0x50004f2c (99 bytes) (field m_eventUriMap:)
--> java.util.Collections$SynchronizedMap@0x50003921 (20 bytes) (field m:)
--> java.util.WeakHashMap@0x5000391d (36 bytes) (field table:)
--> Instance of java.util.WeakHashMap$Entry[]@0x5000740b (256 bytes) (Element 1 of Instance of java.util.WeakHashMap$Entry[]:)
--> java.util.WeakHashMap$Entry@0x50002a75 (28 bytes) (field value:)
--> net.feedtree.core.messaging.FeedUpdateEvent@0x50002203 (12 bytes) (field m_data:)
--> java.lang.String@0x50002211 (16 bytes) (field value:)
--> ""@0x50002212 (0 bytes)

newer: older: