I have a web java application, which serves rest-like json data to javascript client side.
Server part is processing HPPT GET/POST requests only.
Technology stack:
JDK 1.6
Debian linux 6/7
tomacat 6.0.36
spring 3.2.1
hibernate entity manager
MySql 5.0.23
+ some dependent libs
Now application is not working on a virtual hosting server, because tomcat process is terminated by system due to memory consumption limit.
JVM takes 20-30mb in total, but tomcat process allocates >1Gb in resident set memory. Such process memory size is not acceptable in my environment. I used pmap, top tools to identify that.
I spent a very long time on dump analysis with tools like jmap, MAT, and I fixed a Classloader leaks, but that didn't help.
Various combinations of JVM heap size, perm gen size, garbage collection tuning do not help to reduce memory consumption.
After all, I'm suspecting a native memory leak. I have identified some debug details with valgrind tool.
==3156== 517 bytes in 1 blocks are definitely lost in loss record 217 of 227
==3156== at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==3156== by 0x6448D36: os::malloc(unsigned long) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6448DA0: os::strdup(char const*) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x61235C3: ClassLoader::setup_bootstrap_search_path() (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6123C91: ClassLoader::initialize() (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6259356: init_globals() (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6555F9A: Threads::create_vm(JavaVMInitArgs*, bool*) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x62A15CF: JNI_CreateJavaVM (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x40299D: JavaMain (in /usr/lib/jvm/java-6-openjdk/jre/bin/java)
==3156== by 0x50468C9: start_thread (pthread_create.c:300)
==3156==
==3156== 2,952 bytes in 1 blocks are possibly lost in loss record 225 of 227
==3156== at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==3156== by 0x6448D36: os::malloc(unsigned long) (in /usr/lib/jvm/java-6-openjdk /jre/lib/amd64/server/libjvm.so)
==3156== by 0x5FFAB60: CHeapObj::operator new(unsigned long) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x654EBB7: Thread::operator new(unsigned long) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6555F44: Threads::create_vm(JavaVMInitArgs*, bool*) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x62A15CF: JNI_CreateJavaVM (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x40299D: JavaMain (in /usr/lib/jvm/java-6-openjdk/jre/bin/java)
==3156== by 0x50468C9: start_thread (pthread_create.c:300)
==3156==
==3156== LEAK SUMMARY:
==3156== definitely lost: 2,556 bytes in 32 blocks
==3156== indirectly lost: 240 bytes in 10 blocks
==3156== possibly lost: 6,456 bytes in 14 blocks
==3156== still reachable: 32,668 bytes in 302 blocks
==3156== suppressed: 0 bytes in 0 blocks
==3156== Reachable blocks (those to which a pointer was found) are not shown.
==3156== To see them, rerun with: --leak-check=full --show-reachable=yes
==3156==
==3156== For counts of detected and suppressed errors, rerun with: -v
==3156== ERROR SUMMARY: 29 errors from 29 contexts (suppressed: 16 from 7)
It seems Tomcat process allocated system memory (resources) outside of JVM and never clear it.
I haven't done any specific coding related to resource processing. No NIO, JNI... Please help me to find a cause of native memory leak! I appreciate your help!
Server part is processing HPPT GET/POST requests only.
Technology stack:
JDK 1.6
Debian linux 6/7
tomacat 6.0.36
spring 3.2.1
hibernate entity manager
MySql 5.0.23
+ some dependent libs
Now application is not working on a virtual hosting server, because tomcat process is terminated by system due to memory consumption limit.
JVM takes 20-30mb in total, but tomcat process allocates >1Gb in resident set memory. Such process memory size is not acceptable in my environment. I used pmap, top tools to identify that.
I spent a very long time on dump analysis with tools like jmap, MAT, and I fixed a Classloader leaks, but that didn't help.
Various combinations of JVM heap size, perm gen size, garbage collection tuning do not help to reduce memory consumption.
After all, I'm suspecting a native memory leak. I have identified some debug details with valgrind tool.
==3156== 517 bytes in 1 blocks are definitely lost in loss record 217 of 227
==3156== at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==3156== by 0x6448D36: os::malloc(unsigned long) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6448DA0: os::strdup(char const*) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x61235C3: ClassLoader::setup_bootstrap_search_path() (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6123C91: ClassLoader::initialize() (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6259356: init_globals() (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6555F9A: Threads::create_vm(JavaVMInitArgs*, bool*) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x62A15CF: JNI_CreateJavaVM (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x40299D: JavaMain (in /usr/lib/jvm/java-6-openjdk/jre/bin/java)
==3156== by 0x50468C9: start_thread (pthread_create.c:300)
==3156==
==3156== 2,952 bytes in 1 blocks are possibly lost in loss record 225 of 227
==3156== at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==3156== by 0x6448D36: os::malloc(unsigned long) (in /usr/lib/jvm/java-6-openjdk /jre/lib/amd64/server/libjvm.so)
==3156== by 0x5FFAB60: CHeapObj::operator new(unsigned long) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x654EBB7: Thread::operator new(unsigned long) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x6555F44: Threads::create_vm(JavaVMInitArgs*, bool*) (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x62A15CF: JNI_CreateJavaVM (in /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so)
==3156== by 0x40299D: JavaMain (in /usr/lib/jvm/java-6-openjdk/jre/bin/java)
==3156== by 0x50468C9: start_thread (pthread_create.c:300)
==3156==
==3156== LEAK SUMMARY:
==3156== definitely lost: 2,556 bytes in 32 blocks
==3156== indirectly lost: 240 bytes in 10 blocks
==3156== possibly lost: 6,456 bytes in 14 blocks
==3156== still reachable: 32,668 bytes in 302 blocks
==3156== suppressed: 0 bytes in 0 blocks
==3156== Reachable blocks (those to which a pointer was found) are not shown.
==3156== To see them, rerun with: --leak-check=full --show-reachable=yes
==3156==
==3156== For counts of detected and suppressed errors, rerun with: -v
==3156== ERROR SUMMARY: 29 errors from 29 contexts (suppressed: 16 from 7)
It seems Tomcat process allocated system memory (resources) outside of JVM and never clear it.
I haven't done any specific coding related to resource processing. No NIO, JNI... Please help me to find a cause of native memory leak! I appreciate your help!