Hi,
Our Spring application has jackrabbit content repository. It runs quartz based crons. It uses hibernate underneath.
We are seeing the following exception in the logs.
2013-06-05 12:12:27,616 [ClusterNode-node1] ERROR org.apache.jackrabbit.core.cluster.ClusterNode - Periodic sync of journal failed: Couldn't sync the cluster node
org.apache.jackrabbit.core.cluster.ClusterExceptio n: Couldn't sync the cluster node
at org.apache.jackrabbit.core.cluster.ClusterNode.syn c(ClusterNode.java:331)
at org.apache.jackrabbit.core.cluster.ClusterNode.run (ClusterNode.java:293)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsExce ption: Communications link failure
The last packet successfully received from the server was 362,492 milliseconds ago. The last packet sent successfully to the server was 19,111 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInsta nce0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInsta nce(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newI nstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Construc tor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:41 1)
at com.mysql.jdbc.SQLError.createCommunicationsExcept ion(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:3102)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:2991)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:3532)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:20 02)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionIm pl.java:2618)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(Connec tionImpl.java:5000)
at org.apache.commons.dbcp.DelegatingConnection.setAu toCommit(DelegatingConnection.java:371)
at org.apache.commons.dbcp.DelegatingConnection.setAu toCommit(DelegatingConnection.java:371)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuar dConnectionWrapper.setAutoCommit(PoolingDataSource .java:328)
at org.apache.jackrabbit.core.util.db.ConnectionHelpe r.startBatch(ConnectionHelper.java:240)
at org.apache.jackrabbit.core.journal.DatabaseJournal .startBatch(DatabaseJournal.java:506)
at org.apache.jackrabbit.core.journal.DatabaseJournal .doSync(DatabaseJournal.java:450)
at org.apache.jackrabbit.core.journal.AbstractJournal .internalSync(AbstractJournal.java:206)
at org.apache.jackrabbit.core.journal.AbstractJournal .sync(AbstractJournal.java:189)
at org.apache.jackrabbit.core.cluster.ClusterNode.syn c(ClusterNode.java:328)
... 2 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2552 )
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:3002)
... 17 more
2013-06-05 12:12:29,934 [QuartzScheduler_org.springframework.scheduling.qua rtz.SchedulerFactoryBean#0-bf1-rmaicic1.ads.bf1.yahoo.com1370389426455_MisfireHan dler] ERROR org.springframework.scheduling.quartz.LocalDataSou rceJobStore - MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source 'springNonTxDataSource.org.springframework.schedul ing.quartz.SchedulerFactoryBean#0': java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.org.springframework.schedul ing.quartz.SchedulerFactoryBean#0': java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space [See nested exception: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonMan agedTXConnection(JobStoreCMT.java:165)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRec overMisfires(JobStoreSupport.java:3132)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$Misfi reHandler.manage(JobStoreSupport.java:3923)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$Misfi reHandler.run(JobStoreSupport.java:3943)
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
We are seeing that our spring application has around 73 connections to our remote mysql server.
netstat -a | grep mysql | wc -l
63
1. How do I find which class in the application corresponds to each of the 63 connections.
This number starts with around 14 and increases to a large number like 73. The cause needs to be investigated.
2. How do I find the DB username used for each of these connections.
3. I had seen around 87 threads running under tomcat at the time of the error. Please find them attached in this email
4. Some pointers on how to proceed further would help.
Thanks
-Priyanka
Our Spring application has jackrabbit content repository. It runs quartz based crons. It uses hibernate underneath.
We are seeing the following exception in the logs.
2013-06-05 12:12:27,616 [ClusterNode-node1] ERROR org.apache.jackrabbit.core.cluster.ClusterNode - Periodic sync of journal failed: Couldn't sync the cluster node
org.apache.jackrabbit.core.cluster.ClusterExceptio n: Couldn't sync the cluster node
at org.apache.jackrabbit.core.cluster.ClusterNode.syn c(ClusterNode.java:331)
at org.apache.jackrabbit.core.cluster.ClusterNode.run (ClusterNode.java:293)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsExce ption: Communications link failure
The last packet successfully received from the server was 362,492 milliseconds ago. The last packet sent successfully to the server was 19,111 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInsta nce0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInsta nce(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newI nstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Construc tor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:41 1)
at com.mysql.jdbc.SQLError.createCommunicationsExcept ion(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:3102)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:2991)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:3532)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:20 02)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionIm pl.java:2618)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(Connec tionImpl.java:5000)
at org.apache.commons.dbcp.DelegatingConnection.setAu toCommit(DelegatingConnection.java:371)
at org.apache.commons.dbcp.DelegatingConnection.setAu toCommit(DelegatingConnection.java:371)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuar dConnectionWrapper.setAutoCommit(PoolingDataSource .java:328)
at org.apache.jackrabbit.core.util.db.ConnectionHelpe r.startBatch(ConnectionHelper.java:240)
at org.apache.jackrabbit.core.journal.DatabaseJournal .startBatch(DatabaseJournal.java:506)
at org.apache.jackrabbit.core.journal.DatabaseJournal .doSync(DatabaseJournal.java:450)
at org.apache.jackrabbit.core.journal.AbstractJournal .internalSync(AbstractJournal.java:206)
at org.apache.jackrabbit.core.journal.AbstractJournal .sync(AbstractJournal.java:189)
at org.apache.jackrabbit.core.cluster.ClusterNode.syn c(ClusterNode.java:328)
... 2 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2552 )
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:3002)
... 17 more
2013-06-05 12:12:29,934 [QuartzScheduler_org.springframework.scheduling.qua rtz.SchedulerFactoryBean#0-bf1-rmaicic1.ads.bf1.yahoo.com1370389426455_MisfireHan dler] ERROR org.springframework.scheduling.quartz.LocalDataSou rceJobStore - MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source 'springNonTxDataSource.org.springframework.schedul ing.quartz.SchedulerFactoryBean#0': java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.org.springframework.schedul ing.quartz.SchedulerFactoryBean#0': java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space [See nested exception: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonMan agedTXConnection(JobStoreCMT.java:165)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRec overMisfires(JobStoreSupport.java:3132)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$Misfi reHandler.manage(JobStoreSupport.java:3923)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$Misfi reHandler.run(JobStoreSupport.java:3943)
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
We are seeing that our spring application has around 73 connections to our remote mysql server.
netstat -a | grep mysql | wc -l
63
1. How do I find which class in the application corresponds to each of the 63 connections.
This number starts with around 14 and increases to a large number like 73. The cause needs to be investigated.
2. How do I find the DB username used for each of these connections.
3. I had seen around 87 threads running under tomcat at the time of the error. Please find them attached in this email
4. Some pointers on how to proceed further would help.
Thanks
-Priyanka