Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!
Labels (1)
avatar
Super Guru

SYMPTOM:

beeline connection to hiveserver2 is failing with the following exception intermittently.

"The initCause method cannot be used. To set the cause of this exception, use a constructor with a Throwable[] argument."

further looking at the HiveServer2 logs we saw stack trace like this

ava.sql.SQLException: Could not retrieve transation read-only status server
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
        at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2259)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getProductName(MetaStoreDirectSql.java:171)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.determineDbType(MetaStoreDirectSql.java:152)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:122)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:300)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:263)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
        at com.sun.proxy.$Proxy7.setConf(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.setConf(HiveMetaStore.java:523)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:56)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5798)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
        ... 35 more
Caused by: java.sql.SQLException: Could not retrieve transation read-only status server
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1086)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:951)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:941)
        at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3972)
        at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3943)
        at com.jolbox.bonecp.ConnectionHandle.isReadOnly(ConnectionHandle.java:867)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:422)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.getNucleusConnection(RDBMSStoreManager.java:1382)
        at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2245)
        ... 51 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure


The last packet successfully received from the server was 381,109 milliseconds ago.  The last packet sent successfully to the server was
381,109 milliseconds ago.
        at sun.reflect.GeneratedConstructorAccessor55.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3988)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2598)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3966)
        ... 56 more
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3969)

ROOT CAUSE:

After analyzing TCP DUMP we found that it's MySQL server who is dropping network packets very frequently.

WORKAROUND:

NA

RESOLUTION:

need to check and fix the network issue between HiveServer2 host and MySQL host.

3,071 Views
Version history
Last update:
‎12-21-2016 05:07 PM
Updated by:
Contributors