Support Questions
Find answers, ask questions, and share your expertise

CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Highlighted

CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Rising Star

First of I am a CDH and Cloudera Manager (starving developers version) user for 4+ years now and a big thanks to the CDH folks.

 

I am facing some errors in the new 3 node CDH cluster I setup this weekend. I rebuilt 3 Ubuntu Boxes 12.04LTS and installed thru Cloudera Manager as brand new installation. For the MYSQL hive metastore, I am using  

 

1. The host inspector fails on one host 

 

+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/196-host-inspector:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-jdbc.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
+ echo_and_exec /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/196-host-inspector:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-jdbc.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ echo 'Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java' -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/196-host-inspector:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-jdbc.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ exec /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/196-host-inspector:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-jdbc.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
Exception in thread "main" java.lang.NoClassDefFoundError: com/cloudera/enterprise/I18nKey
	at com.cloudera.cmf.inspector.Inspection.<clinit>(Inspection.java:40)
	at com.cloudera.cmf.inspector.Inspection$Inspections.<clinit>(Inspection.java:18)
	at com.cloudera.cmf.inspector.Inspector.parseArgs(Inspector.java:101)
	at com.cloudera.cmf.inspector.Inspector.run(Inspector.java:47)
	at com.cloudera.cmf.inspector.Inspector.main(Inspector.java:43)
Caused by: java.lang.ClassNotFoundException: com.cloudera.enterprise.I18nKey
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 5 more

 

Hive Metastore fails

Hive starts up correctly when I start the entire cluster and I can issue commands like hive -e "show databases" and t works. But within 3 minutes Hive Metastore dies.

 

 

2016-01-25 12:15:46,138 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-3-thread-200]: Retrying HMSHandler after

2000 ms (attempt 4 of 10) with error: javax.jdo.JDODataStoreException: You have an error in your SQL syntax; check the manual that

corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)

        at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:275)

        at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1059)

        at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:988)

        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)

        at com.sun.proxy.$Proxy10.getTable(Unknown Source)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1801)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1773)

        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:133)

        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)

        at com.sun.proxy.$Proxy12.get_table(Unknown Source)

        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table.getResult(ThriftHiveMetastore.java:9266)

        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table.getResult(ThriftHiveMetastore.java:9250)

        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)

        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)

        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

  

 

 

 

Inspector Results Validations

 Inspector failed on the following hosts...
  • hadoop03.corp.rhalf.com: Inspector did not run successfully.
Inspector ran on 2 hosts.
 The following failures were observed in checking hostnames...
 No errors were found while looking for conflicting init scripts.
 No errors were found while checking /etc/hosts.
 All hosts resolved localhost to 127.0.0.1.
 All hosts checked resolved each other's hostnames correctly and in a timely manner.
 Host clocks are approximately in sync (within ten minutes).
 Host time zones are consistent across the cluster.
 No users or groups are missing.
 No conflicts detected between packages and parcels.
 No kernel versions that are known to be bad are running.
 All hosts have /proc/sys/vm/swappiness set to 0.
 No performance concerns with Transparent Huge Pages settings.
 CDH 5 Hue Python version dependency is satisfied.
 0 hosts are running CDH 4 and 3 hosts are running CDH5.
 All checked hosts in each cluster are running the same version of components.
 All managed hosts have consistent versions of Java.
 All checked Cloudera Management Daemons versions are consistent with the server.
 All checked Cloudera Management Agents versions are consistent with the server.

Version Summary HostsComponent Version Release CDH Version

Cluster 1 — CDH 5
hadoop[01-03].corp.rhalf.com
Bigtop-Tomcat (CDH 5 only)0.7.0+cdh5.5.1+01.cdh5.5.1.p0.14CDH 5
Crunch (CDH 5 only)0.11.0+cdh5.5.1+771.cdh5.5.1.p0.14CDH 5
Flume NG1.6.0+cdh5.5.1+291.cdh5.5.1.p0.16CDH 5
MapReduce 12.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
Hadoop2.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
HDFS2.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
HttpFS2.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
hadoop-kms2.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
MapReduce 22.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
YARN2.6.0+cdh5.5.1+9241.cdh5.5.1.p0.15CDH 5
HBase1.0.0+cdh5.5.1+2741.cdh5.5.1.p0.15CDH 5
Lily HBase Indexer1.5+cdh5.5.1+571.cdh5.5.1.p0.14CDH 5
Hive1.1.0+cdh5.5.1+3271.cdh5.5.1.p0.15CDH 5
HCatalog1.1.0+cdh5.5.1+3271.cdh5.5.1.p0.15CDH 5
Hue3.9.0+cdh5.5.1+3331.cdh5.5.1.p0.14CDH 5
Impala2.3.0+cdh5.5.1+01.cdh5.5.1.p0.17CDH 5
Kite (CDH 5 only)1.0.0+cdh5.5.1+1161.cdh5.5.1.p0.14CDH 5
Llama (CDH 5 only)1.0.0+cdh5.5.1+01.cdh5.5.1.p0.14CDH 5
Mahout0.9+cdh5.5.1+261.cdh5.5.1.p0.15CDH 5
Oozie4.1.0+cdh5.5.1+2231.cdh5.5.1.p0.14CDH 5
Parquet1.5.0+cdh5.5.1+1761.cdh5.5.1.p0.15CDH 5
Pig0.12.0+cdh5.5.1+721.cdh5.5.1.p0.15CDH 5
sentry1.5.1+cdh5.5.1+1061.cdh5.5.1.p0.15CDH 5
Solr4.10.3+cdh5.5.1+3251.cdh5.5.1.p0.15CDH 5
spark1.5.0+cdh5.5.1+941.cdh5.5.1.p0.15CDH 5
Sqoop21.99.5+cdh5.5.1+331.cdh5.5.1.p0.14CDH 5
Sqoop1.4.6+cdh5.5.1+291.cdh5.5.1.p0.14CDH 5
Whirr0.9.0+cdh5.5.1+171.cdh5.5.1.p0.15CDH 5
Zookeeper3.4.5+cdh5.5.1+911.cdh5.5.1.p0.15CDH 5
Cloudera Manager Management Daemons5.5.11.cm551.p0.8Not applicable
Supervisord3.0-cm5.5.1UnavailableNot applicable
Java 6JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)UnavailableNot applicable
Java 7JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)UnavailableNot applicable
Cloudera Manager Agent5.5.11.cm551.p0.8Not applicable
6 REPLIES 6
Highlighted

Re: CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Super Collaborator

The following exception seems odd as it suggests that java is unable to locate a jar

 

Exception in thread "main" java.lang.NoClassDefFoundError: com/cloudera/enterprise/I18nKey

 

 

This is a commonly used jar, and it is installed as part of the cloudera-manager-daemons package under /usr/share/cmf/common_jars/common-5.5.1.jar. (in your output /usr/share/cmf/lib/*)

Would you mind re-installing (or force re-install) CM Server and Daemon [1]

 

For Hive, what is the version of your MySQL JDBC Driver and Server - check if the issue persists after upgrading your Connector/J [2] as it may be a defect from MySQL JDBC [3]

 

 

[1] http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b_auto.html

[2] http://dev.mysql.com/downloads/connector/j/

[3] https://netbeans.org/bugzilla/show_bug.cgi?id=224260

 

 

Highlighted

Re: CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Rising Star

Thanks I was using a 5.6.23 version 

 

I found an earlier version 5.5.40b that works in my 9 node cluster ; so I pointed Hive metaStore to that and recreated all the metastore, anon etc databases. And create the hive tables thru CM

 

So Hive is working now :-) 

 

Now on to trying your recommendation for fixing the Hosts Inspector 

 

Highlighted

Re: CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Rising Star

OK good news

 

I found that the follwing jars were missing on the node where the host inspector was failing 

/usr/share/cmf/lib

 

cdh-client-5.5.1.jar

cloudera-manager-api-5.5.1.jar

common-5.5.1.jar

db-common-5.5.1.jar

event-common-5.5.1.jar -> ../common_jars/event-common-5.5.1.jar

nav-common-5.5.1.jar

common-thrifts-5.5.1.jar

db-common-5.5.1.jar

debug-5.5.1.jar

nav-common-5.5.1.jar

protocol-5.5.1.jar

security-5.5.1.jar

 

I added these manually 

 

Now the host inspectors work correctly 

 

Highlighted

Re: CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Rising Star

The Hive problem that I stated here is gone but I cannot create EXTERNAL tables

 

I have logged the issue here 

http://community.cloudera.com/t5/Cloudera-Manager-Installation/Incomplete-HDFS-URI-While-Creating-An...

 

Highlighted

Re: CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Rising Star

Actually MANAGED and EXTERNAL tables this is an issue 

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException Incomplete HDFS URI, no host: hdfs://hadoop01.corp.rhalf.com:8020:8020/user/hive/warehouse/cdr.db/cdr_mjp_uni_srch_usr_hist)

 

Re: CDH 5.5.1 - Issues in Hive Metastore and Hosts Inspector

Super Collaborator

Based on your feedback, the two issues intially reported in this thread seems to have been resolved. 

For the additional "... MANAGED and EXTERNAL tables this is an issue", let's move our conversation in the thread you've linked previously.