CDH 5.1 Hive Metastore Server can't stay up

[#1]

CDH 5.1 is installed using Cloudera Manager 5.1. Oracle database is used for Hive Metastore and tables were populated after install. However, Hive Metastore Server was not started. I followed instruction in the installation guide to onfigure the Hive Metastore and edit hive-site.xml. After a restart, Hive Metastore Server stays up for about 5 minutes and then goes down. During that 5 minutes, There are about 11 hive user sessions connecting to the database. Below is the messages I found in the log:

2014-09-05 10:23:15,587 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Starting hive metastore on port 9083
2014-09-05 10:23:15,768 WARN org.apache.hadoop.hive.conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
2014-09-05 10:23:15,864 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2014-09-05 10:23:15,901 INFO org.apache.hadoop.hive.metastore.ObjectStore: ObjectStore, initialize called
2014-09-05 10:23:16,222 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
2014-09-05 10:23:16,644 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
2014-09-05 10:23:17,453 WARN org.apache.hadoop.hive.conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
2014-09-05 10:23:17,453 INFO org.apache.hadoop.hive.metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2014-09-05 10:23:17,502 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
2014-09-05 10:23:17,611 INFO org.apache.hadoop.hive.metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: ORA-00922: missing or invalid option

2014-09-05 10:23:21,465 INFO org.apache.hadoop.hive.metastore.ObjectStore: Initialized ObjectStore

[#2]

You should not have needed to modify hive-site.xml. What instructions did you follow?

Is there any information in the role's stderr log? Do the Java heap size charts look troubling around the time of the failure?

[#3]

I followed the section of Configuring the Hive Metastore in the CDH 5.1 installaiton guide. I modified hive-site.xml because I did not see javax.jdo.option.ConnectionURL in it. There is no stderr log for Hive Metastore role. I can't find Java heap size charts either.

During the installation, the Spark service can't start due to some data nodes are not started. After manually starting data nodes and Spark service, we manaully finish the rest of steps, like deploy client configuration file. Do you think that can be why javax.jdo.option.ConnectionURL is not in the hive-stie.xml?

Any advice would be appreciated.

[#4]

If you click on the Hive Metastore and look in the processes tab, you should be able to find the stderr.

You do not need to manually modify hive-site to add the connection URL. You are probably looking at the wrong copy hive-site.xml. The one used by the metastore is also shown on the processes tab.

The java heap size charts should be visible if you click on the Hive Metastore Server and then on Charts Library and look for the Resident Memory chart.

[#5]

Thanks. I found the the Resident Memory chart and the stderr. The processes tab only appears when the Hive Metastore Server is up, that is why I don't see it when the role is down.

The resident memory usage reached 1.1GB before the role went down. This situation repeats every single time. Is there is a limit set for java heap size? I am pasting the end of stderr below.

/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/lib/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.8.8.jar:/usr/lib/hadoop-mapreduce/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-mapreduce/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jaxb-ap
i-2.2.2.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/jsr305-1.3.9.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-
mapreduce/zookeeper.jar:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -n '' ]]
+ copy_hive_default /var/run/cloudera-scm-agent/process/454-hive-HIVEMETASTORE
+ [[ -z 5 ]]
+ '[' 1 -ne 1 ']'
+ [[ 5 == 3 ]]
+ [[ metastore == webhcat ]]
+ [[ metastore == updatelocation ]]
+ [[ metastore == create_metastore_tables ]]
+ [[ metastore == upgrade_metastore ]]
+ exec /usr/lib/hive/bin/hive --config /var/run/cloudera-scm-agent/process/454-hive-HIVEMETASTORE --service metastore -p 9083
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/09/05 17:14:57 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
+ kill -9 16129

[#6]

Yes, you can configure the heap size for the Hive Metastore Server. It's in the configuration page under Resource Management.

[#7]

I changed Hive: Client Java Heap size to 2GB, but the run time environment of Hive Metastore the process still shows 1GB. I need help how to change java heap size for metastore and make it effective. Thanks in advance.

[#8]

The client java heapsize will change the client process' heap, not the Metastore Server's heap. The client is the process that is run locally when you run "hive" or "beeline". Look for the relevant configuration for your Hive Metastore Server.

[#9]

I can't find any relevant heap size configuration for Hive Metastore Server. I did a search of heap size on Cloudera Manager page and two found related to Hive.

Hive: Client Java Heap Size in Bytes

Hive: Java Heap Size of HiveServer2 in Bytes

Neither of them works.

[#10]

You want:
Java Heap Size of Hive Metastore Server in Bytes

You can use the search box to find it, or look under Hive Metastore Server Default Group -> Resource Management for it.

Note that you can expand "Hive Metastore Server Default Group" to find the subcategory "Resource Management".