question Re: CDH 5.1 Hive Metastore Server can't stay up in Archives of Support Questions (Read Only)

CDH 5.1 Hive Metastore Server can't stay up

tliu — Fri, 16 Sep 2022 09:06:55 GMT

CDH 5.1 is installed using Cloudera Manager 5.1. Oracle database is used for Hive Metastore and tables were populated after install. However, Hive Metastore Server was not started. I followed instruction in the installation guide to onfigure the Hive Metastore and edit hive-site.xml. After a restart, Hive Metastore Server stays up for about 5 minutes and then goes down. During that 5 minutes, There are about 11 hive user sessions connecting to the database. Below is the messages I found in the log:

2014-09-05 10:23:15,587 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Starting hive metastore on port 9083
2014-09-05 10:23:15,768 WARN org.apache.hadoop.hive.conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
2014-09-05 10:23:15,864 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2014-09-05 10:23:15,901 INFO org.apache.hadoop.hive.metastore.ObjectStore: ObjectStore, initialize called
2014-09-05 10:23:16,222 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
2014-09-05 10:23:16,644 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
2014-09-05 10:23:17,453 WARN org.apache.hadoop.hive.conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
2014-09-05 10:23:17,453 INFO org.apache.hadoop.hive.metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2014-09-05 10:23:17,502 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
2014-09-05 10:23:17,611 INFO org.apache.hadoop.hive.metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: ORA-00922: missing or invalid option

2014-09-05 10:23:21,465 INFO org.apache.hadoop.hive.metastore.ObjectStore: Initialized ObjectStore

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Fri, 05 Sep 2014 16:51:55 GMT

You should not have needed to modify hive-site.xml. What instructions did you follow?

Is there any information in the role's stderr log? Do the Java heap size charts look troubling around the time of the failure?

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Fri, 05 Sep 2014 20:03:17 GMT

I followed the section of Configuring the Hive Metastore in the CDH 5.1 installaiton guide. I modified hive-site.xml because I did not see javax.jdo.option.ConnectionURL in it. There is no stderr log for Hive Metastore role. I can't find Java heap size charts either.

During the installation, the Spark service can't start due to some data nodes are not started. After manually starting data nodes and Spark service, we manaully finish the rest of steps, like deploy client configuration file. Do you think that can be why javax.jdo.option.ConnectionURL is not in the hive-stie.xml?

Any advice would be appreciated.

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Fri, 05 Sep 2014 20:11:15 GMT

If you click on the Hive Metastore and look in the processes tab, you should be able to find the stderr.

You do not need to manually modify hive-site to add the connection URL. You are probably looking at the wrong copy hive-site.xml. The one used by the metastore is also shown on the processes tab.

The java heap size charts should be visible if you click on the Hive Metastore Server and then on Charts Library and look for the Resident Memory chart.

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Fri, 05 Sep 2014 22:15:17 GMT

Thanks. I found the the Resident Memory chart and the stderr. The processes tab only appears when the Hive Metastore Server is up, that is why I don't see it when the role is down.

The resident memory usage reached 1.1GB before the role went down. This situation repeats every single time. Is there is a limit set for java heap size? I am pasting the end of stderr below.

/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.3.0-cdh5.1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/lib/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.8.8.jar:/usr/lib/hadoop-mapreduce/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-mapreduce/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jaxb-ap
i-2.2.2.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/jsr305-1.3.9.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-
mapreduce/zookeeper.jar:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ [[ -n '' ]]
+ copy_hive_default /var/run/cloudera-scm-agent/process/454-hive-HIVEMETASTORE
+ [[ -z 5 ]]
+ '[' 1 -ne 1 ']'
+ [[ 5 == 3 ]]
+ [[ metastore == webhcat ]]
+ [[ metastore == updatelocation ]]
+ [[ metastore == create_metastore_tables ]]
+ [[ metastore == upgrade_metastore ]]
+ exec /usr/lib/hive/bin/hive --config /var/run/cloudera-scm-agent/process/454-hive-HIVEMETASTORE --service metastore -p 9083
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/09/05 17:14:56 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/09/05 17:14:57 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
+ kill -9 16129

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Sat, 06 Sep 2014 01:24:40 GMT

Yes, you can configure the heap size for the Hive Metastore Server. It's in the configuration page under Resource Management.

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Sun, 07 Sep 2014 15:58:17 GMT

I changed Hive: Client Java Heap size to 2GB, but the run time environment of Hive Metastore the process still shows 1GB. I need help how to change java heap size for metastore and make it effective. Thanks in advance.

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Sun, 07 Sep 2014 19:27:38 GMT

The client java heapsize will change the client process' heap, not the Metastore Server's heap. The client is the process that is run locally when you run "hive" or "beeline". Look for the relevant configuration for your Hive Metastore Server.

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Mon, 08 Sep 2014 20:37:02 GMT

I can't find any relevant heap size configuration for Hive Metastore Server. I did a search of heap size on Cloudera Manager page and two found related to Hive.

Hive: Client Java Heap Size in Bytes

Hive: Java Heap Size of HiveServer2 in Bytes

Neither of them works.

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Mon, 08 Sep 2014 21:09:12 GMT

You want:
Java Heap Size of Hive Metastore Server in Bytes

You can use the search box to find it, or look under Hive Metastore Server Default Group -> Resource Management for it.

Note that you can expand "Hive Metastore Server Default Group" to find the subcategory "Resource Management".

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Mon, 08 Sep 2014 22:35:08 GMT

Thanks for your help. I increased java heap size of hive metastore server in bytes to 6GB. It just took longer (25 minutes) for the process to die. The resident memory usage was 5.8GB.

I noticed that the runtime hive-site.xml has hive.metastore.local set to true and there is no hive.metastore.uris property. We are using Oracle as the data store. Is this normal?

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Mon, 08 Sep 2014 22:38:03 GMT

By default, CM configures /etc/hive/conf/hive-site.xml to talk to the metastore. Setting hive.metastore.local to true is generally a bad idea, and means your clients will talk to the database directly instead of going through the Hive Metastore Server (HMS). This can be controlled by the flag "Bypass Hive Metastore Server" in Hive configuration.

It's quite strange that your HMS would get so much memory when your clients are not talking to it. You might want to try debugging this memory usage on the Hive forums.

There's nothing wrong with using an Oracle database.

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Mon, 08 Sep 2014 22:40:14 GMT

Thank you for your help. I was able to increase java heap size of hive metatstore server in bytes to 6GB. It just took longer (25 minutes) for the process to die. The usage of resident memory was up to 5.8GB.
I notice that the runtime hive-site.xml has hive.metastore.local set to true and hive.metastore.uris is not there. We use Oracle Oracle as metastore. Is this normal?

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Mon, 08 Sep 2014 23:29:17 GMT

Bypass Hive Metastore Server is unchecked. I wonder why the runtime hive-site.xml has hive.metastore.local set to true. The server has 32GB RAM, but I don't think allocating more java heap size would help. It seems to have memory leak.

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Mon, 08 Sep 2014 23:44:08 GMT

You may just need to deploy client configuration. Until you do so, /etc/hive/conf can be stale.

You'll need to ask on hive forums to understand the memory issue.

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Tue, 09 Sep 2014 13:29:24 GMT

I deployed Hive client configuration files and /etc/hive/conf/hive-site.xml timestamp is up to date. The file has defined hive.metastore.uris. However, the runtime hive-site.xml does not hve hive.metastore.uris. The stderr of HMS shows "Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore". Any ideas how to resolve this issue?

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Tue, 09 Sep 2014 17:40:38 GMT

What do you mean by "the runtime hive-site.xml"? How are you running hive? Normally if you run "hive" from the command-line, it'll pick up /etc/hive/conf/hive-site.xml (on the same host as where you ran "hive").

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Tue, 09 Sep 2014 18:19:17 GMT

I restart HMS via Cloudera Manager. On the page of HMS/processes, the hive-site.xml listed under Configuration Files is what I meant runtime hive-site.xm, which has different contents than /etc/hive/conf/hive-site.xml.

Re: CDH 5.1 Hive Metastore Server can't stay up

Darren — Tue, 09 Sep 2014 18:23:59 GMT

I see what you mean now. The HMS is supposed to have different configuration. This is the one process that is supposed to talk to the relational database, so it sets "hive.metastore.local" to true and doesn't need to set the URIs (the URIs would just point to itself!). This is expected and not the cause of your memory issue.

Re: CDH 5.1 Hive Metastore Server can't stay up

tliu — Thu, 11 Sep 2014 15:26:43 GMT

The problem has been resolved by deleting Hive service and adding it back. The health test of HMS failed initially - the metastore canary failed to drop the table it created. Then it became good after a few minutes. The "show tables" command works in the Hive shell now.

The only mistake or cause I can think of is that we started panic after seeing the red/bad health test result right after the install. Then, we troubleshot and edited hive-site.xml by following Configuring the Hive Metastore instruction in the CDH installation guide, which is not necessary and probably caused the issue.